Nov 03 20:00:59.240241 ip-10-0-132-4 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Nov 03 20:00:59.240253 ip-10-0-132-4 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Nov 03 20:00:59.240263 ip-10-0-132-4 systemd[1]: kubelet.service: Failed with result 'resources'.
Nov 03 20:00:59.240587 ip-10-0-132-4 systemd[1]: Failed to start Kubernetes Kubelet.
Nov 03 20:01:09.257584 ip-10-0-132-4 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Nov 03 20:01:09.257600 ip-10-0-132-4 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot f47228acc33d4e1ca9ea7ab885c36768 --
Nov 03 20:03:14.028096 ip-10-0-132-4 systemd[1]: Starting Kubernetes Kubelet...
Nov 03 20:03:14.406655 ip-10-0-132-4 kubenswrapper[2577]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 20:03:14.406655 ip-10-0-132-4 kubenswrapper[2577]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 03 20:03:14.406655 ip-10-0-132-4 kubenswrapper[2577]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 20:03:14.406655 ip-10-0-132-4 kubenswrapper[2577]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 20:03:14.413049 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.412933    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 20:03:14.413049 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.413045    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 20:03:14.413117 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.413057    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 20:03:14.413117 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.413062    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 20:03:14.413117 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.413065    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 20:03:14.413117 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.413069    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 20:03:14.413117 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.413072    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 20:03:14.413117 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.413076    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 20:03:14.413117 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.413082    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 20:03:14.413117 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.413085    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 20:03:14.413117 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.413089    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 20:03:14.413117 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.413091    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 20:03:14.413117 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.413095    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 20:03:14.413117 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.413098    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 20:03:14.413117 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.413101    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 20:03:14.413117 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.413103    2577 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 20:03:14.413117 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.413106    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 20:03:14.413117 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.413109    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 20:03:14.413117 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.413112    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 20:03:14.413117 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.413115    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 20:03:14.413117 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.413118    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 20:03:14.413600 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.413120    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 20:03:14.413600 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.413124    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 20:03:14.413600 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.413127    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 20:03:14.413600 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.413129    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 20:03:14.413600 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.413132    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 20:03:14.413600 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.413135    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 20:03:14.413600 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.413138    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 20:03:14.413600 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.413140    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 20:03:14.413600 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.413143    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 20:03:14.413600 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.413145    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 20:03:14.413600 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.413148    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 20:03:14.413600 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.413152    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 20:03:14.413600 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.413155    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 20:03:14.413600 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.413157    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 20:03:14.413600 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.413160    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 20:03:14.413600 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.413162    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 20:03:14.413600 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.413165    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 20:03:14.413600 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.413167    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 20:03:14.413600 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.413170    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 20:03:14.414059 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.413172    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 20:03:14.414059 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.413175    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 20:03:14.414059 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.413178    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 20:03:14.414059 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.413181    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 20:03:14.414059 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.413183    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 20:03:14.414059 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.413186    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 20:03:14.414059 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.413188    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 20:03:14.414059 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.413191    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 20:03:14.414059 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.413193    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 20:03:14.414059 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.413195    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 20:03:14.414059 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.413199    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 20:03:14.414059 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.413203    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 20:03:14.414059 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.413207    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 20:03:14.414059 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.413209    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 20:03:14.414059 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.413212    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 20:03:14.414059 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.413215    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 20:03:14.414059 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.413218    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 20:03:14.414059 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.413221    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 20:03:14.414059 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.413224    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 20:03:14.414503 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.413226    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 20:03:14.414503 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.413229    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 20:03:14.414503 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.413232    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 20:03:14.414503 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.413234    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 20:03:14.414503 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.413237    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 20:03:14.414503 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.413239    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 20:03:14.414503 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.413242    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 20:03:14.414503 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.413245    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 20:03:14.414503 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.413247    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 20:03:14.414503 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.413250    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 20:03:14.414503 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.413252    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 20:03:14.414503 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.413254    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 20:03:14.414503 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.413257    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 20:03:14.414503 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.413259    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 20:03:14.414503 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.413262    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 20:03:14.414503 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.413264    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 20:03:14.414503 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.413267    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 20:03:14.414503 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.413269    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 20:03:14.414503 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.413272    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 20:03:14.414972 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.413274    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 20:03:14.414972 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.413277    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 20:03:14.414972 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.413279    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 20:03:14.414972 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.413281    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 20:03:14.414972 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.413284    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 20:03:14.414972 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.413287    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 20:03:14.414972 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.413289    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 20:03:14.414972 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.413291    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 20:03:14.414972 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.413295    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 20:03:14.414972 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.413297    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 20:03:14.414972 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.413301    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 20:03:14.414972 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.413303    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 20:03:14.414972 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.413306    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 20:03:14.414972 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.413308    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 20:03:14.414972 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.413311    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 20:03:14.414972 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.413313    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 20:03:14.414972 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.413316    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 20:03:14.414972 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.413319    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 20:03:14.414972 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.413321    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 20:03:14.414972 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.413324    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 20:03:14.415441 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.413327    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 20:03:14.415441 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.413331    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 20:03:14.415441 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.413333    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 20:03:14.415441 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.413336    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 20:03:14.415441 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.413338    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 20:03:14.415441 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.413340    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 20:03:14.415441 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.413343    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 20:03:14.415441 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.413345    2577 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 20:03:14.415441 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.414150    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 20:03:14.415441 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.414158    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 20:03:14.415441 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.414162    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 20:03:14.415441 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.414164    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 20:03:14.415441 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.414167    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 20:03:14.415441 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.414170    2577 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 20:03:14.415441 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.414173    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 20:03:14.415441 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.414175    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 20:03:14.415441 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.414178    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 20:03:14.415441 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.414180    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 20:03:14.415441 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.414183    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 20:03:14.415948 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.414186    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 20:03:14.415948 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.414189    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 20:03:14.415948 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.414192    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 20:03:14.415948 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.414195    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 20:03:14.415948 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.414197    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 20:03:14.415948 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.414201    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 20:03:14.415948 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.414204    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 20:03:14.415948 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.414206    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 20:03:14.415948 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.414209    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 20:03:14.415948 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.414211    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 20:03:14.415948 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.414214    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 20:03:14.415948 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.414217    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 20:03:14.415948 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.414219    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 20:03:14.415948 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.414222    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 20:03:14.415948 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.414225    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 20:03:14.415948 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.414228    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 20:03:14.415948 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.414231    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 20:03:14.415948 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.414233    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 20:03:14.415948 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.414236    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 20:03:14.416393 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.414238    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 20:03:14.416393 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.414241    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 20:03:14.416393 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.414244    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 20:03:14.416393 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.414246    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 20:03:14.416393 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.414249    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 20:03:14.416393 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.414251    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 20:03:14.416393 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.414254    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 20:03:14.416393 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.414256    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 20:03:14.416393 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.414259    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 20:03:14.416393 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.414261    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 20:03:14.416393 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.414264    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 20:03:14.416393 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.414267    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 20:03:14.416393 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.414269    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 20:03:14.416393 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.414274    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 20:03:14.416393 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.414279    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 20:03:14.416393 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.414284    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 20:03:14.416393 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.414288    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 20:03:14.416393 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.414292    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 20:03:14.416393 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.414295    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 20:03:14.416867 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.414298    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 20:03:14.416867 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.414301    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 20:03:14.416867 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.414303    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 20:03:14.416867 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.414306    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 20:03:14.416867 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.414309    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 20:03:14.416867 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.414311    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 20:03:14.416867 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.414314    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 20:03:14.416867 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.414316    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 20:03:14.416867 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.414318    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 20:03:14.416867 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.414322    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 20:03:14.416867 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.414324    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 20:03:14.416867 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.414327    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 20:03:14.416867 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.414329    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 20:03:14.416867 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.414332    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 20:03:14.416867 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.414334    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 20:03:14.416867 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.414337    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 20:03:14.416867 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.414340    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 20:03:14.416867 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.414342    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 20:03:14.416867 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.414345    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 20:03:14.417968 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.414347    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 20:03:14.417968 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.414350    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 20:03:14.417968 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.414352    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 20:03:14.417968 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.414355    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 20:03:14.417968 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.414357    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 20:03:14.417968 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.414359    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 20:03:14.417968 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.414362    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 20:03:14.417968 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.414365    2577 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 20:03:14.417968 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.414368    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 20:03:14.417968 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.414371    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 20:03:14.417968 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.414374    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 20:03:14.417968 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.414377    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 20:03:14.417968 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.414380    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 20:03:14.417968 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.414382    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 20:03:14.417968 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.414385    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 20:03:14.417968 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.414387    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 20:03:14.417968 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.414390    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 20:03:14.417968 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.414392    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 20:03:14.417968 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.414395    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 20:03:14.417968 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.414397    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 20:03:14.418441 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.414400    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 20:03:14.418441 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.414402    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 20:03:14.418441 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.414404    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 20:03:14.418441 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.414407    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 20:03:14.418441 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.414410    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 20:03:14.418441 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.414413    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 20:03:14.418441 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.414415    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 20:03:14.418441 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.414418    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 20:03:14.418441 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.414420    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 20:03:14.418441 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.414423    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 20:03:14.418441 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.414425    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 20:03:14.418441 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.414427    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 20:03:14.418441 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.414430    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 20:03:14.418441 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.414434    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 20:03:14.418441 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.414436    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 20:03:14.418441 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.414439    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 20:03:14.418441 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.414441    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 20:03:14.418441 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.414443    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 20:03:14.418441 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.414976    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 20:03:14.418441 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.414982    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 20:03:14.418935 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.414986    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 20:03:14.418935 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.414989    2577 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 20:03:14.418935 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.414991    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 20:03:14.418935 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.414994    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 20:03:14.418935 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.414997    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 20:03:14.418935 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.415000    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 20:03:14.418935 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.415003    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 20:03:14.418935 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.415008    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 20:03:14.418935 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.415011    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 20:03:14.418935 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.415014    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 20:03:14.418935 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.415016    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 20:03:14.418935 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.415019    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 20:03:14.418935 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.415022    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 20:03:14.418935 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.415025    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 20:03:14.418935 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.415027    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 20:03:14.418935 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.415031    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 20:03:14.418935 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.415034    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 20:03:14.418935 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.415036    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 20:03:14.418935 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.415039    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 20:03:14.419394 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.415042    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 20:03:14.419394 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.415045    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 20:03:14.419394 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.415047    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 20:03:14.419394 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.415050    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 20:03:14.419394 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.415053    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 20:03:14.419394 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.415056    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 20:03:14.419394 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.415058    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 20:03:14.419394 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.415061    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 20:03:14.419394 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.415064    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 20:03:14.419394 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.415066    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 20:03:14.419394 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.415069    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 20:03:14.419394 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.415071    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 20:03:14.419394 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.415074    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 20:03:14.419394 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.415077    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 20:03:14.419394 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.415080    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 20:03:14.419394 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.415083    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 20:03:14.419394 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.415085    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 20:03:14.419394 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.415088    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 20:03:14.419394 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.415090    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 20:03:14.419394 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.415093    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 20:03:14.419874 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.415096    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 20:03:14.419874 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.415098    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 20:03:14.419874 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.415101    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 20:03:14.419874 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.415104    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 20:03:14.419874 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.415106    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 20:03:14.419874 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.415109    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 20:03:14.419874 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.415111    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 20:03:14.419874 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.415113    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 20:03:14.419874 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.415116    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 20:03:14.419874 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.415119    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 20:03:14.419874 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.415122    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 20:03:14.419874 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.415125    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 20:03:14.419874 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.415127    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 20:03:14.419874 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.415130    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 20:03:14.419874 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.415132    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 20:03:14.419874 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.415135    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 20:03:14.419874 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.415138    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 20:03:14.419874 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.415141    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 20:03:14.419874 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.415143    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 20:03:14.419874 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.415146    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 20:03:14.420353 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.415148    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 20:03:14.420353 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.415151    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 20:03:14.420353 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.415154    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 20:03:14.420353 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.415156    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 20:03:14.420353 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.415159    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 20:03:14.420353 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.415162    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 20:03:14.420353 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.415164    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 20:03:14.420353 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.415167    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 20:03:14.420353 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.415169    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 20:03:14.420353 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.415172    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 20:03:14.420353 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.415174    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 20:03:14.420353 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.415177    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 20:03:14.420353 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.415180    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 20:03:14.420353 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.415182    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 20:03:14.420353 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.415185    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 20:03:14.420353 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.415187    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 20:03:14.420353 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.415190    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 20:03:14.420353 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.415193    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 20:03:14.420353 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.415195    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 20:03:14.420840 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.415197    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 20:03:14.420840 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.415200    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 20:03:14.420840 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.415203    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 20:03:14.420840 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.415206    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 20:03:14.420840 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.415208    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 20:03:14.420840 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.415211    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 20:03:14.420840 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.415214    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 20:03:14.420840 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.415217    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 20:03:14.420840 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.415220    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 20:03:14.420840 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.415222    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 20:03:14.420840 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.415227    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 20:03:14.420840 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.415230    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 20:03:14.420840 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.415233    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 20:03:14.420840 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.415235    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 20:03:14.420840 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.415238    2577 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 20:03:14.420840 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.415240    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 20:03:14.420840 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.415244    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 20:03:14.420840 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.415247    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 20:03:14.420840 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.415249    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 20:03:14.421282 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.415251    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 20:03:14.421282 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.415254    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 20:03:14.421282 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.415256    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 20:03:14.421282 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.415259    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 20:03:14.421282 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.415261    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 20:03:14.421282 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.415264    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 20:03:14.421282 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.415266    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 20:03:14.421282 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.416143    2577 flags.go:64] FLAG: --address="0.0.0.0"
Nov 03 20:03:14.421282 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.416154    2577 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 03 20:03:14.421282 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.416161    2577 flags.go:64] FLAG: --anonymous-auth="true"
Nov 03 20:03:14.421282 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.416166    2577 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 03 20:03:14.421282 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.416171    2577 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 03 20:03:14.421282 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.416174    2577 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 03 20:03:14.421282 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.416179    2577 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 03 20:03:14.421282 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.416184    2577 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 03 20:03:14.421282 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.416187    2577 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 03 20:03:14.421282 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.416190    2577 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 03 20:03:14.421282 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.416195    2577 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 03 20:03:14.421282 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.416198    2577 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 03 20:03:14.421282 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.416202    2577 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 03 20:03:14.421282 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.416206    2577 flags.go:64] FLAG: --cgroup-root=""
Nov 03 20:03:14.421282 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.416209    2577 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 03 20:03:14.421840 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.416212    2577 flags.go:64] FLAG: --client-ca-file=""
Nov 03 20:03:14.421840 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.416215    2577 flags.go:64] FLAG: --cloud-provider="external"
Nov 03 20:03:14.421840 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.416218    2577 flags.go:64] FLAG: --cluster-dns="[]"
Nov 03 20:03:14.421840 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.416223    2577 flags.go:64] FLAG: --cluster-domain=""
Nov 03 20:03:14.421840 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.416227    2577 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 03 20:03:14.421840 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.416230    2577 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 03 20:03:14.421840 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.416233    2577 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 03 20:03:14.421840 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.416237    2577 flags.go:64] FLAG: --container-log-max-files="5"
Nov 03 20:03:14.421840 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.416242    2577 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 03 20:03:14.421840 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.416245    2577 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 03 20:03:14.421840 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.416248    2577 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 03 20:03:14.421840 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.416251    2577 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 03 20:03:14.421840 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.416255    2577 flags.go:64] FLAG: --contention-profiling="false"
Nov 03 20:03:14.421840 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.416258    2577 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 03 20:03:14.421840 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.416261    2577 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 03 20:03:14.421840 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.416264    2577 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 03 20:03:14.421840 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.416267    2577 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 03 20:03:14.421840 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.416271    2577 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 03 20:03:14.421840 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.416274    2577 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 03 20:03:14.421840 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.416277    2577 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 03 20:03:14.421840 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.416281    2577 flags.go:64] FLAG: --enable-load-reader="false"
Nov 03 20:03:14.421840 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.416284    2577 flags.go:64] FLAG: --enable-server="true"
Nov 03 20:03:14.421840 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.416287    2577 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 03 20:03:14.421840 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.416291    2577 flags.go:64] FLAG: --event-burst="100"
Nov 03 20:03:14.421840 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.416294    2577 flags.go:64] FLAG: --event-qps="50"
Nov 03 20:03:14.422461 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.416297    2577 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 03 20:03:14.422461 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.416300    2577 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 03 20:03:14.422461 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.416303    2577 flags.go:64] FLAG: --eviction-hard=""
Nov 03 20:03:14.422461 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.416311    2577 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 03 20:03:14.422461 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.416314    2577 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 03 20:03:14.422461 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.416317    2577 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 03 20:03:14.422461 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.416322    2577 flags.go:64] FLAG: --eviction-soft=""
Nov 03 20:03:14.422461 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.416325    2577 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 03 20:03:14.422461 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.416328    2577 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 03 20:03:14.422461 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.416331    2577 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 03 20:03:14.422461 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.416334    2577 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 03 20:03:14.422461 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.416338    2577 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 03 20:03:14.422461 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.416341    2577 flags.go:64] FLAG: --fail-swap-on="true"
Nov 03 20:03:14.422461 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.416343    2577 flags.go:64] FLAG: --feature-gates=""
Nov 03 20:03:14.422461 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.416348    2577 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 03 20:03:14.422461 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.416352    2577 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 03 20:03:14.422461 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.416355    2577 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 03 20:03:14.422461 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.416359    2577 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 03 20:03:14.422461 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.416362    2577 flags.go:64] FLAG: --healthz-port="10248"
Nov 03 20:03:14.422461 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.416365    2577 flags.go:64] FLAG: --help="false"
Nov 03 20:03:14.422461 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.416368    2577 flags.go:64] FLAG: --hostname-override="ip-10-0-132-4.ec2.internal"
Nov 03 20:03:14.422461 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.416371    2577 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 03 20:03:14.422461 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.416374    2577 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 03 20:03:14.422461 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.416377    2577 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 03 20:03:14.423142 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.416380    2577 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 03 20:03:14.423142 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.416384    2577 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 03 20:03:14.423142 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.416387    2577 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 03 20:03:14.423142 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.416390    2577 flags.go:64] FLAG: --image-service-endpoint=""
Nov 03 20:03:14.423142 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.416393    2577 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 03 20:03:14.423142 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.416396    2577 flags.go:64] FLAG: --kube-api-burst="100"
Nov 03 20:03:14.423142 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.416399    2577 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 03 20:03:14.423142 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.416402    2577 flags.go:64] FLAG: --kube-api-qps="50"
Nov 03 20:03:14.423142 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.416405    2577 flags.go:64] FLAG: --kube-reserved=""
Nov 03 20:03:14.423142 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.416409    2577 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 03 20:03:14.423142 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.416412    2577 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 03 20:03:14.423142 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.416415    2577 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 03 20:03:14.423142 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.416419    2577 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 03 20:03:14.423142 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.416422    2577 flags.go:64] FLAG: --lock-file=""
Nov 03 20:03:14.423142 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.416426    2577 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 03 20:03:14.423142 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.416429    2577 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 03 20:03:14.423142 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.416432    2577 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 03 20:03:14.423142 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.416438    2577 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 03 20:03:14.423142 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.416441    2577 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 03 20:03:14.423142 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.416444    2577 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 03 20:03:14.423142 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.416447    2577 flags.go:64] FLAG: --logging-format="text"
Nov 03 20:03:14.423142 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.416450    2577 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 03 20:03:14.423142 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.416453    2577 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 03 20:03:14.423142 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.416456    2577 flags.go:64] FLAG: --manifest-url=""
Nov 03 20:03:14.424013 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.416459    2577 flags.go:64] FLAG: --manifest-url-header=""
Nov 03 20:03:14.424013 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.416463    2577 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 03 20:03:14.424013 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.416467    2577 flags.go:64] FLAG: --max-open-files="1000000"
Nov 03 20:03:14.424013 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.416470    2577 flags.go:64] FLAG: --max-pods="110"
Nov 03 20:03:14.424013 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.416474    2577 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 03 20:03:14.424013 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.416477    2577 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 03 20:03:14.424013 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.416480    2577 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 03 20:03:14.424013 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.416482    2577 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 03 20:03:14.424013 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.416485    2577 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 03 20:03:14.424013 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.416488    2577 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 03 20:03:14.424013 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.416491    2577 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 03 20:03:14.424013 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.416499    2577 flags.go:64] FLAG: --node-status-max-images="50"
Nov 03 20:03:14.424013 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.416501    2577 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 03 20:03:14.424013 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.416504    2577 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 03 20:03:14.424013 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.416508    2577 flags.go:64] FLAG: --pod-cidr=""
Nov 03 20:03:14.424013 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.416511    2577 flags.go:64] FLAG: --pod-manifest-path=""
Nov 03 20:03:14.424013 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.416514    2577 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 03 20:03:14.424013 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.416517    2577 flags.go:64] FLAG: --pods-per-core="0"
Nov 03 20:03:14.424013 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.416520    2577 flags.go:64] FLAG: --port="10250"
Nov 03 20:03:14.424013 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.416523    2577 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 03 20:03:14.424013 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.416526    2577 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-098aef875b2b4ec37"
Nov 03 20:03:14.424013 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.416530    2577 flags.go:64] FLAG: --qos-reserved=""
Nov 03 20:03:14.424013 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.416534    2577 flags.go:64] FLAG: --read-only-port="10255"
Nov 03 20:03:14.424013 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.416537    2577 flags.go:64] FLAG: --register-node="true"
Nov 03 20:03:14.424013 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.416540    2577 flags.go:64] FLAG: --register-with-taints=""
Nov 03 20:03:14.424013 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.416545    2577 flags.go:64] FLAG: --registry-burst="10"
Nov 03 20:03:14.424743 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.416548    2577 flags.go:64] FLAG: --registry-qps="5"
Nov 03 20:03:14.424743 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.416551    2577 flags.go:64] FLAG: --reserved-cpus=""
Nov 03 20:03:14.424743 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.416554    2577 flags.go:64] FLAG: --reserved-memory=""
Nov 03 20:03:14.424743 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.416558    2577 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 03 20:03:14.424743 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.416561    2577 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 03 20:03:14.424743 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.416564    2577 flags.go:64] FLAG: --rotate-certificates="false"
Nov 03 20:03:14.424743 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.416567    2577 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 03 20:03:14.424743 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.416570    2577 flags.go:64] FLAG: --runonce="false"
Nov 03 20:03:14.424743 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.416573    2577 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 03 20:03:14.424743 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.416576    2577 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 03 20:03:14.424743 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.416579    2577 flags.go:64] FLAG: --seccomp-default="false"
Nov 03 20:03:14.424743 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.416582    2577 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 03 20:03:14.424743 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.416585    2577 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 03 20:03:14.424743 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.416588    2577 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 03 20:03:14.424743 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.416592    2577 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 03 20:03:14.424743 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.416595    2577 flags.go:64] FLAG: --storage-driver-password="root"
Nov 03 20:03:14.424743 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.416598    2577 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 03 20:03:14.424743 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.416601    2577 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 03 20:03:14.424743 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.416604    2577 flags.go:64] FLAG: --storage-driver-user="root"
Nov 03 20:03:14.424743 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.416607    2577 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 03 20:03:14.424743 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.416610    2577 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 03 20:03:14.424743 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.416613    2577 flags.go:64] FLAG: --system-cgroups=""
Nov 03 20:03:14.424743 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.416616    2577 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 03 20:03:14.424743 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.416621    2577 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 03 20:03:14.424743 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.416624    2577 flags.go:64] FLAG: --tls-cert-file=""
Nov 03 20:03:14.425347 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.416627    2577 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 03 20:03:14.425347 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.416632    2577 flags.go:64] FLAG: --tls-min-version=""
Nov 03 20:03:14.425347 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.416635    2577 flags.go:64] FLAG: --tls-private-key-file=""
Nov 03 20:03:14.425347 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.416639    2577 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 03 20:03:14.425347 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.416642    2577 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 03 20:03:14.425347 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.416645    2577 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 03 20:03:14.425347 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.416648    2577 flags.go:64] FLAG: --v="2"
Nov 03 20:03:14.425347 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.416653    2577 flags.go:64] FLAG: --version="false"
Nov 03 20:03:14.425347 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.416657    2577 flags.go:64] FLAG: --vmodule=""
Nov 03 20:03:14.425347 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.416661    2577 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 03 20:03:14.425347 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.416665    2577 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 03 20:03:14.425347 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.416818    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 20:03:14.425347 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.416822    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 20:03:14.425347 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.416826    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 20:03:14.425347 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.416828    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 20:03:14.425347 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.416831    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 20:03:14.425347 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.416834    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 20:03:14.425347 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.416836    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 20:03:14.425347 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.416839    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 20:03:14.425347 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.416841    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 20:03:14.425347 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.416844    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 20:03:14.425347 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.416847    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 20:03:14.425917 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.416850    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 20:03:14.425917 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.416852    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 20:03:14.425917 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.416855    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 20:03:14.425917 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.416857    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 20:03:14.425917 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.416860    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 20:03:14.425917 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.416863    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 20:03:14.425917 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.416865    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 20:03:14.425917 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.416868    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 20:03:14.425917 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.416871    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 20:03:14.425917 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.416873    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 20:03:14.425917 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.416876    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 20:03:14.425917 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.416881    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 20:03:14.425917 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.416885    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 20:03:14.425917 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.416891    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 20:03:14.425917 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.416895    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 20:03:14.425917 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.416898    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 20:03:14.425917 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.416901    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 20:03:14.425917 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.416903    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 20:03:14.425917 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.416906    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 20:03:14.426373 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.416908    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 20:03:14.426373 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.416911    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 20:03:14.426373 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.416913    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 20:03:14.426373 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.416916    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 20:03:14.426373 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.416918    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 20:03:14.426373 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.416921    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 20:03:14.426373 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.416923    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 20:03:14.426373 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.416926    2577 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 20:03:14.426373 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.416928    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 20:03:14.426373 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.416931    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 20:03:14.426373 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.416933    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 20:03:14.426373 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.416936    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 20:03:14.426373 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.416938    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 20:03:14.426373 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.416941    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 20:03:14.426373 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.416944    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 20:03:14.426373 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.416946    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 20:03:14.426373 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.416948    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 20:03:14.426373 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.416951    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 20:03:14.426373 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.416953    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 20:03:14.426373 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.416956    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 20:03:14.426868 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.416958    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 20:03:14.426868 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.416962    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 20:03:14.426868 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.416966    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 20:03:14.426868 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.416970    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 20:03:14.426868 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.416973    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 20:03:14.426868 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.416976    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 20:03:14.426868 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.416980    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 20:03:14.426868 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.416983    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 20:03:14.426868 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.416986    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 20:03:14.426868 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.416989    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 20:03:14.426868 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.416992    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 20:03:14.426868 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.416995    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 20:03:14.426868 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.416997    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 20:03:14.426868 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.417000    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 20:03:14.426868 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.417003    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 20:03:14.426868 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.417005    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 20:03:14.426868 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.417008    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 20:03:14.426868 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.417010    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 20:03:14.426868 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.417013    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 20:03:14.426868 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.417015    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 20:03:14.427342 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.417018    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 20:03:14.427342 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.417020    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 20:03:14.427342 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.417023    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 20:03:14.427342 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.417026    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 20:03:14.427342 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.417028    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 20:03:14.427342 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.417031    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 20:03:14.427342 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.417034    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 20:03:14.427342 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.417036    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 20:03:14.427342 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.417039    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 20:03:14.427342 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.417041    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 20:03:14.427342 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.417044    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 20:03:14.427342 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.417046    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 20:03:14.427342 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.417049    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 20:03:14.427342 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.417052    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 20:03:14.427342 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.417054    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 20:03:14.427342 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.417059    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 20:03:14.427342 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.417061    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 20:03:14.427342 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.417064    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 20:03:14.427342 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.417068    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 20:03:14.427342 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.417070    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 20:03:14.427827 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.417073    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 20:03:14.427827 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.417076    2577 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 20:03:14.427827 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.417079    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 20:03:14.427827 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.417081    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 20:03:14.427827 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.417084    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 20:03:14.427827 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.417086    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 20:03:14.427827 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.417089    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 20:03:14.427827 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.417091    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 20:03:14.427827 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.417094    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 20:03:14.427827 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.417097    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 20:03:14.427827 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.417099    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 20:03:14.427827 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.417101    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 20:03:14.427827 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.417104    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 20:03:14.427827 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.417107    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 20:03:14.427827 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.417109    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 20:03:14.427827 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.417113    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 20:03:14.428208 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.417140    2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 20:03:14.428208 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.417176    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 20:03:14.428208 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.417181    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 20:03:14.428208 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.417184    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 20:03:14.428208 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.417187    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 20:03:14.428208 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.417189    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 20:03:14.428208 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.417192    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 20:03:14.428208 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.417194    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 20:03:14.428208 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.417198    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 20:03:14.428208 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.417200    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 20:03:14.428208 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.417203    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 20:03:14.428208 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.417205    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 20:03:14.428208 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.417210    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 20:03:14.428208 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.417213    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 20:03:14.428208 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.417217    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 20:03:14.428208 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.417220    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 20:03:14.428585 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.417222    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 20:03:14.428585 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.417225    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 20:03:14.428585 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.417228    2577 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 20:03:14.428585 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.417231    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 20:03:14.428585 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.417233    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 20:03:14.428585 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.417236    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 20:03:14.428585 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.417239    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 20:03:14.428585 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.417242    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 20:03:14.428585 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.417244    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 20:03:14.428585 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.417247    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 20:03:14.428585 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.417249    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 20:03:14.428585 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.417252    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 20:03:14.428585 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.417255    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 20:03:14.428585 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.417257    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 20:03:14.428585 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.417260    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 20:03:14.428585 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.417263    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 20:03:14.428585 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.417265    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 20:03:14.428585 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.417268    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 20:03:14.428585 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.417270    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 20:03:14.429121 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.417273    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 20:03:14.429121 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.417276    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 20:03:14.429121 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.417278    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 20:03:14.429121 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.417281    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 20:03:14.429121 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.417283    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 20:03:14.429121 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.417287    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 20:03:14.429121 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.417290    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 20:03:14.429121 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.417292    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 20:03:14.429121 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.417295    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 20:03:14.429121 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.417299    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 20:03:14.429121 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.417301    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 20:03:14.429121 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.417305    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 20:03:14.429121 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.417308    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 20:03:14.429121 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.417311    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 20:03:14.429121 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.417313    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 20:03:14.429121 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.417316    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 20:03:14.429121 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.417323    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 20:03:14.429121 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.417326    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 20:03:14.429121 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.417329    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 20:03:14.429574 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.417331    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 20:03:14.429574 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.417335    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 20:03:14.429574 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.417340    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 20:03:14.429574 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.417343    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 20:03:14.429574 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.417346    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 20:03:14.429574 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.417349    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 20:03:14.429574 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.417351    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 20:03:14.429574 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.417354    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 20:03:14.429574 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.417356    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 20:03:14.429574 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.417359    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 20:03:14.429574 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.417361    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 20:03:14.429574 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.417364    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 20:03:14.429574 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.417366    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 20:03:14.429574 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.417369    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 20:03:14.429574 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.417371    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 20:03:14.429574 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.417374    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 20:03:14.429574 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.417376    2577 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 20:03:14.429574 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.417379    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 20:03:14.429574 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.417381    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 20:03:14.429574 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.417384    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 20:03:14.430045 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.417387    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 20:03:14.430045 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.417389    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 20:03:14.430045 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.417394    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 20:03:14.430045 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.417397    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 20:03:14.430045 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.417401    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 20:03:14.430045 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.417404    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 20:03:14.430045 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.417406    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 20:03:14.430045 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.417409    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 20:03:14.430045 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.417412    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 20:03:14.430045 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.417414    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 20:03:14.430045 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.417419    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 20:03:14.430045 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.417422    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 20:03:14.430045 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.417425    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 20:03:14.430045 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.417428    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 20:03:14.430045 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.417430    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 20:03:14.430045 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.417433    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 20:03:14.430045 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.417435    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 20:03:14.430045 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.417438    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 20:03:14.430045 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.417440    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 20:03:14.430494 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.417443    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 20:03:14.430494 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.417445    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 20:03:14.430494 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.417448    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 20:03:14.430494 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.417450    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 20:03:14.430494 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.417453    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 20:03:14.430494 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.417455    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 20:03:14.430494 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.417458    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 20:03:14.430494 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.417461    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 20:03:14.430494 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.417463    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 20:03:14.430494 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.417466    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 20:03:14.430494 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.417469    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 20:03:14.430494 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.417471    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 20:03:14.430494 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.417474    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 20:03:14.430494 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.417478    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 20:03:14.430494 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.425111    2577 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 03 20:03:14.430494 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.425125    2577 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 03 20:03:14.430494 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.425703    2577 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 03 20:03:14.430494 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.425710    2577 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 03 20:03:14.430494 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.425783    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 20:03:14.431000 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.425790    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 20:03:14.431000 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.425794    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 20:03:14.431000 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.425797    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 20:03:14.431000 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.425800    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 20:03:14.431000 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.425803    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 20:03:14.431000 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.425806    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 20:03:14.431000 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.425809    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 20:03:14.431000 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.425811    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 20:03:14.431000 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.425814    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 20:03:14.431000 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.425817    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 20:03:14.431000 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.425819    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 20:03:14.431000 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.425822    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 20:03:14.431000 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.425825    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 20:03:14.431000 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.425828    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 20:03:14.431000 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.425831    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 20:03:14.431000 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.425834    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 20:03:14.431000 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.425837    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 20:03:14.431000 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.425840    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 20:03:14.431000 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.425843    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 20:03:14.431000 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.425846    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 20:03:14.431470 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.425848    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 20:03:14.431470 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.425851    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 20:03:14.431470 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.425859    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 20:03:14.431470 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.425862    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 20:03:14.431470 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.425864    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 20:03:14.431470 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.425867    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 20:03:14.431470 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.425870    2577 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 20:03:14.431470 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.425872    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 20:03:14.431470 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.425876    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 20:03:14.431470 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.425879    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 20:03:14.431470 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.425882    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 20:03:14.431470 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.425885    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 20:03:14.431470 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.425889    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 20:03:14.431470 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.425893    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 20:03:14.431470 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.425896    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 20:03:14.431470 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.425899    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 20:03:14.431470 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.425902    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 20:03:14.431470 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.425905    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 20:03:14.431470 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.425908    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 20:03:14.431922 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.425911    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 20:03:14.431922 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.425914    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 20:03:14.431922 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.425917    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 20:03:14.431922 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.425920    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 20:03:14.431922 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.425922    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 20:03:14.431922 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.425925    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 20:03:14.431922 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.425929    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 20:03:14.431922 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.425932    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 20:03:14.431922 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.425935    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 20:03:14.431922 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.425938    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 20:03:14.431922 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.425941    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 20:03:14.431922 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.425943    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 20:03:14.431922 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.425946    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 20:03:14.431922 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.425949    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 20:03:14.431922 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.425951    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 20:03:14.431922 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.425954    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 20:03:14.431922 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.425957    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 20:03:14.431922 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.425959    2577 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 20:03:14.431922 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.425962    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 20:03:14.431922 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.425965    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 20:03:14.432378 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.425967    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 20:03:14.432378 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.425971    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 20:03:14.432378 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.425974    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 20:03:14.432378 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.425977    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 20:03:14.432378 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.425980    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 20:03:14.432378 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.425983    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 20:03:14.432378 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.425985    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 20:03:14.432378 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.425988    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 20:03:14.432378 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.425991    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 20:03:14.432378 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.425993    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 20:03:14.432378 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.425996    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 20:03:14.432378 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.425999    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 20:03:14.432378 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426001    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 20:03:14.432378 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426004    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 20:03:14.432378 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426006    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 20:03:14.432378 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426009    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 20:03:14.432378 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426011    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 20:03:14.432378 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426014    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 20:03:14.432378 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426017    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 20:03:14.432838 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426020    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 20:03:14.432838 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426023    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 20:03:14.432838 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426025    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 20:03:14.432838 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426028    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 20:03:14.432838 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426031    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 20:03:14.432838 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426033    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 20:03:14.432838 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426036    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 20:03:14.432838 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426038    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 20:03:14.432838 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426041    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 20:03:14.432838 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426044    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 20:03:14.432838 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426046    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 20:03:14.432838 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426049    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 20:03:14.432838 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426051    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 20:03:14.432838 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426055    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 20:03:14.432838 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426058    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 20:03:14.432838 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426060    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 20:03:14.432838 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426063    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 20:03:14.432838 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426067    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 20:03:14.432838 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426072    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 20:03:14.432838 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426075    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 20:03:14.433301 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426077    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 20:03:14.433301 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426080    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 20:03:14.433301 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426083    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 20:03:14.433301 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426086    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 20:03:14.433301 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426088    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 20:03:14.433301 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426091    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 20:03:14.433301 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426093    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 20:03:14.433301 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.426123    2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 20:03:14.433301 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426259    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 20:03:14.433301 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426265    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 20:03:14.433301 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426268    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 20:03:14.433301 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426272    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 20:03:14.433301 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426275    2577 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 20:03:14.433301 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426278    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 20:03:14.433301 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426281    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 20:03:14.433301 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426284    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 20:03:14.433703 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426287    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 20:03:14.433703 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426290    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 20:03:14.433703 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426292    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 20:03:14.433703 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426295    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 20:03:14.433703 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426298    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 20:03:14.433703 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426300    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 20:03:14.433703 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426303    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 20:03:14.433703 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426306    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 20:03:14.433703 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426308    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 20:03:14.433703 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426311    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 20:03:14.433703 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426314    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 20:03:14.433703 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426317    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 20:03:14.433703 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426320    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 20:03:14.433703 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426322    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 20:03:14.433703 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426325    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 20:03:14.433703 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426328    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 20:03:14.433703 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426330    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 20:03:14.433703 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426333    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 20:03:14.433703 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426336    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 20:03:14.433703 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426338    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 20:03:14.434274 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426341    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 20:03:14.434274 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426343    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 20:03:14.434274 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426346    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 20:03:14.434274 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426349    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 20:03:14.434274 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426351    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 20:03:14.434274 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426354    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 20:03:14.434274 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426356    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 20:03:14.434274 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426359    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 20:03:14.434274 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426362    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 20:03:14.434274 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426364    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 20:03:14.434274 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426368    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 20:03:14.434274 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426371    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 20:03:14.434274 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426374    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 20:03:14.434274 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426377    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 20:03:14.434274 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426379    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 20:03:14.434274 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426382    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 20:03:14.434274 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426385    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 20:03:14.434274 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426388    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 20:03:14.434274 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426390    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 20:03:14.434274 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426393    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 20:03:14.434744 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426396    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 20:03:14.434744 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426399    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 20:03:14.434744 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426401    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 20:03:14.434744 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426404    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 20:03:14.434744 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426407    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 20:03:14.434744 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426409    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 20:03:14.434744 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426412    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 20:03:14.434744 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426415    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 20:03:14.434744 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426418    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 20:03:14.434744 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426420    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 20:03:14.434744 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426423    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 20:03:14.434744 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426426    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 20:03:14.434744 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426428    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 20:03:14.434744 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426431    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 20:03:14.434744 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426434    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 20:03:14.434744 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426436    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 20:03:14.434744 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426439    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 20:03:14.434744 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426442    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 20:03:14.434744 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426444    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 20:03:14.435197 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426447    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 20:03:14.435197 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426449    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 20:03:14.435197 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426452    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 20:03:14.435197 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426454    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 20:03:14.435197 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426458    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 20:03:14.435197 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426460    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 20:03:14.435197 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426463    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 20:03:14.435197 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426465    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 20:03:14.435197 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426469    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 20:03:14.435197 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426472    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 20:03:14.435197 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426476    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 20:03:14.435197 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426480    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 20:03:14.435197 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426483    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 20:03:14.435197 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426487    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 20:03:14.435197 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426490    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 20:03:14.435197 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426492    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 20:03:14.435197 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426495    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 20:03:14.435197 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426498    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 20:03:14.435622 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426500    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 20:03:14.435622 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426503    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 20:03:14.435622 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426506    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 20:03:14.435622 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426509    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 20:03:14.435622 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426511    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 20:03:14.435622 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426514    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 20:03:14.435622 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426516    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 20:03:14.435622 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426519    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 20:03:14.435622 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426522    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 20:03:14.435622 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426524    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 20:03:14.435622 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426526    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 20:03:14.435622 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426529    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 20:03:14.435622 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426532    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 20:03:14.435622 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426534    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 20:03:14.435622 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426537    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 20:03:14.435622 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426539    2577 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 20:03:14.435622 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426542    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 20:03:14.435622 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426544    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 20:03:14.435622 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426547    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 20:03:14.435622 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426550    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 20:03:14.436095 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426552    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 20:03:14.436095 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.426575    2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 20:03:14.436095 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426609    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 20:03:14.436095 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426613    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 20:03:14.436095 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426616    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 20:03:14.436095 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426619    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 20:03:14.436095 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426622    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 20:03:14.436095 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426626    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 20:03:14.436095 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426629    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 20:03:14.436095 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426632    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 20:03:14.436095 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426635    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 20:03:14.436095 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426638    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 20:03:14.436095 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426641    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 20:03:14.436095 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426644    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 20:03:14.436095 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426647    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 20:03:14.436450 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426650    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 20:03:14.436450 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426653    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 20:03:14.436450 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426656    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 20:03:14.436450 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426660    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 20:03:14.436450 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426664    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 20:03:14.436450 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426668    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 20:03:14.436450 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426678    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 20:03:14.436450 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426683    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 20:03:14.436450 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426685    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 20:03:14.436450 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426688    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 20:03:14.436450 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426691    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 20:03:14.436450 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426693    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 20:03:14.436450 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426696    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 20:03:14.436450 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426699    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 20:03:14.436450 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426701    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 20:03:14.436450 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426704    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 20:03:14.436450 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426709    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 20:03:14.436450 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426712    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 20:03:14.436893 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426715    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 20:03:14.436893 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426718    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 20:03:14.436893 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426720    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 20:03:14.436893 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426723    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 20:03:14.436893 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426725    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 20:03:14.436893 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426728    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 20:03:14.436893 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426731    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 20:03:14.436893 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426733    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 20:03:14.436893 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426736    2577 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 20:03:14.436893 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426739    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 20:03:14.436893 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426741    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 20:03:14.436893 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426744    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 20:03:14.436893 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426746    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 20:03:14.436893 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426749    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 20:03:14.436893 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426752    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 20:03:14.436893 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426755    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 20:03:14.436893 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426757    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 20:03:14.436893 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426760    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 20:03:14.436893 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426762    2577 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 20:03:14.436893 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426765    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 20:03:14.437403 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426767    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 20:03:14.437403 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426783    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 20:03:14.437403 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426786    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 20:03:14.437403 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426789    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 20:03:14.437403 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426792    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 20:03:14.437403 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426794    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 20:03:14.437403 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426797    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 20:03:14.437403 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426799    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 20:03:14.437403 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426802    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 20:03:14.437403 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426804    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 20:03:14.437403 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426809    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 20:03:14.437403 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426811    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 20:03:14.437403 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426815    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 20:03:14.437403 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426818    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 20:03:14.437403 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426821    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 20:03:14.437403 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426823    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 20:03:14.437403 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426826    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 20:03:14.437403 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426829    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 20:03:14.437403 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426831    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 20:03:14.437403 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426834    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 20:03:14.437998 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426836    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 20:03:14.437998 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426839    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 20:03:14.437998 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426842    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 20:03:14.437998 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426845    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 20:03:14.437998 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426847    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 20:03:14.437998 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426850    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 20:03:14.437998 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426853    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 20:03:14.437998 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426855    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 20:03:14.437998 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426858    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 20:03:14.437998 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426860    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 20:03:14.437998 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426863    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 20:03:14.437998 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426866    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 20:03:14.437998 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426869    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 20:03:14.437998 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426872    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 20:03:14.437998 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426875    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 20:03:14.437998 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426877    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 20:03:14.437998 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426880    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 20:03:14.437998 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426883    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 20:03:14.437998 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426886    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 20:03:14.437998 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426888    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 20:03:14.438485 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426891    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 20:03:14.438485 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426894    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 20:03:14.438485 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426898    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 20:03:14.438485 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426900    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 20:03:14.438485 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426903    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 20:03:14.438485 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426906    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 20:03:14.438485 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426909    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 20:03:14.438485 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426912    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 20:03:14.438485 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426914    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 20:03:14.438485 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426917    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 20:03:14.438485 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426919    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 20:03:14.438485 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426922    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 20:03:14.438485 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426926    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 20:03:14.438485 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426928    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 20:03:14.438485 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:14.426931    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 20:03:14.438485 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.426985    2577 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 03 20:03:14.438485 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.428910    2577 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Nov 03 20:03:14.438485 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.430040    2577 server.go:1014] "Starting client certificate rotation"
Nov 03 20:03:14.438930 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.430131    2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 03 20:03:14.438930 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.430166    2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 03 20:03:14.438930 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.432329    2577 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 20:03:14.438930 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.433591    2577 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 20:03:14.446215 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.446050    2577 log.go:25] "Validated CRI v1 runtime API"
Nov 03 20:03:14.449556 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.449540    2577 log.go:25] "Validated CRI v1 image API"
Nov 03 20:03:14.450300 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.450277    2577 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 03 20:03:14.453816 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.453793    2577 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 94a93814-ebbe-4de0-a302-9c3ea42c5c57:/dev/nvme0n1p4 a9c160b5-f97e-4f1d-8e22-2e39f347ad8a:/dev/nvme0n1p3]
Nov 03 20:03:14.453884 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.453814    2577 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Nov 03 20:03:14.459499 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.459379    2577 manager.go:217] Machine: {Timestamp:2026-11-03 20:03:14.457979665 +0000 UTC m=+0.318092193 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:2960641 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec28a54264ec73b825d5e6ca81be3d07 SystemUUID:ec28a542-64ec-73b8-25d5-e6ca81be3d07 BootID:f47228ac-c33d-4e1c-a9ea-7ab885c36768 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:fb:2c:5c:27 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:fb:2c:5c:27 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:16:3b:33:15:54:08 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 03 20:03:14.459916 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.459905    2577 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 03 20:03:14.460005 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.459994    2577 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 03 20:03:14.461926 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.461901    2577 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 03 20:03:14.462100 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.461938    2577 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-132-4.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 03 20:03:14.462593 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.462583    2577 topology_manager.go:143] "Creating topology manager with none policy"
Nov 03 20:03:14.462625 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.462595    2577 container_manager_linux.go:311] "Creating device plugin manager"
Nov 03 20:03:14.462625 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.462620    2577 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 20:03:14.462688 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.462636    2577 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 20:03:14.463390 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.463379    2577 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 03 20:03:14.466666 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.466650    2577 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 03 20:03:14.466835 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.466823    2577 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 03 20:03:14.467698 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.467687    2577 kubelet.go:493] "Attempting to sync node with API server"
Nov 03 20:03:14.467753 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.467709    2577 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 03 20:03:14.467753 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.467719    2577 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 03 20:03:14.467753 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.467728    2577 kubelet.go:405] "Adding apiserver pod source"
Nov 03 20:03:14.467753 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.467735    2577 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 03 20:03:14.469992 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.469798    2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 20:03:14.472339 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.472321    2577 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 03 20:03:14.475983 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.475971    2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 20:03:14.476796 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.476769    2577 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 03 20:03:14.476840 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.476814    2577 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 03 20:03:14.477363 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.477352    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 03 20:03:14.477401 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.477370    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 03 20:03:14.477401 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.477376    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 03 20:03:14.477401 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.477382    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 03 20:03:14.477401 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.477387    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 03 20:03:14.477401 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.477393    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 03 20:03:14.477872 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.477862    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 03 20:03:14.477903 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.477873    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 03 20:03:14.477903 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.477880    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 03 20:03:14.477903 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.477886    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 03 20:03:14.477903 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.477895    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 03 20:03:14.477903 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.477904    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 03 20:03:14.479034 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.479025    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 03 20:03:14.479076 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.479037    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 03 20:03:14.482874 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.482860    2577 server.go:1263] "Started kubelet"
Nov 03 20:03:14.482936 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.482911    2577 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 03 20:03:14.483053 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.482973    2577 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 03 20:03:14.483137 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.483077    2577 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 03 20:03:14.483700 ip-10-0-132-4 systemd[1]: Started Kubernetes Kubelet.
Nov 03 20:03:14.486536 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.485644    2577 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 03 20:03:14.486653 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.486639    2577 server.go:317] "Adding debug handlers to kubelet server"
Nov 03 20:03:14.488041 ip-10-0-132-4 kubenswrapper[2577]: E1103 20:03:14.488021    2577 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Nov 03 20:03:14.490959 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.490942    2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 03 20:03:14.491310 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.491295    2577 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 03 20:03:14.491668 ip-10-0-132-4 kubenswrapper[2577]: E1103 20:03:14.491647    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-4.ec2.internal\" not found"
Nov 03 20:03:14.492100 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.492084    2577 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 03 20:03:14.492183 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.492173    2577 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 03 20:03:14.492262 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.492084    2577 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 03 20:03:14.492314 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.492287    2577 factory.go:55] Registering systemd factory
Nov 03 20:03:14.492314 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.492303    2577 factory.go:223] Registration of the systemd container factory successfully
Nov 03 20:03:14.492409 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.492319    2577 reconstruct.go:97] "Volume reconstruction finished"
Nov 03 20:03:14.492409 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.492331    2577 reconciler.go:29] "Reconciler: start to sync state"
Nov 03 20:03:14.492577 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.492562    2577 factory.go:153] Registering CRI-O factory
Nov 03 20:03:14.492618 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.492582    2577 factory.go:223] Registration of the crio container factory successfully
Nov 03 20:03:14.492668 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.492647    2577 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 03 20:03:14.492711 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.492694    2577 factory.go:103] Registering Raw factory
Nov 03 20:03:14.492746 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.492711    2577 manager.go:1199] Started watching for new ooms in manager
Nov 03 20:03:14.493691 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.493677    2577 manager.go:319] Starting recovery of all containers
Nov 03 20:03:14.504149 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.504130    2577 manager.go:324] Recovery completed
Nov 03 20:03:14.510433 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.510416    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 20:03:14.512531 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.512184    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-4.ec2.internal" event="NodeHasSufficientMemory"
Nov 03 20:03:14.512531 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.512233    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-4.ec2.internal" event="NodeHasNoDiskPressure"
Nov 03 20:03:14.512531 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.512247    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-4.ec2.internal" event="NodeHasSufficientPID"
Nov 03 20:03:14.512531 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.512255    2577 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 03 20:03:14.512531 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.512336    2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-hq5vj"
Nov 03 20:03:14.512814 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.512801    2577 cpu_manager.go:226] "Starting" policy="none"
Nov 03 20:03:14.512870 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.512814    2577 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 03 20:03:14.512870 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.512835    2577 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 03 20:03:14.512870 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.512849    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 20:03:14.513566 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.513553    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-4.ec2.internal" event="NodeHasSufficientMemory"
Nov 03 20:03:14.513634 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.513583    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-4.ec2.internal" event="NodeHasNoDiskPressure"
Nov 03 20:03:14.513634 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.513597    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-4.ec2.internal" event="NodeHasSufficientPID"
Nov 03 20:03:14.513930 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.513917    2577 policy_none.go:50] "Start"
Nov 03 20:03:14.513977 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.513936    2577 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 03 20:03:14.514005 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.513985    2577 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 03 20:03:14.519661 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.519646    2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-hq5vj"
Nov 03 20:03:14.522307 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.522285    2577 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-4.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Nov 03 20:03:14.523202 ip-10-0-132-4 kubenswrapper[2577]: E1103 20:03:14.522275    2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-4.ec2.internal.18e4a190b4e8f44c  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-4.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-132-4.ec2.internal,},FirstTimestamp:2026-11-03 20:03:14.482828364 +0000 UTC m=+0.342940891,LastTimestamp:2026-11-03 20:03:14.482828364 +0000 UTC m=+0.342940891,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-4.ec2.internal,}"
Nov 03 20:03:14.535555 ip-10-0-132-4 kubenswrapper[2577]: E1103 20:03:14.535528    2577 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-4.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Nov 03 20:03:14.536259 ip-10-0-132-4 kubenswrapper[2577]: E1103 20:03:14.536159    2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-4.ec2.internal.18e4a190b6a95b80  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-4.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-132-4.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-132-4.ec2.internal,},FirstTimestamp:2026-11-03 20:03:14.512214912 +0000 UTC m=+0.372327439,LastTimestamp:2026-11-03 20:03:14.512214912 +0000 UTC m=+0.372327439,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-4.ec2.internal,}"
Nov 03 20:03:14.536807 ip-10-0-132-4 kubenswrapper[2577]: E1103 20:03:14.536769    2577 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-132-4.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Nov 03 20:03:14.536865 ip-10-0-132-4 kubenswrapper[2577]: E1103 20:03:14.536768    2577 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Nov 03 20:03:14.536865 ip-10-0-132-4 kubenswrapper[2577]: E1103 20:03:14.536768    2577 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Nov 03 20:03:14.557629 ip-10-0-132-4 kubenswrapper[2577]: E1103 20:03:14.557554    2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-4.ec2.internal.18e4a190b6a9b9e5  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-4.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-132-4.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-132-4.ec2.internal,},FirstTimestamp:2026-11-03 20:03:14.512239077 +0000 UTC m=+0.372351605,LastTimestamp:2026-11-03 20:03:14.512239077 +0000 UTC m=+0.372351605,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-4.ec2.internal,}"
Nov 03 20:03:14.573650 ip-10-0-132-4 kubenswrapper[2577]: E1103 20:03:14.573554    2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-4.ec2.internal.18e4a190b6a9ea8e  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-4.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-132-4.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-132-4.ec2.internal,},FirstTimestamp:2026-11-03 20:03:14.512251534 +0000 UTC m=+0.372364061,LastTimestamp:2026-11-03 20:03:14.512251534 +0000 UTC m=+0.372364061,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-4.ec2.internal,}"
Nov 03 20:03:14.657136 ip-10-0-132-4 kubenswrapper[2577]: E1103 20:03:14.592624    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-4.ec2.internal\" not found"
Nov 03 20:03:14.657136 ip-10-0-132-4 kubenswrapper[2577]: E1103 20:03:14.598309    2577 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-132-4.ec2.internal.18e4a190b6a95b80\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-4.ec2.internal.18e4a190b6a95b80  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-4.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-132-4.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-132-4.ec2.internal,},FirstTimestamp:2026-11-03 20:03:14.512214912 +0000 UTC m=+0.372327439,LastTimestamp:2026-11-03 20:03:14.513565716 +0000 UTC m=+0.373678250,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-4.ec2.internal,}"
Nov 03 20:03:14.657136 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.613684    2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 03 20:03:14.657136 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.614808    2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 03 20:03:14.657136 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.614829    2577 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 03 20:03:14.657136 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.614848    2577 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 03 20:03:14.657136 ip-10-0-132-4 kubenswrapper[2577]: E1103 20:03:14.614880    2577 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Nov 03 20:03:14.657136 ip-10-0-132-4 kubenswrapper[2577]: E1103 20:03:14.621382    2577 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-132-4.ec2.internal.18e4a190b6a9b9e5\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-4.ec2.internal.18e4a190b6a9b9e5  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-4.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-132-4.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-132-4.ec2.internal,},FirstTimestamp:2026-11-03 20:03:14.512239077 +0000 UTC m=+0.372351605,LastTimestamp:2026-11-03 20:03:14.513589416 +0000 UTC m=+0.373701945,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-4.ec2.internal,}"
Nov 03 20:03:14.657136 ip-10-0-132-4 kubenswrapper[2577]: E1103 20:03:14.640931    2577 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-132-4.ec2.internal.18e4a190b6a9ea8e\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-4.ec2.internal.18e4a190b6a9ea8e  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-4.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-132-4.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-132-4.ec2.internal,},FirstTimestamp:2026-11-03 20:03:14.512251534 +0000 UTC m=+0.372364061,LastTimestamp:2026-11-03 20:03:14.513603463 +0000 UTC m=+0.373715994,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-4.ec2.internal,}"
Nov 03 20:03:14.657136 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.654444    2577 policy_none.go:44] "Start"
Nov 03 20:03:14.662438 ip-10-0-132-4 kubenswrapper[2577]: E1103 20:03:14.662415    2577 reflector.go:204] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass"
Nov 03 20:03:14.693116 ip-10-0-132-4 kubenswrapper[2577]: E1103 20:03:14.693093    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-4.ec2.internal\" not found"
Nov 03 20:03:14.696349 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.696321    2577 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 03 20:03:14.698616 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.698581    2577 manager.go:341] "Starting Device Plugin manager"
Nov 03 20:03:14.699489 ip-10-0-132-4 kubenswrapper[2577]: E1103 20:03:14.699472    2577 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 03 20:03:14.699489 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.699486    2577 server.go:87] "Starting device plugin registration server"
Nov 03 20:03:14.699832 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.699808    2577 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 03 20:03:14.699832 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.699820    2577 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 03 20:03:14.700000 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.699934    2577 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 03 20:03:14.700051 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.700017    2577 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 03 20:03:14.700051 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.700027    2577 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 03 20:03:14.700497 ip-10-0-132-4 kubenswrapper[2577]: E1103 20:03:14.700480    2577 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 03 20:03:14.700558 ip-10-0-132-4 kubenswrapper[2577]: E1103 20:03:14.700530    2577 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-4.ec2.internal\" not found"
Nov 03 20:03:14.715485 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.715462    2577 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-4.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-132-4.ec2.internal"]
Nov 03 20:03:14.715589 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.715523    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 20:03:14.716399 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.716384    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-4.ec2.internal" event="NodeHasSufficientMemory"
Nov 03 20:03:14.716490 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.716414    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-4.ec2.internal" event="NodeHasNoDiskPressure"
Nov 03 20:03:14.716490 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.716427    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-4.ec2.internal" event="NodeHasSufficientPID"
Nov 03 20:03:14.717748 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.717733    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 20:03:14.717919 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.717904    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-4.ec2.internal"
Nov 03 20:03:14.717966 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.717936    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 20:03:14.718457 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.718440    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-4.ec2.internal" event="NodeHasSufficientMemory"
Nov 03 20:03:14.718537 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.718462    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-4.ec2.internal" event="NodeHasNoDiskPressure"
Nov 03 20:03:14.718537 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.718471    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-4.ec2.internal" event="NodeHasSufficientPID"
Nov 03 20:03:14.718537 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.718518    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-4.ec2.internal" event="NodeHasSufficientMemory"
Nov 03 20:03:14.718537 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.718538    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-4.ec2.internal" event="NodeHasNoDiskPressure"
Nov 03 20:03:14.718663 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.718553    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-4.ec2.internal" event="NodeHasSufficientPID"
Nov 03 20:03:14.719617 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.719599    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-4.ec2.internal"
Nov 03 20:03:14.719694 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.719631    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 20:03:14.720263 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.720248    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-4.ec2.internal" event="NodeHasSufficientMemory"
Nov 03 20:03:14.720333 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.720271    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-4.ec2.internal" event="NodeHasNoDiskPressure"
Nov 03 20:03:14.720333 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.720280    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-4.ec2.internal" event="NodeHasSufficientPID"
Nov 03 20:03:14.752334 ip-10-0-132-4 kubenswrapper[2577]: E1103 20:03:14.752310    2577 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-4.ec2.internal\" not found" node="ip-10-0-132-4.ec2.internal"
Nov 03 20:03:14.756034 ip-10-0-132-4 kubenswrapper[2577]: E1103 20:03:14.756015    2577 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-4.ec2.internal\" not found" node="ip-10-0-132-4.ec2.internal"
Nov 03 20:03:14.767351 ip-10-0-132-4 kubenswrapper[2577]: E1103 20:03:14.767334    2577 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-132-4.ec2.internal\" not found" node="ip-10-0-132-4.ec2.internal"
Nov 03 20:03:14.793947 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.793924    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a66c9af704a53c4c9b04653e0d8d8d8f-config\") pod \"kube-apiserver-proxy-ip-10-0-132-4.ec2.internal\" (UID: \"a66c9af704a53c4c9b04653e0d8d8d8f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-4.ec2.internal"
Nov 03 20:03:14.794037 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.793951    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5dff8c1e3d20edc13c5947130e8a5c14-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-4.ec2.internal\" (UID: \"5dff8c1e3d20edc13c5947130e8a5c14\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-4.ec2.internal"
Nov 03 20:03:14.794037 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.793968    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5dff8c1e3d20edc13c5947130e8a5c14-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-4.ec2.internal\" (UID: \"5dff8c1e3d20edc13c5947130e8a5c14\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-4.ec2.internal"
Nov 03 20:03:14.800047 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.800032    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 20:03:14.800995 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.800980    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-4.ec2.internal" event="NodeHasSufficientMemory"
Nov 03 20:03:14.801075 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.801008    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-4.ec2.internal" event="NodeHasNoDiskPressure"
Nov 03 20:03:14.801075 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.801026    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-4.ec2.internal" event="NodeHasSufficientPID"
Nov 03 20:03:14.801075 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.801053    2577 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-132-4.ec2.internal"
Nov 03 20:03:14.822968 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.822949    2577 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-132-4.ec2.internal"
Nov 03 20:03:14.823030 ip-10-0-132-4 kubenswrapper[2577]: E1103 20:03:14.822974    2577 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-132-4.ec2.internal\": node \"ip-10-0-132-4.ec2.internal\" not found"
Nov 03 20:03:14.871955 ip-10-0-132-4 kubenswrapper[2577]: E1103 20:03:14.871928    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-4.ec2.internal\" not found"
Nov 03 20:03:14.894088 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.894060    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5dff8c1e3d20edc13c5947130e8a5c14-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-4.ec2.internal\" (UID: \"5dff8c1e3d20edc13c5947130e8a5c14\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-4.ec2.internal"
Nov 03 20:03:14.894168 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.894099    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5dff8c1e3d20edc13c5947130e8a5c14-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-4.ec2.internal\" (UID: \"5dff8c1e3d20edc13c5947130e8a5c14\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-4.ec2.internal"
Nov 03 20:03:14.894168 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.894124    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a66c9af704a53c4c9b04653e0d8d8d8f-config\") pod \"kube-apiserver-proxy-ip-10-0-132-4.ec2.internal\" (UID: \"a66c9af704a53c4c9b04653e0d8d8d8f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-4.ec2.internal"
Nov 03 20:03:14.894168 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.894158    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a66c9af704a53c4c9b04653e0d8d8d8f-config\") pod \"kube-apiserver-proxy-ip-10-0-132-4.ec2.internal\" (UID: \"a66c9af704a53c4c9b04653e0d8d8d8f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-4.ec2.internal"
Nov 03 20:03:14.894262 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.894162    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5dff8c1e3d20edc13c5947130e8a5c14-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-4.ec2.internal\" (UID: \"5dff8c1e3d20edc13c5947130e8a5c14\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-4.ec2.internal"
Nov 03 20:03:14.894262 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:14.894166    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5dff8c1e3d20edc13c5947130e8a5c14-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-4.ec2.internal\" (UID: \"5dff8c1e3d20edc13c5947130e8a5c14\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-4.ec2.internal"
Nov 03 20:03:14.972762 ip-10-0-132-4 kubenswrapper[2577]: E1103 20:03:14.972710    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-4.ec2.internal\" not found"
Nov 03 20:03:15.055885 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:15.055852    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-4.ec2.internal"
Nov 03 20:03:15.058429 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:15.058412    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-4.ec2.internal"
Nov 03 20:03:15.072855 ip-10-0-132-4 kubenswrapper[2577]: E1103 20:03:15.072833    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-4.ec2.internal\" not found"
Nov 03 20:03:15.173884 ip-10-0-132-4 kubenswrapper[2577]: E1103 20:03:15.173830    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-4.ec2.internal\" not found"
Nov 03 20:03:15.274735 ip-10-0-132-4 kubenswrapper[2577]: E1103 20:03:15.274685    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-4.ec2.internal\" not found"
Nov 03 20:03:15.375419 ip-10-0-132-4 kubenswrapper[2577]: E1103 20:03:15.375381    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-4.ec2.internal\" not found"
Nov 03 20:03:15.430011 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:15.429982    2577 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Nov 03 20:03:15.475939 ip-10-0-132-4 kubenswrapper[2577]: E1103 20:03:15.475906    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-4.ec2.internal\" not found"
Nov 03 20:03:15.521095 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:15.521017    2577 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 20:03:15.548538 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:15.548310    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5dff8c1e3d20edc13c5947130e8a5c14.slice/crio-7e5ed4938ac5a455602f5935787f510257d22cfd87e815829fbc27ba58c1a45f WatchSource:0}: Error finding container 7e5ed4938ac5a455602f5935787f510257d22cfd87e815829fbc27ba58c1a45f: Status 404 returned error can't find the container with id 7e5ed4938ac5a455602f5935787f510257d22cfd87e815829fbc27ba58c1a45f
Nov 03 20:03:15.548774 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:15.548754    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda66c9af704a53c4c9b04653e0d8d8d8f.slice/crio-b7d5a4479f6b4cfc2d84e97abf7442f621815298db4681aa2e43df8ccb9ec0da WatchSource:0}: Error finding container b7d5a4479f6b4cfc2d84e97abf7442f621815298db4681aa2e43df8ccb9ec0da: Status 404 returned error can't find the container with id b7d5a4479f6b4cfc2d84e97abf7442f621815298db4681aa2e43df8ccb9ec0da
Nov 03 20:03:15.553566 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:15.553549    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 20:03:15.592615 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:15.592581    2577 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-4.ec2.internal"
Nov 03 20:03:15.613989 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:15.613964    2577 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 20:03:15.615985 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:15.615969    2577 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-4.ec2.internal"
Nov 03 20:03:15.618499 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:15.618456    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-4.ec2.internal" event={"ID":"5dff8c1e3d20edc13c5947130e8a5c14","Type":"ContainerStarted","Data":"7e5ed4938ac5a455602f5935787f510257d22cfd87e815829fbc27ba58c1a45f"}
Nov 03 20:03:15.619531 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:15.619508    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-4.ec2.internal" event={"ID":"a66c9af704a53c4c9b04653e0d8d8d8f","Type":"ContainerStarted","Data":"b7d5a4479f6b4cfc2d84e97abf7442f621815298db4681aa2e43df8ccb9ec0da"}
Nov 03 20:03:15.636261 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:15.636240    2577 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 20:03:15.658026 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:15.657982    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 19:58:14 +0000 UTC" deadline="2028-04-12 18:53:48.938890536 +0000 UTC"
Nov 03 20:03:15.658026 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:15.658023    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12622h50m33.280870863s"
Nov 03 20:03:15.698377 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:15.698350    2577 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 20:03:15.806932 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:15.806839    2577 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 20:03:16.014667 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.014639    2577 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 20:03:16.470115 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.470084    2577 apiserver.go:52] "Watching apiserver"
Nov 03 20:03:16.492331 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.492299    2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Nov 03 20:03:16.507966 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.507933    2577 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 03 20:03:16.508447 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.508420    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-zzt8t","openshift-multus/multus-additional-cni-plugins-mhtmw","kube-system/konnectivity-agent-5p48g","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9f5s5","openshift-network-diagnostics/network-check-target-6dfr5","kube-system/kube-apiserver-proxy-ip-10-0-132-4.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-4.ec2.internal","openshift-multus/multus-h6p95","openshift-multus/network-metrics-daemon-c947w","openshift-network-operator/iptables-alerter-9s4wk","openshift-ovn-kubernetes/ovnkube-node-6sv7j","kube-system/global-pull-secret-syncer-j4jzv","openshift-cluster-node-tuning-operator/tuned-4dmkq","openshift-dns/node-resolver-t6dqw"]
Nov 03 20:03:16.511507 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.511472    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-zzt8t"
Nov 03 20:03:16.514083 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.514061    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-mhtmw"
Nov 03 20:03:16.516719 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.516473    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-5p48g"
Nov 03 20:03:16.518974 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.518953    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9f5s5"
Nov 03 20:03:16.521937 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.521621    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6dfr5"
Nov 03 20:03:16.521937 ip-10-0-132-4 kubenswrapper[2577]: E1103 20:03:16.521686    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6dfr5" podUID="d71eaa2a-9659-4827-9dfc-c1fce02565cb"
Nov 03 20:03:16.524165 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.524143    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-h6p95"
Nov 03 20:03:16.526675 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.526072    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 03 20:03:16.526675 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.526184    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-kqzwq\""
Nov 03 20:03:16.526675 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.526229    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 03 20:03:16.526675 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.526326    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 03 20:03:16.527766 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.527742    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 03 20:03:16.527934 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.527840    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 03 20:03:16.528027 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.527988    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 03 20:03:16.528027 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.527747    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-kzs97\""
Nov 03 20:03:16.528027 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.528019    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 03 20:03:16.528027 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.527899    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 03 20:03:16.528236 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.528124    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c947w"
Nov 03 20:03:16.528236 ip-10-0-132-4 kubenswrapper[2577]: E1103 20:03:16.528203    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c947w" podUID="4c92a5d0-b34f-46a1-8f8f-a9ae3cb8d5a5"
Nov 03 20:03:16.528998 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.528977    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-b4qhs\""
Nov 03 20:03:16.530859 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.530668    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 03 20:03:16.530859 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.530694    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 03 20:03:16.532142 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.532105    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 03 20:03:16.532614 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.532592    2577 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 03 20:03:16.532706 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.532670    2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-wmht8"
Nov 03 20:03:16.533561 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.533543    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-9s4wk"
Nov 03 20:03:16.536322 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.536303    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-6sv7j"
Nov 03 20:03:16.539738 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.539709    2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-wmht8"
Nov 03 20:03:16.540083 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.540067    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j4jzv"
Nov 03 20:03:16.540156 ip-10-0-132-4 kubenswrapper[2577]: E1103 20:03:16.540126    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j4jzv" podUID="51dffd76-5c30-440c-85ed-d81221663833"
Nov 03 20:03:16.544108 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.543855    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-4dmkq"
Nov 03 20:03:16.546339 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.546321    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-t6dqw"
Nov 03 20:03:16.593396 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.593373    2577 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 03 20:03:16.604212 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.604182    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9126a982-59c9-4cec-a665-5f1bc0ae3389-host-kubelet\") pod \"ovnkube-node-6sv7j\" (UID: \"9126a982-59c9-4cec-a665-5f1bc0ae3389\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sv7j"
Nov 03 20:03:16.604358 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.604224    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9126a982-59c9-4cec-a665-5f1bc0ae3389-env-overrides\") pod \"ovnkube-node-6sv7j\" (UID: \"9126a982-59c9-4cec-a665-5f1bc0ae3389\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sv7j"
Nov 03 20:03:16.604358 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.604272    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/51dffd76-5c30-440c-85ed-d81221663833-dbus\") pod \"global-pull-secret-syncer-j4jzv\" (UID: \"51dffd76-5c30-440c-85ed-d81221663833\") " pod="kube-system/global-pull-secret-syncer-j4jzv"
Nov 03 20:03:16.604358 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.604324    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/65008653-586d-4a76-a53f-7efe1f6d839b-registration-dir\") pod \"aws-ebs-csi-driver-node-9f5s5\" (UID: \"65008653-586d-4a76-a53f-7efe1f6d839b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9f5s5"
Nov 03 20:03:16.604358 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.604354    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1d4ce60c-e058-4c29-99e8-5f4f6eead744-serviceca\") pod \"node-ca-zzt8t\" (UID: \"1d4ce60c-e058-4c29-99e8-5f4f6eead744\") " pod="openshift-image-registry/node-ca-zzt8t"
Nov 03 20:03:16.604540 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.604406    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcxdr\" (UniqueName: \"kubernetes.io/projected/d71eaa2a-9659-4827-9dfc-c1fce02565cb-kube-api-access-zcxdr\") pod \"network-check-target-6dfr5\" (UID: \"d71eaa2a-9659-4827-9dfc-c1fce02565cb\") " pod="openshift-network-diagnostics/network-check-target-6dfr5"
Nov 03 20:03:16.604540 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.604432    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9126a982-59c9-4cec-a665-5f1bc0ae3389-host-cni-netd\") pod \"ovnkube-node-6sv7j\" (UID: \"9126a982-59c9-4cec-a665-5f1bc0ae3389\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sv7j"
Nov 03 20:03:16.604540 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.604458    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f21cf70e-682f-490c-bda5-3954df150487-etc-sysctl-conf\") pod \"tuned-4dmkq\" (UID: \"f21cf70e-682f-490c-bda5-3954df150487\") " pod="openshift-cluster-node-tuning-operator/tuned-4dmkq"
Nov 03 20:03:16.604540 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.604510    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/682463ee-d9c9-4e9b-82b7-f1116898debf-multus-daemon-config\") pod \"multus-h6p95\" (UID: \"682463ee-d9c9-4e9b-82b7-f1116898debf\") " pod="openshift-multus/multus-h6p95"
Nov 03 20:03:16.604540 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.604527    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1d4ce60c-e058-4c29-99e8-5f4f6eead744-host\") pod \"node-ca-zzt8t\" (UID: \"1d4ce60c-e058-4c29-99e8-5f4f6eead744\") " pod="openshift-image-registry/node-ca-zzt8t"
Nov 03 20:03:16.604726 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.604554    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/15b5070f-d0d3-4ad8-aa1d-93cc7b96003a-konnectivity-ca\") pod \"konnectivity-agent-5p48g\" (UID: \"15b5070f-d0d3-4ad8-aa1d-93cc7b96003a\") " pod="kube-system/konnectivity-agent-5p48g"
Nov 03 20:03:16.604726 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.604571    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9126a982-59c9-4cec-a665-5f1bc0ae3389-log-socket\") pod \"ovnkube-node-6sv7j\" (UID: \"9126a982-59c9-4cec-a665-5f1bc0ae3389\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sv7j"
Nov 03 20:03:16.604726 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.604586    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/65008653-586d-4a76-a53f-7efe1f6d839b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9f5s5\" (UID: \"65008653-586d-4a76-a53f-7efe1f6d839b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9f5s5"
Nov 03 20:03:16.604726 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.604603    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/682463ee-d9c9-4e9b-82b7-f1116898debf-host-run-multus-certs\") pod \"multus-h6p95\" (UID: \"682463ee-d9c9-4e9b-82b7-f1116898debf\") " pod="openshift-multus/multus-h6p95"
Nov 03 20:03:16.604726 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.604644    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59sqq\" (UniqueName: \"kubernetes.io/projected/f21cf70e-682f-490c-bda5-3954df150487-kube-api-access-59sqq\") pod \"tuned-4dmkq\" (UID: \"f21cf70e-682f-490c-bda5-3954df150487\") " pod="openshift-cluster-node-tuning-operator/tuned-4dmkq"
Nov 03 20:03:16.604726 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.604661    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/15b5070f-d0d3-4ad8-aa1d-93cc7b96003a-agent-certs\") pod \"konnectivity-agent-5p48g\" (UID: \"15b5070f-d0d3-4ad8-aa1d-93cc7b96003a\") " pod="kube-system/konnectivity-agent-5p48g"
Nov 03 20:03:16.604726 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.604682    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/db895326-506d-479d-a526-6bf71be7f967-system-cni-dir\") pod \"multus-additional-cni-plugins-mhtmw\" (UID: \"db895326-506d-479d-a526-6bf71be7f967\") " pod="openshift-multus/multus-additional-cni-plugins-mhtmw"
Nov 03 20:03:16.604726 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.604697    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f21cf70e-682f-490c-bda5-3954df150487-run\") pod \"tuned-4dmkq\" (UID: \"f21cf70e-682f-490c-bda5-3954df150487\") " pod="openshift-cluster-node-tuning-operator/tuned-4dmkq"
Nov 03 20:03:16.605050 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.604716    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/682463ee-d9c9-4e9b-82b7-f1116898debf-system-cni-dir\") pod \"multus-h6p95\" (UID: \"682463ee-d9c9-4e9b-82b7-f1116898debf\") " pod="openshift-multus/multus-h6p95"
Nov 03 20:03:16.605050 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.604758    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/51dffd76-5c30-440c-85ed-d81221663833-original-pull-secret\") pod \"global-pull-secret-syncer-j4jzv\" (UID: \"51dffd76-5c30-440c-85ed-d81221663833\") " pod="kube-system/global-pull-secret-syncer-j4jzv"
Nov 03 20:03:16.605050 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.604774    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f21cf70e-682f-490c-bda5-3954df150487-tmp\") pod \"tuned-4dmkq\" (UID: \"f21cf70e-682f-490c-bda5-3954df150487\") " pod="openshift-cluster-node-tuning-operator/tuned-4dmkq"
Nov 03 20:03:16.605050 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.604790    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/682463ee-d9c9-4e9b-82b7-f1116898debf-hostroot\") pod \"multus-h6p95\" (UID: \"682463ee-d9c9-4e9b-82b7-f1116898debf\") " pod="openshift-multus/multus-h6p95"
Nov 03 20:03:16.605050 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.604808    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2073a8c8-6f07-4817-b7b3-468b327e2712-host-slash\") pod \"iptables-alerter-9s4wk\" (UID: \"2073a8c8-6f07-4817-b7b3-468b327e2712\") " pod="openshift-network-operator/iptables-alerter-9s4wk"
Nov 03 20:03:16.605050 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.604884    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9126a982-59c9-4cec-a665-5f1bc0ae3389-etc-openvswitch\") pod \"ovnkube-node-6sv7j\" (UID: \"9126a982-59c9-4cec-a665-5f1bc0ae3389\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sv7j"
Nov 03 20:03:16.605050 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.604903    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9126a982-59c9-4cec-a665-5f1bc0ae3389-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6sv7j\" (UID: \"9126a982-59c9-4cec-a665-5f1bc0ae3389\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sv7j"
Nov 03 20:03:16.605050 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.604920    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f21cf70e-682f-490c-bda5-3954df150487-var-lib-kubelet\") pod \"tuned-4dmkq\" (UID: \"f21cf70e-682f-490c-bda5-3954df150487\") " pod="openshift-cluster-node-tuning-operator/tuned-4dmkq"
Nov 03 20:03:16.605050 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.604936    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/65008653-586d-4a76-a53f-7efe1f6d839b-etc-selinux\") pod \"aws-ebs-csi-driver-node-9f5s5\" (UID: \"65008653-586d-4a76-a53f-7efe1f6d839b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9f5s5"
Nov 03 20:03:16.605050 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.604985    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8psbp\" (UniqueName: \"kubernetes.io/projected/65008653-586d-4a76-a53f-7efe1f6d839b-kube-api-access-8psbp\") pod \"aws-ebs-csi-driver-node-9f5s5\" (UID: \"65008653-586d-4a76-a53f-7efe1f6d839b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9f5s5"
Nov 03 20:03:16.605050 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.605048    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/682463ee-d9c9-4e9b-82b7-f1116898debf-multus-cni-dir\") pod \"multus-h6p95\" (UID: \"682463ee-d9c9-4e9b-82b7-f1116898debf\") " pod="openshift-multus/multus-h6p95"
Nov 03 20:03:16.605503 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.605089    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9126a982-59c9-4cec-a665-5f1bc0ae3389-ovnkube-script-lib\") pod \"ovnkube-node-6sv7j\" (UID: \"9126a982-59c9-4cec-a665-5f1bc0ae3389\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sv7j"
Nov 03 20:03:16.605503 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.605137    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/682463ee-d9c9-4e9b-82b7-f1116898debf-host-var-lib-cni-bin\") pod \"multus-h6p95\" (UID: \"682463ee-d9c9-4e9b-82b7-f1116898debf\") " pod="openshift-multus/multus-h6p95"
Nov 03 20:03:16.605503 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.605305    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/716cc9cf-d26a-481b-8114-169c6104ce56-tmp-dir\") pod \"node-resolver-t6dqw\" (UID: \"716cc9cf-d26a-481b-8114-169c6104ce56\") " pod="openshift-dns/node-resolver-t6dqw"
Nov 03 20:03:16.605503 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.605384    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrdtp\" (UniqueName: \"kubernetes.io/projected/716cc9cf-d26a-481b-8114-169c6104ce56-kube-api-access-vrdtp\") pod \"node-resolver-t6dqw\" (UID: \"716cc9cf-d26a-481b-8114-169c6104ce56\") " pod="openshift-dns/node-resolver-t6dqw"
Nov 03 20:03:16.605503 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.605419    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9126a982-59c9-4cec-a665-5f1bc0ae3389-run-openvswitch\") pod \"ovnkube-node-6sv7j\" (UID: \"9126a982-59c9-4cec-a665-5f1bc0ae3389\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sv7j"
Nov 03 20:03:16.605503 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.605468    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9126a982-59c9-4cec-a665-5f1bc0ae3389-run-systemd\") pod \"ovnkube-node-6sv7j\" (UID: \"9126a982-59c9-4cec-a665-5f1bc0ae3389\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sv7j"
Nov 03 20:03:16.605698 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.605524    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f21cf70e-682f-490c-bda5-3954df150487-etc-sysctl-d\") pod \"tuned-4dmkq\" (UID: \"f21cf70e-682f-490c-bda5-3954df150487\") " pod="openshift-cluster-node-tuning-operator/tuned-4dmkq"
Nov 03 20:03:16.605698 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.605566    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f21cf70e-682f-490c-bda5-3954df150487-host\") pod \"tuned-4dmkq\" (UID: \"f21cf70e-682f-490c-bda5-3954df150487\") " pod="openshift-cluster-node-tuning-operator/tuned-4dmkq"
Nov 03 20:03:16.605698 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.605608    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/65008653-586d-4a76-a53f-7efe1f6d839b-device-dir\") pod \"aws-ebs-csi-driver-node-9f5s5\" (UID: \"65008653-586d-4a76-a53f-7efe1f6d839b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9f5s5"
Nov 03 20:03:16.605698 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.605650    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nd2w\" (UniqueName: \"kubernetes.io/projected/4c92a5d0-b34f-46a1-8f8f-a9ae3cb8d5a5-kube-api-access-6nd2w\") pod \"network-metrics-daemon-c947w\" (UID: \"4c92a5d0-b34f-46a1-8f8f-a9ae3cb8d5a5\") " pod="openshift-multus/network-metrics-daemon-c947w"
Nov 03 20:03:16.606014 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.605696    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9126a982-59c9-4cec-a665-5f1bc0ae3389-host-run-ovn-kubernetes\") pod \"ovnkube-node-6sv7j\" (UID: \"9126a982-59c9-4cec-a665-5f1bc0ae3389\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sv7j"
Nov 03 20:03:16.606014 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.605741    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9126a982-59c9-4cec-a665-5f1bc0ae3389-ovnkube-config\") pod \"ovnkube-node-6sv7j\" (UID: \"9126a982-59c9-4cec-a665-5f1bc0ae3389\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sv7j"
Nov 03 20:03:16.606014 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.605766    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f21cf70e-682f-490c-bda5-3954df150487-lib-modules\") pod \"tuned-4dmkq\" (UID: \"f21cf70e-682f-490c-bda5-3954df150487\") " pod="openshift-cluster-node-tuning-operator/tuned-4dmkq"
Nov 03 20:03:16.606014 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.605804    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/65008653-586d-4a76-a53f-7efe1f6d839b-socket-dir\") pod \"aws-ebs-csi-driver-node-9f5s5\" (UID: \"65008653-586d-4a76-a53f-7efe1f6d839b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9f5s5"
Nov 03 20:03:16.606014 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.605833    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/682463ee-d9c9-4e9b-82b7-f1116898debf-cnibin\") pod \"multus-h6p95\" (UID: \"682463ee-d9c9-4e9b-82b7-f1116898debf\") " pod="openshift-multus/multus-h6p95"
Nov 03 20:03:16.606014 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.605873    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/682463ee-d9c9-4e9b-82b7-f1116898debf-cni-binary-copy\") pod \"multus-h6p95\" (UID: \"682463ee-d9c9-4e9b-82b7-f1116898debf\") " pod="openshift-multus/multus-h6p95"
Nov 03 20:03:16.606014 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.605924    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/682463ee-d9c9-4e9b-82b7-f1116898debf-multus-socket-dir-parent\") pod \"multus-h6p95\" (UID: \"682463ee-d9c9-4e9b-82b7-f1116898debf\") " pod="openshift-multus/multus-h6p95"
Nov 03 20:03:16.606014 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.605960    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s92q2\" (UniqueName: \"kubernetes.io/projected/1d4ce60c-e058-4c29-99e8-5f4f6eead744-kube-api-access-s92q2\") pod \"node-ca-zzt8t\" (UID: \"1d4ce60c-e058-4c29-99e8-5f4f6eead744\") " pod="openshift-image-registry/node-ca-zzt8t"
Nov 03 20:03:16.606014 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.605986    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9126a982-59c9-4cec-a665-5f1bc0ae3389-systemd-units\") pod \"ovnkube-node-6sv7j\" (UID: \"9126a982-59c9-4cec-a665-5f1bc0ae3389\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sv7j"
Nov 03 20:03:16.606322 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.606028    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/51dffd76-5c30-440c-85ed-d81221663833-kubelet-config\") pod \"global-pull-secret-syncer-j4jzv\" (UID: \"51dffd76-5c30-440c-85ed-d81221663833\") " pod="kube-system/global-pull-secret-syncer-j4jzv"
Nov 03 20:03:16.606322 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.606062    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f21cf70e-682f-490c-bda5-3954df150487-sys\") pod \"tuned-4dmkq\" (UID: \"f21cf70e-682f-490c-bda5-3954df150487\") " pod="openshift-cluster-node-tuning-operator/tuned-4dmkq"
Nov 03 20:03:16.606322 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.606089    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/682463ee-d9c9-4e9b-82b7-f1116898debf-host-var-lib-cni-multus\") pod \"multus-h6p95\" (UID: \"682463ee-d9c9-4e9b-82b7-f1116898debf\") " pod="openshift-multus/multus-h6p95"
Nov 03 20:03:16.606322 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.606120    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlzhf\" (UniqueName: \"kubernetes.io/projected/db895326-506d-479d-a526-6bf71be7f967-kube-api-access-vlzhf\") pod \"multus-additional-cni-plugins-mhtmw\" (UID: \"db895326-506d-479d-a526-6bf71be7f967\") " pod="openshift-multus/multus-additional-cni-plugins-mhtmw"
Nov 03 20:03:16.606322 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.606163    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/716cc9cf-d26a-481b-8114-169c6104ce56-hosts-file\") pod \"node-resolver-t6dqw\" (UID: \"716cc9cf-d26a-481b-8114-169c6104ce56\") " pod="openshift-dns/node-resolver-t6dqw"
Nov 03 20:03:16.606322 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.606190    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9126a982-59c9-4cec-a665-5f1bc0ae3389-var-lib-openvswitch\") pod \"ovnkube-node-6sv7j\" (UID: \"9126a982-59c9-4cec-a665-5f1bc0ae3389\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sv7j"
Nov 03 20:03:16.606322 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.606214    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9126a982-59c9-4cec-a665-5f1bc0ae3389-host-cni-bin\") pod \"ovnkube-node-6sv7j\" (UID: \"9126a982-59c9-4cec-a665-5f1bc0ae3389\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sv7j"
Nov 03 20:03:16.606322 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.606239    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f21cf70e-682f-490c-bda5-3954df150487-etc-systemd\") pod \"tuned-4dmkq\" (UID: \"f21cf70e-682f-490c-bda5-3954df150487\") " pod="openshift-cluster-node-tuning-operator/tuned-4dmkq"
Nov 03 20:03:16.606322 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.606263    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f21cf70e-682f-490c-bda5-3954df150487-etc-tuned\") pod \"tuned-4dmkq\" (UID: \"f21cf70e-682f-490c-bda5-3954df150487\") " pod="openshift-cluster-node-tuning-operator/tuned-4dmkq"
Nov 03 20:03:16.606322 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.606289    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9126a982-59c9-4cec-a665-5f1bc0ae3389-host-slash\") pod \"ovnkube-node-6sv7j\" (UID: \"9126a982-59c9-4cec-a665-5f1bc0ae3389\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sv7j"
Nov 03 20:03:16.606322 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.606312    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9126a982-59c9-4cec-a665-5f1bc0ae3389-node-log\") pod \"ovnkube-node-6sv7j\" (UID: \"9126a982-59c9-4cec-a665-5f1bc0ae3389\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sv7j"
Nov 03 20:03:16.606690 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.606339    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89vxf\" (UniqueName: \"kubernetes.io/projected/9126a982-59c9-4cec-a665-5f1bc0ae3389-kube-api-access-89vxf\") pod \"ovnkube-node-6sv7j\" (UID: \"9126a982-59c9-4cec-a665-5f1bc0ae3389\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sv7j"
Nov 03 20:03:16.606690 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.606419    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/682463ee-d9c9-4e9b-82b7-f1116898debf-host-run-k8s-cni-cncf-io\") pod \"multus-h6p95\" (UID: \"682463ee-d9c9-4e9b-82b7-f1116898debf\") " pod="openshift-multus/multus-h6p95"
Nov 03 20:03:16.606690 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.606445    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/682463ee-d9c9-4e9b-82b7-f1116898debf-host-run-netns\") pod \"multus-h6p95\" (UID: \"682463ee-d9c9-4e9b-82b7-f1116898debf\") " pod="openshift-multus/multus-h6p95"
Nov 03 20:03:16.606690 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.606471    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/682463ee-d9c9-4e9b-82b7-f1116898debf-host-var-lib-kubelet\") pod \"multus-h6p95\" (UID: \"682463ee-d9c9-4e9b-82b7-f1116898debf\") " pod="openshift-multus/multus-h6p95"
Nov 03 20:03:16.606690 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.606496    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/db895326-506d-479d-a526-6bf71be7f967-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mhtmw\" (UID: \"db895326-506d-479d-a526-6bf71be7f967\") " pod="openshift-multus/multus-additional-cni-plugins-mhtmw"
Nov 03 20:03:16.606690 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.606523    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2073a8c8-6f07-4817-b7b3-468b327e2712-iptables-alerter-script\") pod \"iptables-alerter-9s4wk\" (UID: \"2073a8c8-6f07-4817-b7b3-468b327e2712\") " pod="openshift-network-operator/iptables-alerter-9s4wk"
Nov 03 20:03:16.606690 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.606555    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9126a982-59c9-4cec-a665-5f1bc0ae3389-host-run-netns\") pod \"ovnkube-node-6sv7j\" (UID: \"9126a982-59c9-4cec-a665-5f1bc0ae3389\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sv7j"
Nov 03 20:03:16.606690 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.606578    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f21cf70e-682f-490c-bda5-3954df150487-etc-sysconfig\") pod \"tuned-4dmkq\" (UID: \"f21cf70e-682f-490c-bda5-3954df150487\") " pod="openshift-cluster-node-tuning-operator/tuned-4dmkq"
Nov 03 20:03:16.606690 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.606604    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/682463ee-d9c9-4e9b-82b7-f1116898debf-multus-conf-dir\") pod \"multus-h6p95\" (UID: \"682463ee-d9c9-4e9b-82b7-f1116898debf\") " pod="openshift-multus/multus-h6p95"
Nov 03 20:03:16.606690 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.606629    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/682463ee-d9c9-4e9b-82b7-f1116898debf-etc-kubernetes\") pod \"multus-h6p95\" (UID: \"682463ee-d9c9-4e9b-82b7-f1116898debf\") " pod="openshift-multus/multus-h6p95"
Nov 03 20:03:16.606690 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.606654    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/db895326-506d-479d-a526-6bf71be7f967-cnibin\") pod \"multus-additional-cni-plugins-mhtmw\" (UID: \"db895326-506d-479d-a526-6bf71be7f967\") " pod="openshift-multus/multus-additional-cni-plugins-mhtmw"
Nov 03 20:03:16.606690 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.606681    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/db895326-506d-479d-a526-6bf71be7f967-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mhtmw\" (UID: \"db895326-506d-479d-a526-6bf71be7f967\") " pod="openshift-multus/multus-additional-cni-plugins-mhtmw"
Nov 03 20:03:16.607131 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.606707    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f21cf70e-682f-490c-bda5-3954df150487-etc-kubernetes\") pod \"tuned-4dmkq\" (UID: \"f21cf70e-682f-490c-bda5-3954df150487\") " pod="openshift-cluster-node-tuning-operator/tuned-4dmkq"
Nov 03 20:03:16.607131 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.606732    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c92a5d0-b34f-46a1-8f8f-a9ae3cb8d5a5-metrics-certs\") pod \"network-metrics-daemon-c947w\" (UID: \"4c92a5d0-b34f-46a1-8f8f-a9ae3cb8d5a5\") " pod="openshift-multus/network-metrics-daemon-c947w"
Nov 03 20:03:16.607131 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.606768    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9126a982-59c9-4cec-a665-5f1bc0ae3389-run-ovn\") pod \"ovnkube-node-6sv7j\" (UID: \"9126a982-59c9-4cec-a665-5f1bc0ae3389\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sv7j"
Nov 03 20:03:16.607131 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.606795    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/db895326-506d-479d-a526-6bf71be7f967-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mhtmw\" (UID: \"db895326-506d-479d-a526-6bf71be7f967\") " pod="openshift-multus/multus-additional-cni-plugins-mhtmw"
Nov 03 20:03:16.607131 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.606823    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrgcs\" (UniqueName: \"kubernetes.io/projected/2073a8c8-6f07-4817-b7b3-468b327e2712-kube-api-access-jrgcs\") pod \"iptables-alerter-9s4wk\" (UID: \"2073a8c8-6f07-4817-b7b3-468b327e2712\") " pod="openshift-network-operator/iptables-alerter-9s4wk"
Nov 03 20:03:16.607131 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.606939    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9126a982-59c9-4cec-a665-5f1bc0ae3389-ovn-node-metrics-cert\") pod \"ovnkube-node-6sv7j\" (UID: \"9126a982-59c9-4cec-a665-5f1bc0ae3389\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sv7j"
Nov 03 20:03:16.607131 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.606966    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f21cf70e-682f-490c-bda5-3954df150487-etc-modprobe-d\") pod \"tuned-4dmkq\" (UID: \"f21cf70e-682f-490c-bda5-3954df150487\") " pod="openshift-cluster-node-tuning-operator/tuned-4dmkq"
Nov 03 20:03:16.607131 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.607002    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/65008653-586d-4a76-a53f-7efe1f6d839b-sys-fs\") pod \"aws-ebs-csi-driver-node-9f5s5\" (UID: \"65008653-586d-4a76-a53f-7efe1f6d839b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9f5s5"
Nov 03 20:03:16.607131 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.607028    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkjpg\" (UniqueName: \"kubernetes.io/projected/682463ee-d9c9-4e9b-82b7-f1116898debf-kube-api-access-nkjpg\") pod \"multus-h6p95\" (UID: \"682463ee-d9c9-4e9b-82b7-f1116898debf\") " pod="openshift-multus/multus-h6p95"
Nov 03 20:03:16.607131 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.607062    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/db895326-506d-479d-a526-6bf71be7f967-cni-binary-copy\") pod \"multus-additional-cni-plugins-mhtmw\" (UID: \"db895326-506d-479d-a526-6bf71be7f967\") " pod="openshift-multus/multus-additional-cni-plugins-mhtmw"
Nov 03 20:03:16.633792 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.633756    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 03 20:03:16.633985 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.633756    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 03 20:03:16.634501 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.634463    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 03 20:03:16.634640 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.634540    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 03 20:03:16.637548 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.635058    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-2qw54\""
Nov 03 20:03:16.637548 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.635104    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 03 20:03:16.637548 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.635426    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-p4f2k\""
Nov 03 20:03:16.637548 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.635577    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 03 20:03:16.637548 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.635592    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 03 20:03:16.637548 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.635643    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 03 20:03:16.637548 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.635704    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 03 20:03:16.637548 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.635753    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 03 20:03:16.637548 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.635913    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 03 20:03:16.637548 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.635955    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 03 20:03:16.637548 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.636056    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 03 20:03:16.637548 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.636104    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-hwv2m\""
Nov 03 20:03:16.637548 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.636229    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 03 20:03:16.637548 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.636249    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 03 20:03:16.637548 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.636285    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-dfgm6\""
Nov 03 20:03:16.637548 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.636393    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-7kzdq\""
Nov 03 20:03:16.637548 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.636395    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-phn2h\""
Nov 03 20:03:16.637548 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.636422    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 03 20:03:16.708235 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.708202    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9126a982-59c9-4cec-a665-5f1bc0ae3389-run-openvswitch\") pod \"ovnkube-node-6sv7j\" (UID: \"9126a982-59c9-4cec-a665-5f1bc0ae3389\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sv7j"
Nov 03 20:03:16.708235 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.708241    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9126a982-59c9-4cec-a665-5f1bc0ae3389-run-systemd\") pod \"ovnkube-node-6sv7j\" (UID: \"9126a982-59c9-4cec-a665-5f1bc0ae3389\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sv7j"
Nov 03 20:03:16.708509 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.708269    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f21cf70e-682f-490c-bda5-3954df150487-etc-sysctl-d\") pod \"tuned-4dmkq\" (UID: \"f21cf70e-682f-490c-bda5-3954df150487\") " pod="openshift-cluster-node-tuning-operator/tuned-4dmkq"
Nov 03 20:03:16.708509 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.708300    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f21cf70e-682f-490c-bda5-3954df150487-host\") pod \"tuned-4dmkq\" (UID: \"f21cf70e-682f-490c-bda5-3954df150487\") " pod="openshift-cluster-node-tuning-operator/tuned-4dmkq"
Nov 03 20:03:16.708509 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.708316    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/65008653-586d-4a76-a53f-7efe1f6d839b-device-dir\") pod \"aws-ebs-csi-driver-node-9f5s5\" (UID: \"65008653-586d-4a76-a53f-7efe1f6d839b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9f5s5"
Nov 03 20:03:16.708509 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.708333    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nd2w\" (UniqueName: \"kubernetes.io/projected/4c92a5d0-b34f-46a1-8f8f-a9ae3cb8d5a5-kube-api-access-6nd2w\") pod \"network-metrics-daemon-c947w\" (UID: \"4c92a5d0-b34f-46a1-8f8f-a9ae3cb8d5a5\") " pod="openshift-multus/network-metrics-daemon-c947w"
Nov 03 20:03:16.708509 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.708350    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9126a982-59c9-4cec-a665-5f1bc0ae3389-run-openvswitch\") pod \"ovnkube-node-6sv7j\" (UID: \"9126a982-59c9-4cec-a665-5f1bc0ae3389\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sv7j"
Nov 03 20:03:16.708509 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.708355    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9126a982-59c9-4cec-a665-5f1bc0ae3389-host-run-ovn-kubernetes\") pod \"ovnkube-node-6sv7j\" (UID: \"9126a982-59c9-4cec-a665-5f1bc0ae3389\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sv7j"
Nov 03 20:03:16.708509 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.708400    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9126a982-59c9-4cec-a665-5f1bc0ae3389-host-run-ovn-kubernetes\") pod \"ovnkube-node-6sv7j\" (UID: \"9126a982-59c9-4cec-a665-5f1bc0ae3389\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sv7j"
Nov 03 20:03:16.708509 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.708413    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9126a982-59c9-4cec-a665-5f1bc0ae3389-ovnkube-config\") pod \"ovnkube-node-6sv7j\" (UID: \"9126a982-59c9-4cec-a665-5f1bc0ae3389\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sv7j"
Nov 03 20:03:16.708509 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.708442    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f21cf70e-682f-490c-bda5-3954df150487-lib-modules\") pod \"tuned-4dmkq\" (UID: \"f21cf70e-682f-490c-bda5-3954df150487\") " pod="openshift-cluster-node-tuning-operator/tuned-4dmkq"
Nov 03 20:03:16.708509 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.708450    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9126a982-59c9-4cec-a665-5f1bc0ae3389-run-systemd\") pod \"ovnkube-node-6sv7j\" (UID: \"9126a982-59c9-4cec-a665-5f1bc0ae3389\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sv7j"
Nov 03 20:03:16.708509 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.708466    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/65008653-586d-4a76-a53f-7efe1f6d839b-socket-dir\") pod \"aws-ebs-csi-driver-node-9f5s5\" (UID: \"65008653-586d-4a76-a53f-7efe1f6d839b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9f5s5"
Nov 03 20:03:16.708509 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.708494    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/682463ee-d9c9-4e9b-82b7-f1116898debf-cnibin\") pod \"multus-h6p95\" (UID: \"682463ee-d9c9-4e9b-82b7-f1116898debf\") " pod="openshift-multus/multus-h6p95"
Nov 03 20:03:16.709122 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.708519    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/682463ee-d9c9-4e9b-82b7-f1116898debf-cni-binary-copy\") pod \"multus-h6p95\" (UID: \"682463ee-d9c9-4e9b-82b7-f1116898debf\") " pod="openshift-multus/multus-h6p95"
Nov 03 20:03:16.709122 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.708543    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/682463ee-d9c9-4e9b-82b7-f1116898debf-multus-socket-dir-parent\") pod \"multus-h6p95\" (UID: \"682463ee-d9c9-4e9b-82b7-f1116898debf\") " pod="openshift-multus/multus-h6p95"
Nov 03 20:03:16.709122 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.708570    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s92q2\" (UniqueName: \"kubernetes.io/projected/1d4ce60c-e058-4c29-99e8-5f4f6eead744-kube-api-access-s92q2\") pod \"node-ca-zzt8t\" (UID: \"1d4ce60c-e058-4c29-99e8-5f4f6eead744\") " pod="openshift-image-registry/node-ca-zzt8t"
Nov 03 20:03:16.709122 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.708580    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f21cf70e-682f-490c-bda5-3954df150487-etc-sysctl-d\") pod \"tuned-4dmkq\" (UID: \"f21cf70e-682f-490c-bda5-3954df150487\") " pod="openshift-cluster-node-tuning-operator/tuned-4dmkq"
Nov 03 20:03:16.709122 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.708595    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9126a982-59c9-4cec-a665-5f1bc0ae3389-systemd-units\") pod \"ovnkube-node-6sv7j\" (UID: \"9126a982-59c9-4cec-a665-5f1bc0ae3389\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sv7j"
Nov 03 20:03:16.709122 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.708611    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f21cf70e-682f-490c-bda5-3954df150487-host\") pod \"tuned-4dmkq\" (UID: \"f21cf70e-682f-490c-bda5-3954df150487\") " pod="openshift-cluster-node-tuning-operator/tuned-4dmkq"
Nov 03 20:03:16.709122 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.708622    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/51dffd76-5c30-440c-85ed-d81221663833-kubelet-config\") pod \"global-pull-secret-syncer-j4jzv\" (UID: \"51dffd76-5c30-440c-85ed-d81221663833\") " pod="kube-system/global-pull-secret-syncer-j4jzv"
Nov 03 20:03:16.709122 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.708642    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/65008653-586d-4a76-a53f-7efe1f6d839b-device-dir\") pod \"aws-ebs-csi-driver-node-9f5s5\" (UID: \"65008653-586d-4a76-a53f-7efe1f6d839b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9f5s5"
Nov 03 20:03:16.709122 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.708649    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f21cf70e-682f-490c-bda5-3954df150487-sys\") pod \"tuned-4dmkq\" (UID: \"f21cf70e-682f-490c-bda5-3954df150487\") " pod="openshift-cluster-node-tuning-operator/tuned-4dmkq"
Nov 03 20:03:16.709122 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.708675    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/682463ee-d9c9-4e9b-82b7-f1116898debf-host-var-lib-cni-multus\") pod \"multus-h6p95\" (UID: \"682463ee-d9c9-4e9b-82b7-f1116898debf\") " pod="openshift-multus/multus-h6p95"
Nov 03 20:03:16.709122 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.708703    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlzhf\" (UniqueName: \"kubernetes.io/projected/db895326-506d-479d-a526-6bf71be7f967-kube-api-access-vlzhf\") pod \"multus-additional-cni-plugins-mhtmw\" (UID: \"db895326-506d-479d-a526-6bf71be7f967\") " pod="openshift-multus/multus-additional-cni-plugins-mhtmw"
Nov 03 20:03:16.709122 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.708736    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/716cc9cf-d26a-481b-8114-169c6104ce56-hosts-file\") pod \"node-resolver-t6dqw\" (UID: \"716cc9cf-d26a-481b-8114-169c6104ce56\") " pod="openshift-dns/node-resolver-t6dqw"
Nov 03 20:03:16.709122 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.708762    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9126a982-59c9-4cec-a665-5f1bc0ae3389-var-lib-openvswitch\") pod \"ovnkube-node-6sv7j\" (UID: \"9126a982-59c9-4cec-a665-5f1bc0ae3389\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sv7j"
Nov 03 20:03:16.709122 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.708787    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9126a982-59c9-4cec-a665-5f1bc0ae3389-host-cni-bin\") pod \"ovnkube-node-6sv7j\" (UID: \"9126a982-59c9-4cec-a665-5f1bc0ae3389\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sv7j"
Nov 03 20:03:16.709122 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.708844    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f21cf70e-682f-490c-bda5-3954df150487-etc-systemd\") pod \"tuned-4dmkq\" (UID: \"f21cf70e-682f-490c-bda5-3954df150487\") " pod="openshift-cluster-node-tuning-operator/tuned-4dmkq"
Nov 03 20:03:16.709122 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.708871    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f21cf70e-682f-490c-bda5-3954df150487-etc-tuned\") pod \"tuned-4dmkq\" (UID: \"f21cf70e-682f-490c-bda5-3954df150487\") " pod="openshift-cluster-node-tuning-operator/tuned-4dmkq"
Nov 03 20:03:16.709122 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.708908    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9126a982-59c9-4cec-a665-5f1bc0ae3389-host-slash\") pod \"ovnkube-node-6sv7j\" (UID: \"9126a982-59c9-4cec-a665-5f1bc0ae3389\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sv7j"
Nov 03 20:03:16.709122 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.708932    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9126a982-59c9-4cec-a665-5f1bc0ae3389-node-log\") pod \"ovnkube-node-6sv7j\" (UID: \"9126a982-59c9-4cec-a665-5f1bc0ae3389\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sv7j"
Nov 03 20:03:16.710033 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.708960    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89vxf\" (UniqueName: \"kubernetes.io/projected/9126a982-59c9-4cec-a665-5f1bc0ae3389-kube-api-access-89vxf\") pod \"ovnkube-node-6sv7j\" (UID: \"9126a982-59c9-4cec-a665-5f1bc0ae3389\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sv7j"
Nov 03 20:03:16.710033 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.708987    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/682463ee-d9c9-4e9b-82b7-f1116898debf-host-run-k8s-cni-cncf-io\") pod \"multus-h6p95\" (UID: \"682463ee-d9c9-4e9b-82b7-f1116898debf\") " pod="openshift-multus/multus-h6p95"
Nov 03 20:03:16.710033 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.708994    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/51dffd76-5c30-440c-85ed-d81221663833-kubelet-config\") pod \"global-pull-secret-syncer-j4jzv\" (UID: \"51dffd76-5c30-440c-85ed-d81221663833\") " pod="kube-system/global-pull-secret-syncer-j4jzv"
Nov 03 20:03:16.710033 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.709013    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/682463ee-d9c9-4e9b-82b7-f1116898debf-host-run-netns\") pod \"multus-h6p95\" (UID: \"682463ee-d9c9-4e9b-82b7-f1116898debf\") " pod="openshift-multus/multus-h6p95"
Nov 03 20:03:16.710033 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.709041    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/682463ee-d9c9-4e9b-82b7-f1116898debf-host-var-lib-kubelet\") pod \"multus-h6p95\" (UID: \"682463ee-d9c9-4e9b-82b7-f1116898debf\") " pod="openshift-multus/multus-h6p95"
Nov 03 20:03:16.710033 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.709068    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/db895326-506d-479d-a526-6bf71be7f967-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mhtmw\" (UID: \"db895326-506d-479d-a526-6bf71be7f967\") " pod="openshift-multus/multus-additional-cni-plugins-mhtmw"
Nov 03 20:03:16.710033 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.709095    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2073a8c8-6f07-4817-b7b3-468b327e2712-iptables-alerter-script\") pod \"iptables-alerter-9s4wk\" (UID: \"2073a8c8-6f07-4817-b7b3-468b327e2712\") " pod="openshift-network-operator/iptables-alerter-9s4wk"
Nov 03 20:03:16.710033 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.709160    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9126a982-59c9-4cec-a665-5f1bc0ae3389-host-run-netns\") pod \"ovnkube-node-6sv7j\" (UID: \"9126a982-59c9-4cec-a665-5f1bc0ae3389\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sv7j"
Nov 03 20:03:16.710033 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.709190    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f21cf70e-682f-490c-bda5-3954df150487-etc-sysconfig\") pod \"tuned-4dmkq\" (UID: \"f21cf70e-682f-490c-bda5-3954df150487\") " pod="openshift-cluster-node-tuning-operator/tuned-4dmkq"
Nov 03 20:03:16.710033 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.709217    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/682463ee-d9c9-4e9b-82b7-f1116898debf-multus-conf-dir\") pod \"multus-h6p95\" (UID: \"682463ee-d9c9-4e9b-82b7-f1116898debf\") " pod="openshift-multus/multus-h6p95"
Nov 03 20:03:16.710033 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.709244    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/682463ee-d9c9-4e9b-82b7-f1116898debf-etc-kubernetes\") pod \"multus-h6p95\" (UID: \"682463ee-d9c9-4e9b-82b7-f1116898debf\") " pod="openshift-multus/multus-h6p95"
Nov 03 20:03:16.710033 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.709270    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/db895326-506d-479d-a526-6bf71be7f967-cnibin\") pod \"multus-additional-cni-plugins-mhtmw\" (UID: \"db895326-506d-479d-a526-6bf71be7f967\") " pod="openshift-multus/multus-additional-cni-plugins-mhtmw"
Nov 03 20:03:16.710033 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.709298    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/db895326-506d-479d-a526-6bf71be7f967-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mhtmw\" (UID: \"db895326-506d-479d-a526-6bf71be7f967\") " pod="openshift-multus/multus-additional-cni-plugins-mhtmw"
Nov 03 20:03:16.710033 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.709325    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f21cf70e-682f-490c-bda5-3954df150487-etc-kubernetes\") pod \"tuned-4dmkq\" (UID: \"f21cf70e-682f-490c-bda5-3954df150487\") " pod="openshift-cluster-node-tuning-operator/tuned-4dmkq"
Nov 03 20:03:16.710033 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.709352    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c92a5d0-b34f-46a1-8f8f-a9ae3cb8d5a5-metrics-certs\") pod \"network-metrics-daemon-c947w\" (UID: \"4c92a5d0-b34f-46a1-8f8f-a9ae3cb8d5a5\") " pod="openshift-multus/network-metrics-daemon-c947w"
Nov 03 20:03:16.710033 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.709378    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9126a982-59c9-4cec-a665-5f1bc0ae3389-run-ovn\") pod \"ovnkube-node-6sv7j\" (UID: \"9126a982-59c9-4cec-a665-5f1bc0ae3389\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sv7j"
Nov 03 20:03:16.710033 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.709405    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/db895326-506d-479d-a526-6bf71be7f967-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mhtmw\" (UID: \"db895326-506d-479d-a526-6bf71be7f967\") " pod="openshift-multus/multus-additional-cni-plugins-mhtmw"
Nov 03 20:03:16.710872 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.709434    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrgcs\" (UniqueName: \"kubernetes.io/projected/2073a8c8-6f07-4817-b7b3-468b327e2712-kube-api-access-jrgcs\") pod \"iptables-alerter-9s4wk\" (UID: \"2073a8c8-6f07-4817-b7b3-468b327e2712\") " pod="openshift-network-operator/iptables-alerter-9s4wk"
Nov 03 20:03:16.710872 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.709464    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9126a982-59c9-4cec-a665-5f1bc0ae3389-ovn-node-metrics-cert\") pod \"ovnkube-node-6sv7j\" (UID: \"9126a982-59c9-4cec-a665-5f1bc0ae3389\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sv7j"
Nov 03 20:03:16.710872 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.709493    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f21cf70e-682f-490c-bda5-3954df150487-etc-modprobe-d\") pod \"tuned-4dmkq\" (UID: \"f21cf70e-682f-490c-bda5-3954df150487\") " pod="openshift-cluster-node-tuning-operator/tuned-4dmkq"
Nov 03 20:03:16.710872 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.709520    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/65008653-586d-4a76-a53f-7efe1f6d839b-sys-fs\") pod \"aws-ebs-csi-driver-node-9f5s5\" (UID: \"65008653-586d-4a76-a53f-7efe1f6d839b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9f5s5"
Nov 03 20:03:16.710872 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.709546    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkjpg\" (UniqueName: \"kubernetes.io/projected/682463ee-d9c9-4e9b-82b7-f1116898debf-kube-api-access-nkjpg\") pod \"multus-h6p95\" (UID: \"682463ee-d9c9-4e9b-82b7-f1116898debf\") " pod="openshift-multus/multus-h6p95"
Nov 03 20:03:16.710872 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.709557    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f21cf70e-682f-490c-bda5-3954df150487-lib-modules\") pod \"tuned-4dmkq\" (UID: \"f21cf70e-682f-490c-bda5-3954df150487\") " pod="openshift-cluster-node-tuning-operator/tuned-4dmkq"
Nov 03 20:03:16.710872 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.709573    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/db895326-506d-479d-a526-6bf71be7f967-cni-binary-copy\") pod \"multus-additional-cni-plugins-mhtmw\" (UID: \"db895326-506d-479d-a526-6bf71be7f967\") " pod="openshift-multus/multus-additional-cni-plugins-mhtmw"
Nov 03 20:03:16.710872 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.709606    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9126a982-59c9-4cec-a665-5f1bc0ae3389-host-kubelet\") pod \"ovnkube-node-6sv7j\" (UID: \"9126a982-59c9-4cec-a665-5f1bc0ae3389\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sv7j"
Nov 03 20:03:16.710872 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.709634    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9126a982-59c9-4cec-a665-5f1bc0ae3389-env-overrides\") pod \"ovnkube-node-6sv7j\" (UID: \"9126a982-59c9-4cec-a665-5f1bc0ae3389\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sv7j"
Nov 03 20:03:16.710872 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.709665    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/51dffd76-5c30-440c-85ed-d81221663833-dbus\") pod \"global-pull-secret-syncer-j4jzv\" (UID: \"51dffd76-5c30-440c-85ed-d81221663833\") " pod="kube-system/global-pull-secret-syncer-j4jzv"
Nov 03 20:03:16.710872 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.709666    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/65008653-586d-4a76-a53f-7efe1f6d839b-socket-dir\") pod \"aws-ebs-csi-driver-node-9f5s5\" (UID: \"65008653-586d-4a76-a53f-7efe1f6d839b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9f5s5"
Nov 03 20:03:16.710872 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.709696    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/65008653-586d-4a76-a53f-7efe1f6d839b-registration-dir\") pod \"aws-ebs-csi-driver-node-9f5s5\" (UID: \"65008653-586d-4a76-a53f-7efe1f6d839b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9f5s5"
Nov 03 20:03:16.710872 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.709720    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/682463ee-d9c9-4e9b-82b7-f1116898debf-cnibin\") pod \"multus-h6p95\" (UID: \"682463ee-d9c9-4e9b-82b7-f1116898debf\") " pod="openshift-multus/multus-h6p95"
Nov 03 20:03:16.710872 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.709726    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1d4ce60c-e058-4c29-99e8-5f4f6eead744-serviceca\") pod \"node-ca-zzt8t\" (UID: \"1d4ce60c-e058-4c29-99e8-5f4f6eead744\") " pod="openshift-image-registry/node-ca-zzt8t"
Nov 03 20:03:16.710872 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.709759    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcxdr\" (UniqueName: \"kubernetes.io/projected/d71eaa2a-9659-4827-9dfc-c1fce02565cb-kube-api-access-zcxdr\") pod \"network-check-target-6dfr5\" (UID: \"d71eaa2a-9659-4827-9dfc-c1fce02565cb\") " pod="openshift-network-diagnostics/network-check-target-6dfr5"
Nov 03 20:03:16.710872 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.709786    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9126a982-59c9-4cec-a665-5f1bc0ae3389-host-cni-netd\") pod \"ovnkube-node-6sv7j\" (UID: \"9126a982-59c9-4cec-a665-5f1bc0ae3389\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sv7j"
Nov 03 20:03:16.710872 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.709816    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f21cf70e-682f-490c-bda5-3954df150487-etc-sysctl-conf\") pod \"tuned-4dmkq\" (UID: \"f21cf70e-682f-490c-bda5-3954df150487\") " pod="openshift-cluster-node-tuning-operator/tuned-4dmkq"
Nov 03 20:03:16.711730 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.709844    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/682463ee-d9c9-4e9b-82b7-f1116898debf-multus-daemon-config\") pod \"multus-h6p95\" (UID: \"682463ee-d9c9-4e9b-82b7-f1116898debf\") " pod="openshift-multus/multus-h6p95"
Nov 03 20:03:16.711730 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.709887    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1d4ce60c-e058-4c29-99e8-5f4f6eead744-host\") pod \"node-ca-zzt8t\" (UID: \"1d4ce60c-e058-4c29-99e8-5f4f6eead744\") " pod="openshift-image-registry/node-ca-zzt8t"
Nov 03 20:03:16.711730 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.709915    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/15b5070f-d0d3-4ad8-aa1d-93cc7b96003a-konnectivity-ca\") pod \"konnectivity-agent-5p48g\" (UID: \"15b5070f-d0d3-4ad8-aa1d-93cc7b96003a\") " pod="kube-system/konnectivity-agent-5p48g"
Nov 03 20:03:16.711730 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.709941    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9126a982-59c9-4cec-a665-5f1bc0ae3389-log-socket\") pod \"ovnkube-node-6sv7j\" (UID: \"9126a982-59c9-4cec-a665-5f1bc0ae3389\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sv7j"
Nov 03 20:03:16.711730 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.709968    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/65008653-586d-4a76-a53f-7efe1f6d839b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9f5s5\" (UID: \"65008653-586d-4a76-a53f-7efe1f6d839b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9f5s5"
Nov 03 20:03:16.711730 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.709996    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/682463ee-d9c9-4e9b-82b7-f1116898debf-host-run-multus-certs\") pod \"multus-h6p95\" (UID: \"682463ee-d9c9-4e9b-82b7-f1116898debf\") " pod="openshift-multus/multus-h6p95"
Nov 03 20:03:16.711730 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.710026    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59sqq\" (UniqueName: \"kubernetes.io/projected/f21cf70e-682f-490c-bda5-3954df150487-kube-api-access-59sqq\") pod \"tuned-4dmkq\" (UID: \"f21cf70e-682f-490c-bda5-3954df150487\") " pod="openshift-cluster-node-tuning-operator/tuned-4dmkq"
Nov 03 20:03:16.711730 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.710050    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/15b5070f-d0d3-4ad8-aa1d-93cc7b96003a-agent-certs\") pod \"konnectivity-agent-5p48g\" (UID: \"15b5070f-d0d3-4ad8-aa1d-93cc7b96003a\") " pod="kube-system/konnectivity-agent-5p48g"
Nov 03 20:03:16.711730 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.710085    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/db895326-506d-479d-a526-6bf71be7f967-system-cni-dir\") pod \"multus-additional-cni-plugins-mhtmw\" (UID: \"db895326-506d-479d-a526-6bf71be7f967\") " pod="openshift-multus/multus-additional-cni-plugins-mhtmw"
Nov 03 20:03:16.711730 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.710112    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f21cf70e-682f-490c-bda5-3954df150487-run\") pod \"tuned-4dmkq\" (UID: \"f21cf70e-682f-490c-bda5-3954df150487\") " pod="openshift-cluster-node-tuning-operator/tuned-4dmkq"
Nov 03 20:03:16.711730 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.710141    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/682463ee-d9c9-4e9b-82b7-f1116898debf-system-cni-dir\") pod \"multus-h6p95\" (UID: \"682463ee-d9c9-4e9b-82b7-f1116898debf\") " pod="openshift-multus/multus-h6p95"
Nov 03 20:03:16.711730 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.710161    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/682463ee-d9c9-4e9b-82b7-f1116898debf-multus-socket-dir-parent\") pod \"multus-h6p95\" (UID: \"682463ee-d9c9-4e9b-82b7-f1116898debf\") " pod="openshift-multus/multus-h6p95"
Nov 03 20:03:16.711730 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.710169    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/51dffd76-5c30-440c-85ed-d81221663833-original-pull-secret\") pod \"global-pull-secret-syncer-j4jzv\" (UID: \"51dffd76-5c30-440c-85ed-d81221663833\") " pod="kube-system/global-pull-secret-syncer-j4jzv"
Nov 03 20:03:16.711730 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.710198    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f21cf70e-682f-490c-bda5-3954df150487-tmp\") pod \"tuned-4dmkq\" (UID: \"f21cf70e-682f-490c-bda5-3954df150487\") " pod="openshift-cluster-node-tuning-operator/tuned-4dmkq"
Nov 03 20:03:16.711730 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.710241    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/682463ee-d9c9-4e9b-82b7-f1116898debf-hostroot\") pod \"multus-h6p95\" (UID: \"682463ee-d9c9-4e9b-82b7-f1116898debf\") " pod="openshift-multus/multus-h6p95"
Nov 03 20:03:16.711730 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.710271    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2073a8c8-6f07-4817-b7b3-468b327e2712-host-slash\") pod \"iptables-alerter-9s4wk\" (UID: \"2073a8c8-6f07-4817-b7b3-468b327e2712\") " pod="openshift-network-operator/iptables-alerter-9s4wk"
Nov 03 20:03:16.711730 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.710299    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9126a982-59c9-4cec-a665-5f1bc0ae3389-etc-openvswitch\") pod \"ovnkube-node-6sv7j\" (UID: \"9126a982-59c9-4cec-a665-5f1bc0ae3389\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sv7j"
Nov 03 20:03:16.712617 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.710334    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9126a982-59c9-4cec-a665-5f1bc0ae3389-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6sv7j\" (UID: \"9126a982-59c9-4cec-a665-5f1bc0ae3389\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sv7j"
Nov 03 20:03:16.712617 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.710362    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f21cf70e-682f-490c-bda5-3954df150487-var-lib-kubelet\") pod \"tuned-4dmkq\" (UID: \"f21cf70e-682f-490c-bda5-3954df150487\") " pod="openshift-cluster-node-tuning-operator/tuned-4dmkq"
Nov 03 20:03:16.712617 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.710389    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/65008653-586d-4a76-a53f-7efe1f6d839b-etc-selinux\") pod \"aws-ebs-csi-driver-node-9f5s5\" (UID: \"65008653-586d-4a76-a53f-7efe1f6d839b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9f5s5"
Nov 03 20:03:16.712617 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.710421    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8psbp\" (UniqueName: \"kubernetes.io/projected/65008653-586d-4a76-a53f-7efe1f6d839b-kube-api-access-8psbp\") pod \"aws-ebs-csi-driver-node-9f5s5\" (UID: \"65008653-586d-4a76-a53f-7efe1f6d839b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9f5s5"
Nov 03 20:03:16.712617 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.710449    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/682463ee-d9c9-4e9b-82b7-f1116898debf-multus-cni-dir\") pod \"multus-h6p95\" (UID: \"682463ee-d9c9-4e9b-82b7-f1116898debf\") " pod="openshift-multus/multus-h6p95"
Nov 03 20:03:16.712617 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.710455    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9126a982-59c9-4cec-a665-5f1bc0ae3389-systemd-units\") pod \"ovnkube-node-6sv7j\" (UID: \"9126a982-59c9-4cec-a665-5f1bc0ae3389\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sv7j"
Nov 03 20:03:16.712617 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.710479    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9126a982-59c9-4cec-a665-5f1bc0ae3389-ovnkube-script-lib\") pod \"ovnkube-node-6sv7j\" (UID: \"9126a982-59c9-4cec-a665-5f1bc0ae3389\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sv7j"
Nov 03 20:03:16.712617 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.710508    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/682463ee-d9c9-4e9b-82b7-f1116898debf-host-var-lib-cni-bin\") pod \"multus-h6p95\" (UID: \"682463ee-d9c9-4e9b-82b7-f1116898debf\") " pod="openshift-multus/multus-h6p95"
Nov 03 20:03:16.712617 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.710536    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/716cc9cf-d26a-481b-8114-169c6104ce56-tmp-dir\") pod \"node-resolver-t6dqw\" (UID: \"716cc9cf-d26a-481b-8114-169c6104ce56\") " pod="openshift-dns/node-resolver-t6dqw"
Nov 03 20:03:16.712617 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.710574    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrdtp\" (UniqueName: \"kubernetes.io/projected/716cc9cf-d26a-481b-8114-169c6104ce56-kube-api-access-vrdtp\") pod \"node-resolver-t6dqw\" (UID: \"716cc9cf-d26a-481b-8114-169c6104ce56\") " pod="openshift-dns/node-resolver-t6dqw"
Nov 03 20:03:16.712617 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.710713    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f21cf70e-682f-490c-bda5-3954df150487-sys\") pod \"tuned-4dmkq\" (UID: \"f21cf70e-682f-490c-bda5-3954df150487\") " pod="openshift-cluster-node-tuning-operator/tuned-4dmkq"
Nov 03 20:03:16.712617 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.710769    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/682463ee-d9c9-4e9b-82b7-f1116898debf-host-var-lib-cni-multus\") pod \"multus-h6p95\" (UID: \"682463ee-d9c9-4e9b-82b7-f1116898debf\") " pod="openshift-multus/multus-h6p95"
Nov 03 20:03:16.712617 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.710971    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9126a982-59c9-4cec-a665-5f1bc0ae3389-host-run-netns\") pod \"ovnkube-node-6sv7j\" (UID: \"9126a982-59c9-4cec-a665-5f1bc0ae3389\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sv7j"
Nov 03 20:03:16.712617 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.711038    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/db895326-506d-479d-a526-6bf71be7f967-cnibin\") pod \"multus-additional-cni-plugins-mhtmw\" (UID: \"db895326-506d-479d-a526-6bf71be7f967\") " pod="openshift-multus/multus-additional-cni-plugins-mhtmw"
Nov 03 20:03:16.712617 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.711036    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/682463ee-d9c9-4e9b-82b7-f1116898debf-multus-conf-dir\") pod \"multus-h6p95\" (UID: \"682463ee-d9c9-4e9b-82b7-f1116898debf\") " pod="openshift-multus/multus-h6p95"
Nov 03 20:03:16.712617 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.711073    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/682463ee-d9c9-4e9b-82b7-f1116898debf-etc-kubernetes\") pod \"multus-h6p95\" (UID: \"682463ee-d9c9-4e9b-82b7-f1116898debf\") " pod="openshift-multus/multus-h6p95"
Nov 03 20:03:16.712617 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.711143    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/716cc9cf-d26a-481b-8114-169c6104ce56-hosts-file\") pod \"node-resolver-t6dqw\" (UID: \"716cc9cf-d26a-481b-8114-169c6104ce56\") " pod="openshift-dns/node-resolver-t6dqw"
Nov 03 20:03:16.712617 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.711150    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f21cf70e-682f-490c-bda5-3954df150487-run\") pod \"tuned-4dmkq\" (UID: \"f21cf70e-682f-490c-bda5-3954df150487\") " pod="openshift-cluster-node-tuning-operator/tuned-4dmkq"
Nov 03 20:03:16.713523 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.711185    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/db895326-506d-479d-a526-6bf71be7f967-system-cni-dir\") pod \"multus-additional-cni-plugins-mhtmw\" (UID: \"db895326-506d-479d-a526-6bf71be7f967\") " pod="openshift-multus/multus-additional-cni-plugins-mhtmw"
Nov 03 20:03:16.713523 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.711201    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9126a982-59c9-4cec-a665-5f1bc0ae3389-ovnkube-config\") pod \"ovnkube-node-6sv7j\" (UID: \"9126a982-59c9-4cec-a665-5f1bc0ae3389\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sv7j"
Nov 03 20:03:16.713523 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.711206    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9126a982-59c9-4cec-a665-5f1bc0ae3389-host-kubelet\") pod \"ovnkube-node-6sv7j\" (UID: \"9126a982-59c9-4cec-a665-5f1bc0ae3389\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sv7j"
Nov 03 20:03:16.713523 ip-10-0-132-4 kubenswrapper[2577]: E1103 20:03:16.711291    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 20:03:16.713523 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.711303    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f21cf70e-682f-490c-bda5-3954df150487-var-lib-kubelet\") pod \"tuned-4dmkq\" (UID: \"f21cf70e-682f-490c-bda5-3954df150487\") " pod="openshift-cluster-node-tuning-operator/tuned-4dmkq"
Nov 03 20:03:16.713523 ip-10-0-132-4 kubenswrapper[2577]: E1103 20:03:16.711377    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51dffd76-5c30-440c-85ed-d81221663833-original-pull-secret podName:51dffd76-5c30-440c-85ed-d81221663833 nodeName:}" failed. No retries permitted until 2026-11-03 20:03:17.211336603 +0000 UTC m=+3.071449141 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/51dffd76-5c30-440c-85ed-d81221663833-original-pull-secret") pod "global-pull-secret-syncer-j4jzv" (UID: "51dffd76-5c30-440c-85ed-d81221663833") : object "kube-system"/"original-pull-secret" not registered
Nov 03 20:03:16.713523 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.711436    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/682463ee-d9c9-4e9b-82b7-f1116898debf-multus-cni-dir\") pod \"multus-h6p95\" (UID: \"682463ee-d9c9-4e9b-82b7-f1116898debf\") " pod="openshift-multus/multus-h6p95"
Nov 03 20:03:16.713523 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.711484    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/65008653-586d-4a76-a53f-7efe1f6d839b-etc-selinux\") pod \"aws-ebs-csi-driver-node-9f5s5\" (UID: \"65008653-586d-4a76-a53f-7efe1f6d839b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9f5s5"
Nov 03 20:03:16.713523 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.711633    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9126a982-59c9-4cec-a665-5f1bc0ae3389-env-overrides\") pod \"ovnkube-node-6sv7j\" (UID: \"9126a982-59c9-4cec-a665-5f1bc0ae3389\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sv7j"
Nov 03 20:03:16.713523 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.711685    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2073a8c8-6f07-4817-b7b3-468b327e2712-host-slash\") pod \"iptables-alerter-9s4wk\" (UID: \"2073a8c8-6f07-4817-b7b3-468b327e2712\") " pod="openshift-network-operator/iptables-alerter-9s4wk"
Nov 03 20:03:16.713523 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.711701    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/51dffd76-5c30-440c-85ed-d81221663833-dbus\") pod \"global-pull-secret-syncer-j4jzv\" (UID: \"51dffd76-5c30-440c-85ed-d81221663833\") " pod="kube-system/global-pull-secret-syncer-j4jzv"
Nov 03 20:03:16.713523 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.711721    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/682463ee-d9c9-4e9b-82b7-f1116898debf-hostroot\") pod \"multus-h6p95\" (UID: \"682463ee-d9c9-4e9b-82b7-f1116898debf\") " pod="openshift-multus/multus-h6p95"
Nov 03 20:03:16.713523 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.711739    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/65008653-586d-4a76-a53f-7efe1f6d839b-registration-dir\") pod \"aws-ebs-csi-driver-node-9f5s5\" (UID: \"65008653-586d-4a76-a53f-7efe1f6d839b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9f5s5"
Nov 03 20:03:16.713523 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.711749    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1d4ce60c-e058-4c29-99e8-5f4f6eead744-host\") pod \"node-ca-zzt8t\" (UID: \"1d4ce60c-e058-4c29-99e8-5f4f6eead744\") " pod="openshift-image-registry/node-ca-zzt8t"
Nov 03 20:03:16.713523 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.711955    2577 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 03 20:03:16.713523 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.711978    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9126a982-59c9-4cec-a665-5f1bc0ae3389-host-cni-netd\") pod \"ovnkube-node-6sv7j\" (UID: \"9126a982-59c9-4cec-a665-5f1bc0ae3389\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sv7j"
Nov 03 20:03:16.713523 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.712045    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f21cf70e-682f-490c-bda5-3954df150487-etc-kubernetes\") pod \"tuned-4dmkq\" (UID: \"f21cf70e-682f-490c-bda5-3954df150487\") " pod="openshift-cluster-node-tuning-operator/tuned-4dmkq"
Nov 03 20:03:16.713523 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.712069    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9126a982-59c9-4cec-a665-5f1bc0ae3389-ovnkube-script-lib\") pod \"ovnkube-node-6sv7j\" (UID: \"9126a982-59c9-4cec-a665-5f1bc0ae3389\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sv7j"
Nov 03 20:03:16.714605 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.712114    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f21cf70e-682f-490c-bda5-3954df150487-etc-sysctl-conf\") pod \"tuned-4dmkq\" (UID: \"f21cf70e-682f-490c-bda5-3954df150487\") " pod="openshift-cluster-node-tuning-operator/tuned-4dmkq"
Nov 03 20:03:16.714605 ip-10-0-132-4 kubenswrapper[2577]: E1103 20:03:16.712136    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 20:03:16.714605 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.712142    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/682463ee-d9c9-4e9b-82b7-f1116898debf-host-var-lib-cni-bin\") pod \"multus-h6p95\" (UID: \"682463ee-d9c9-4e9b-82b7-f1116898debf\") " pod="openshift-multus/multus-h6p95"
Nov 03 20:03:16.714605 ip-10-0-132-4 kubenswrapper[2577]: E1103 20:03:16.712192    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c92a5d0-b34f-46a1-8f8f-a9ae3cb8d5a5-metrics-certs podName:4c92a5d0-b34f-46a1-8f8f-a9ae3cb8d5a5 nodeName:}" failed. No retries permitted until 2026-11-03 20:03:17.212171657 +0000 UTC m=+3.072284186 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4c92a5d0-b34f-46a1-8f8f-a9ae3cb8d5a5-metrics-certs") pod "network-metrics-daemon-c947w" (UID: "4c92a5d0-b34f-46a1-8f8f-a9ae3cb8d5a5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 20:03:16.714605 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.712241    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9126a982-59c9-4cec-a665-5f1bc0ae3389-run-ovn\") pod \"ovnkube-node-6sv7j\" (UID: \"9126a982-59c9-4cec-a665-5f1bc0ae3389\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sv7j"
Nov 03 20:03:16.714605 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.712438    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/716cc9cf-d26a-481b-8114-169c6104ce56-tmp-dir\") pod \"node-resolver-t6dqw\" (UID: \"716cc9cf-d26a-481b-8114-169c6104ce56\") " pod="openshift-dns/node-resolver-t6dqw"
Nov 03 20:03:16.714605 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.712544    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/65008653-586d-4a76-a53f-7efe1f6d839b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9f5s5\" (UID: \"65008653-586d-4a76-a53f-7efe1f6d839b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9f5s5"
Nov 03 20:03:16.714605 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.712598    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9126a982-59c9-4cec-a665-5f1bc0ae3389-log-socket\") pod \"ovnkube-node-6sv7j\" (UID: \"9126a982-59c9-4cec-a665-5f1bc0ae3389\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sv7j"
Nov 03 20:03:16.714605 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.712640    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/682463ee-d9c9-4e9b-82b7-f1116898debf-cni-binary-copy\") pod \"multus-h6p95\" (UID: \"682463ee-d9c9-4e9b-82b7-f1116898debf\") " pod="openshift-multus/multus-h6p95"
Nov 03 20:03:16.714605 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.712664    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/682463ee-d9c9-4e9b-82b7-f1116898debf-multus-daemon-config\") pod \"multus-h6p95\" (UID: \"682463ee-d9c9-4e9b-82b7-f1116898debf\") " pod="openshift-multus/multus-h6p95"
Nov 03 20:03:16.714605 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.712733    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/65008653-586d-4a76-a53f-7efe1f6d839b-sys-fs\") pod \"aws-ebs-csi-driver-node-9f5s5\" (UID: \"65008653-586d-4a76-a53f-7efe1f6d839b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9f5s5"
Nov 03 20:03:16.714605 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.712774    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f21cf70e-682f-490c-bda5-3954df150487-etc-modprobe-d\") pod \"tuned-4dmkq\" (UID: \"f21cf70e-682f-490c-bda5-3954df150487\") " pod="openshift-cluster-node-tuning-operator/tuned-4dmkq"
Nov 03 20:03:16.714605 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.712783    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9126a982-59c9-4cec-a665-5f1bc0ae3389-etc-openvswitch\") pod \"ovnkube-node-6sv7j\" (UID: \"9126a982-59c9-4cec-a665-5f1bc0ae3389\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sv7j"
Nov 03 20:03:16.714605 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.712828    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/682463ee-d9c9-4e9b-82b7-f1116898debf-host-run-multus-certs\") pod \"multus-h6p95\" (UID: \"682463ee-d9c9-4e9b-82b7-f1116898debf\") " pod="openshift-multus/multus-h6p95"
Nov 03 20:03:16.714605 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.713031    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9126a982-59c9-4cec-a665-5f1bc0ae3389-var-lib-openvswitch\") pod \"ovnkube-node-6sv7j\" (UID: \"9126a982-59c9-4cec-a665-5f1bc0ae3389\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sv7j"
Nov 03 20:03:16.714605 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.713079    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9126a982-59c9-4cec-a665-5f1bc0ae3389-host-slash\") pod \"ovnkube-node-6sv7j\" (UID: \"9126a982-59c9-4cec-a665-5f1bc0ae3389\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sv7j"
Nov 03 20:03:16.714605 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.713130    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9126a982-59c9-4cec-a665-5f1bc0ae3389-node-log\") pod \"ovnkube-node-6sv7j\" (UID: \"9126a982-59c9-4cec-a665-5f1bc0ae3389\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sv7j"
Nov 03 20:03:16.715304 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.713178    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9126a982-59c9-4cec-a665-5f1bc0ae3389-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6sv7j\" (UID: \"9126a982-59c9-4cec-a665-5f1bc0ae3389\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sv7j"
Nov 03 20:03:16.715304 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.713226    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/682463ee-d9c9-4e9b-82b7-f1116898debf-host-var-lib-kubelet\") pod \"multus-h6p95\" (UID: \"682463ee-d9c9-4e9b-82b7-f1116898debf\") " pod="openshift-multus/multus-h6p95"
Nov 03 20:03:16.715304 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.713270    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/682463ee-d9c9-4e9b-82b7-f1116898debf-host-run-k8s-cni-cncf-io\") pod \"multus-h6p95\" (UID: \"682463ee-d9c9-4e9b-82b7-f1116898debf\") " pod="openshift-multus/multus-h6p95"
Nov 03 20:03:16.715304 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.713316    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/682463ee-d9c9-4e9b-82b7-f1116898debf-host-run-netns\") pod \"multus-h6p95\" (UID: \"682463ee-d9c9-4e9b-82b7-f1116898debf\") " pod="openshift-multus/multus-h6p95"
Nov 03 20:03:16.715304 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.713542    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1d4ce60c-e058-4c29-99e8-5f4f6eead744-serviceca\") pod \"node-ca-zzt8t\" (UID: \"1d4ce60c-e058-4c29-99e8-5f4f6eead744\") " pod="openshift-image-registry/node-ca-zzt8t"
Nov 03 20:03:16.715304 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.713876    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2073a8c8-6f07-4817-b7b3-468b327e2712-iptables-alerter-script\") pod \"iptables-alerter-9s4wk\" (UID: \"2073a8c8-6f07-4817-b7b3-468b327e2712\") " pod="openshift-network-operator/iptables-alerter-9s4wk"
Nov 03 20:03:16.715304 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.714045    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/db895326-506d-479d-a526-6bf71be7f967-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mhtmw\" (UID: \"db895326-506d-479d-a526-6bf71be7f967\") " pod="openshift-multus/multus-additional-cni-plugins-mhtmw"
Nov 03 20:03:16.715304 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.714080    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/15b5070f-d0d3-4ad8-aa1d-93cc7b96003a-konnectivity-ca\") pod \"konnectivity-agent-5p48g\" (UID: \"15b5070f-d0d3-4ad8-aa1d-93cc7b96003a\") " pod="kube-system/konnectivity-agent-5p48g"
Nov 03 20:03:16.715304 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.714157    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9126a982-59c9-4cec-a665-5f1bc0ae3389-host-cni-bin\") pod \"ovnkube-node-6sv7j\" (UID: \"9126a982-59c9-4cec-a665-5f1bc0ae3389\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sv7j"
Nov 03 20:03:16.715304 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.714445    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f21cf70e-682f-490c-bda5-3954df150487-etc-systemd\") pod \"tuned-4dmkq\" (UID: \"f21cf70e-682f-490c-bda5-3954df150487\") " pod="openshift-cluster-node-tuning-operator/tuned-4dmkq"
Nov 03 20:03:16.715304 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.710982    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f21cf70e-682f-490c-bda5-3954df150487-etc-sysconfig\") pod \"tuned-4dmkq\" (UID: \"f21cf70e-682f-490c-bda5-3954df150487\") " pod="openshift-cluster-node-tuning-operator/tuned-4dmkq"
Nov 03 20:03:16.715304 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.714475    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/db895326-506d-479d-a526-6bf71be7f967-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mhtmw\" (UID: \"db895326-506d-479d-a526-6bf71be7f967\") " pod="openshift-multus/multus-additional-cni-plugins-mhtmw"
Nov 03 20:03:16.715304 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.714657    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/682463ee-d9c9-4e9b-82b7-f1116898debf-system-cni-dir\") pod \"multus-h6p95\" (UID: \"682463ee-d9c9-4e9b-82b7-f1116898debf\") " pod="openshift-multus/multus-h6p95"
Nov 03 20:03:16.715304 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.715095    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/db895326-506d-479d-a526-6bf71be7f967-cni-binary-copy\") pod \"multus-additional-cni-plugins-mhtmw\" (UID: \"db895326-506d-479d-a526-6bf71be7f967\") " pod="openshift-multus/multus-additional-cni-plugins-mhtmw"
Nov 03 20:03:16.718826 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.718785    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9126a982-59c9-4cec-a665-5f1bc0ae3389-ovn-node-metrics-cert\") pod \"ovnkube-node-6sv7j\" (UID: \"9126a982-59c9-4cec-a665-5f1bc0ae3389\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sv7j"
Nov 03 20:03:16.719312 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.719285    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f21cf70e-682f-490c-bda5-3954df150487-etc-tuned\") pod \"tuned-4dmkq\" (UID: \"f21cf70e-682f-490c-bda5-3954df150487\") " pod="openshift-cluster-node-tuning-operator/tuned-4dmkq"
Nov 03 20:03:16.720234 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.720177    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/15b5070f-d0d3-4ad8-aa1d-93cc7b96003a-agent-certs\") pod \"konnectivity-agent-5p48g\" (UID: \"15b5070f-d0d3-4ad8-aa1d-93cc7b96003a\") " pod="kube-system/konnectivity-agent-5p48g"
Nov 03 20:03:16.721370 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.721329    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f21cf70e-682f-490c-bda5-3954df150487-tmp\") pod \"tuned-4dmkq\" (UID: \"f21cf70e-682f-490c-bda5-3954df150487\") " pod="openshift-cluster-node-tuning-operator/tuned-4dmkq"
Nov 03 20:03:16.726465 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.726430    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/db895326-506d-479d-a526-6bf71be7f967-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mhtmw\" (UID: \"db895326-506d-479d-a526-6bf71be7f967\") " pod="openshift-multus/multus-additional-cni-plugins-mhtmw"
Nov 03 20:03:16.729596 ip-10-0-132-4 kubenswrapper[2577]: E1103 20:03:16.729574    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 20:03:16.729727 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.729701    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nd2w\" (UniqueName: \"kubernetes.io/projected/4c92a5d0-b34f-46a1-8f8f-a9ae3cb8d5a5-kube-api-access-6nd2w\") pod \"network-metrics-daemon-c947w\" (UID: \"4c92a5d0-b34f-46a1-8f8f-a9ae3cb8d5a5\") " pod="openshift-multus/network-metrics-daemon-c947w"
Nov 03 20:03:16.729865 ip-10-0-132-4 kubenswrapper[2577]: E1103 20:03:16.729708    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 20:03:16.729865 ip-10-0-132-4 kubenswrapper[2577]: E1103 20:03:16.729775    2577 projected.go:196] Error preparing data for projected volume kube-api-access-zcxdr for pod openshift-network-diagnostics/network-check-target-6dfr5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 20:03:16.729865 ip-10-0-132-4 kubenswrapper[2577]: E1103 20:03:16.729841    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d71eaa2a-9659-4827-9dfc-c1fce02565cb-kube-api-access-zcxdr podName:d71eaa2a-9659-4827-9dfc-c1fce02565cb nodeName:}" failed. No retries permitted until 2026-11-03 20:03:17.229820344 +0000 UTC m=+3.089932881 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-zcxdr" (UniqueName: "kubernetes.io/projected/d71eaa2a-9659-4827-9dfc-c1fce02565cb-kube-api-access-zcxdr") pod "network-check-target-6dfr5" (UID: "d71eaa2a-9659-4827-9dfc-c1fce02565cb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 20:03:16.732499 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.732470    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8psbp\" (UniqueName: \"kubernetes.io/projected/65008653-586d-4a76-a53f-7efe1f6d839b-kube-api-access-8psbp\") pod \"aws-ebs-csi-driver-node-9f5s5\" (UID: \"65008653-586d-4a76-a53f-7efe1f6d839b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9f5s5"
Nov 03 20:03:16.732597 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.732555    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrdtp\" (UniqueName: \"kubernetes.io/projected/716cc9cf-d26a-481b-8114-169c6104ce56-kube-api-access-vrdtp\") pod \"node-resolver-t6dqw\" (UID: \"716cc9cf-d26a-481b-8114-169c6104ce56\") " pod="openshift-dns/node-resolver-t6dqw"
Nov 03 20:03:16.735740 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.735712    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrgcs\" (UniqueName: \"kubernetes.io/projected/2073a8c8-6f07-4817-b7b3-468b327e2712-kube-api-access-jrgcs\") pod \"iptables-alerter-9s4wk\" (UID: \"2073a8c8-6f07-4817-b7b3-468b327e2712\") " pod="openshift-network-operator/iptables-alerter-9s4wk"
Nov 03 20:03:16.736121 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.736098    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlzhf\" (UniqueName: \"kubernetes.io/projected/db895326-506d-479d-a526-6bf71be7f967-kube-api-access-vlzhf\") pod \"multus-additional-cni-plugins-mhtmw\" (UID: \"db895326-506d-479d-a526-6bf71be7f967\") " pod="openshift-multus/multus-additional-cni-plugins-mhtmw"
Nov 03 20:03:16.736121 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.736113    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89vxf\" (UniqueName: \"kubernetes.io/projected/9126a982-59c9-4cec-a665-5f1bc0ae3389-kube-api-access-89vxf\") pod \"ovnkube-node-6sv7j\" (UID: \"9126a982-59c9-4cec-a665-5f1bc0ae3389\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sv7j"
Nov 03 20:03:16.736276 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.736192    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s92q2\" (UniqueName: \"kubernetes.io/projected/1d4ce60c-e058-4c29-99e8-5f4f6eead744-kube-api-access-s92q2\") pod \"node-ca-zzt8t\" (UID: \"1d4ce60c-e058-4c29-99e8-5f4f6eead744\") " pod="openshift-image-registry/node-ca-zzt8t"
Nov 03 20:03:16.736505 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.736458    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkjpg\" (UniqueName: \"kubernetes.io/projected/682463ee-d9c9-4e9b-82b7-f1116898debf-kube-api-access-nkjpg\") pod \"multus-h6p95\" (UID: \"682463ee-d9c9-4e9b-82b7-f1116898debf\") " pod="openshift-multus/multus-h6p95"
Nov 03 20:03:16.738168 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.738148    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-59sqq\" (UniqueName: \"kubernetes.io/projected/f21cf70e-682f-490c-bda5-3954df150487-kube-api-access-59sqq\") pod \"tuned-4dmkq\" (UID: \"f21cf70e-682f-490c-bda5-3954df150487\") " pod="openshift-cluster-node-tuning-operator/tuned-4dmkq"
Nov 03 20:03:16.829831 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.829514    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-zzt8t"
Nov 03 20:03:16.842928 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:16.842891    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d4ce60c_e058_4c29_99e8_5f4f6eead744.slice/crio-cbc010db7797f543fd2ceb98399bf1cae3f0916fdf67b47f9a13d1916ae59363 WatchSource:0}: Error finding container cbc010db7797f543fd2ceb98399bf1cae3f0916fdf67b47f9a13d1916ae59363: Status 404 returned error can't find the container with id cbc010db7797f543fd2ceb98399bf1cae3f0916fdf67b47f9a13d1916ae59363
Nov 03 20:03:16.852027 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.851998    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-5p48g"
Nov 03 20:03:16.860648 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.860428    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-mhtmw"
Nov 03 20:03:16.863561 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:16.863536    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15b5070f_d0d3_4ad8_aa1d_93cc7b96003a.slice/crio-2368c89e09f8a61dcbe91cdf62b26e6fc92980ef44c8d851cd77c563cae625c6 WatchSource:0}: Error finding container 2368c89e09f8a61dcbe91cdf62b26e6fc92980ef44c8d851cd77c563cae625c6: Status 404 returned error can't find the container with id 2368c89e09f8a61dcbe91cdf62b26e6fc92980ef44c8d851cd77c563cae625c6
Nov 03 20:03:16.864657 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.864635    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9f5s5"
Nov 03 20:03:16.867082 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.867023    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-h6p95"
Nov 03 20:03:16.873066 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:16.871625    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb895326_506d_479d_a526_6bf71be7f967.slice/crio-1a93c727fa866700775189988b37b9b744284f0a816bb9a05732853f97812258 WatchSource:0}: Error finding container 1a93c727fa866700775189988b37b9b744284f0a816bb9a05732853f97812258: Status 404 returned error can't find the container with id 1a93c727fa866700775189988b37b9b744284f0a816bb9a05732853f97812258
Nov 03 20:03:16.873066 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.872461    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-9s4wk"
Nov 03 20:03:16.879416 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:16.879382    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65008653_586d_4a76_a53f_7efe1f6d839b.slice/crio-8dd1760f632c5c28135a89fef44e119cf12f4fd22c927b2490d8a4995b99427a WatchSource:0}: Error finding container 8dd1760f632c5c28135a89fef44e119cf12f4fd22c927b2490d8a4995b99427a: Status 404 returned error can't find the container with id 8dd1760f632c5c28135a89fef44e119cf12f4fd22c927b2490d8a4995b99427a
Nov 03 20:03:16.885455 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:16.885364    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod682463ee_d9c9_4e9b_82b7_f1116898debf.slice/crio-0e73128cc29efba8f4b0ef35efead17c585a7fb5ac4db5f96fbfcde7e777d2d5 WatchSource:0}: Error finding container 0e73128cc29efba8f4b0ef35efead17c585a7fb5ac4db5f96fbfcde7e777d2d5: Status 404 returned error can't find the container with id 0e73128cc29efba8f4b0ef35efead17c585a7fb5ac4db5f96fbfcde7e777d2d5
Nov 03 20:03:16.896804 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.896765    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-4dmkq"
Nov 03 20:03:16.899365 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.899345    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-t6dqw"
Nov 03 20:03:16.901199 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:16.901176    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-6sv7j"
Nov 03 20:03:16.912979 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:16.912881    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf21cf70e_682f_490c_bda5_3954df150487.slice/crio-2d654bb89ba8785ae38bae15a38c8508d4891aae7028d40ed527a9aa6702b0d2 WatchSource:0}: Error finding container 2d654bb89ba8785ae38bae15a38c8508d4891aae7028d40ed527a9aa6702b0d2: Status 404 returned error can't find the container with id 2d654bb89ba8785ae38bae15a38c8508d4891aae7028d40ed527a9aa6702b0d2
Nov 03 20:03:16.914801 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:16.914772    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod716cc9cf_d26a_481b_8114_169c6104ce56.slice/crio-baed098a69878751166eba110018725dc271d8ffc557e40cfc38922072195360 WatchSource:0}: Error finding container baed098a69878751166eba110018725dc271d8ffc557e40cfc38922072195360: Status 404 returned error can't find the container with id baed098a69878751166eba110018725dc271d8ffc557e40cfc38922072195360
Nov 03 20:03:16.918694 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:16.918669    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9126a982_59c9_4cec_a665_5f1bc0ae3389.slice/crio-5f4449d3d1de720c3b851fbffba06f75746a71960acf6a437b0ae3f926e304ca WatchSource:0}: Error finding container 5f4449d3d1de720c3b851fbffba06f75746a71960acf6a437b0ae3f926e304ca: Status 404 returned error can't find the container with id 5f4449d3d1de720c3b851fbffba06f75746a71960acf6a437b0ae3f926e304ca
Nov 03 20:03:17.214100 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:17.214067    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/51dffd76-5c30-440c-85ed-d81221663833-original-pull-secret\") pod \"global-pull-secret-syncer-j4jzv\" (UID: \"51dffd76-5c30-440c-85ed-d81221663833\") " pod="kube-system/global-pull-secret-syncer-j4jzv"
Nov 03 20:03:17.214234 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:17.214120    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c92a5d0-b34f-46a1-8f8f-a9ae3cb8d5a5-metrics-certs\") pod \"network-metrics-daemon-c947w\" (UID: \"4c92a5d0-b34f-46a1-8f8f-a9ae3cb8d5a5\") " pod="openshift-multus/network-metrics-daemon-c947w"
Nov 03 20:03:17.214291 ip-10-0-132-4 kubenswrapper[2577]: E1103 20:03:17.214233    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 20:03:17.214291 ip-10-0-132-4 kubenswrapper[2577]: E1103 20:03:17.214242    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 20:03:17.214405 ip-10-0-132-4 kubenswrapper[2577]: E1103 20:03:17.214293    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c92a5d0-b34f-46a1-8f8f-a9ae3cb8d5a5-metrics-certs podName:4c92a5d0-b34f-46a1-8f8f-a9ae3cb8d5a5 nodeName:}" failed. No retries permitted until 2026-11-03 20:03:18.214277596 +0000 UTC m=+4.074390113 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4c92a5d0-b34f-46a1-8f8f-a9ae3cb8d5a5-metrics-certs") pod "network-metrics-daemon-c947w" (UID: "4c92a5d0-b34f-46a1-8f8f-a9ae3cb8d5a5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 20:03:17.214405 ip-10-0-132-4 kubenswrapper[2577]: E1103 20:03:17.214307    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51dffd76-5c30-440c-85ed-d81221663833-original-pull-secret podName:51dffd76-5c30-440c-85ed-d81221663833 nodeName:}" failed. No retries permitted until 2026-11-03 20:03:18.214301326 +0000 UTC m=+4.074413842 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/51dffd76-5c30-440c-85ed-d81221663833-original-pull-secret") pod "global-pull-secret-syncer-j4jzv" (UID: "51dffd76-5c30-440c-85ed-d81221663833") : object "kube-system"/"original-pull-secret" not registered
Nov 03 20:03:17.315455 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:17.314782    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcxdr\" (UniqueName: \"kubernetes.io/projected/d71eaa2a-9659-4827-9dfc-c1fce02565cb-kube-api-access-zcxdr\") pod \"network-check-target-6dfr5\" (UID: \"d71eaa2a-9659-4827-9dfc-c1fce02565cb\") " pod="openshift-network-diagnostics/network-check-target-6dfr5"
Nov 03 20:03:17.315455 ip-10-0-132-4 kubenswrapper[2577]: E1103 20:03:17.315031    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 20:03:17.315455 ip-10-0-132-4 kubenswrapper[2577]: E1103 20:03:17.315054    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 20:03:17.315455 ip-10-0-132-4 kubenswrapper[2577]: E1103 20:03:17.315068    2577 projected.go:196] Error preparing data for projected volume kube-api-access-zcxdr for pod openshift-network-diagnostics/network-check-target-6dfr5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 20:03:17.315455 ip-10-0-132-4 kubenswrapper[2577]: E1103 20:03:17.315133    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d71eaa2a-9659-4827-9dfc-c1fce02565cb-kube-api-access-zcxdr podName:d71eaa2a-9659-4827-9dfc-c1fce02565cb nodeName:}" failed. No retries permitted until 2026-11-03 20:03:18.315110618 +0000 UTC m=+4.175223135 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-zcxdr" (UniqueName: "kubernetes.io/projected/d71eaa2a-9659-4827-9dfc-c1fce02565cb-kube-api-access-zcxdr") pod "network-check-target-6dfr5" (UID: "d71eaa2a-9659-4827-9dfc-c1fce02565cb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 20:03:17.543372 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:17.543191    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 19:58:16 +0000 UTC" deadline="2028-07-25 21:43:38.620516094 +0000 UTC"
Nov 03 20:03:17.543372 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:17.543254    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15121h40m21.077265281s"
Nov 03 20:03:17.640389 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:17.640347    2577 generic.go:356] "Generic (PLEG): container finished" podID="5dff8c1e3d20edc13c5947130e8a5c14" containerID="a36b3d172800908c76d5e9fe34ca4648289f454367148e1e5ea3959a3de88550" exitCode=0
Nov 03 20:03:17.640568 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:17.640447    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-4.ec2.internal" event={"ID":"5dff8c1e3d20edc13c5947130e8a5c14","Type":"ContainerDied","Data":"a36b3d172800908c76d5e9fe34ca4648289f454367148e1e5ea3959a3de88550"}
Nov 03 20:03:17.648152 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:17.648053    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mhtmw" event={"ID":"db895326-506d-479d-a526-6bf71be7f967","Type":"ContainerStarted","Data":"1a93c727fa866700775189988b37b9b744284f0a816bb9a05732853f97812258"}
Nov 03 20:03:17.649937 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:17.649907    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-5p48g" event={"ID":"15b5070f-d0d3-4ad8-aa1d-93cc7b96003a","Type":"ContainerStarted","Data":"2368c89e09f8a61dcbe91cdf62b26e6fc92980ef44c8d851cd77c563cae625c6"}
Nov 03 20:03:17.652219 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:17.652191    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-zzt8t" event={"ID":"1d4ce60c-e058-4c29-99e8-5f4f6eead744","Type":"ContainerStarted","Data":"cbc010db7797f543fd2ceb98399bf1cae3f0916fdf67b47f9a13d1916ae59363"}
Nov 03 20:03:17.655246 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:17.655223    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-4.ec2.internal" event={"ID":"a66c9af704a53c4c9b04653e0d8d8d8f","Type":"ContainerStarted","Data":"02df7ae991422a8e3fe518fd89cfc64e29f3dedc413076f1956aeec1ec7c5185"}
Nov 03 20:03:17.665642 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:17.665600    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6sv7j" event={"ID":"9126a982-59c9-4cec-a665-5f1bc0ae3389","Type":"ContainerStarted","Data":"5f4449d3d1de720c3b851fbffba06f75746a71960acf6a437b0ae3f926e304ca"}
Nov 03 20:03:17.674252 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:17.674220    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-t6dqw" event={"ID":"716cc9cf-d26a-481b-8114-169c6104ce56","Type":"ContainerStarted","Data":"baed098a69878751166eba110018725dc271d8ffc557e40cfc38922072195360"}
Nov 03 20:03:17.677335 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:17.677307    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-4dmkq" event={"ID":"f21cf70e-682f-490c-bda5-3954df150487","Type":"ContainerStarted","Data":"2d654bb89ba8785ae38bae15a38c8508d4891aae7028d40ed527a9aa6702b0d2"}
Nov 03 20:03:17.683816 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:17.683732    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-9s4wk" event={"ID":"2073a8c8-6f07-4817-b7b3-468b327e2712","Type":"ContainerStarted","Data":"fdfee0f501f2c3bd7ac1c00b1dfd5135480bcf94c3cc699f4bc1c05502ba7e57"}
Nov 03 20:03:17.693676 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:17.693616    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-h6p95" event={"ID":"682463ee-d9c9-4e9b-82b7-f1116898debf","Type":"ContainerStarted","Data":"0e73128cc29efba8f4b0ef35efead17c585a7fb5ac4db5f96fbfcde7e777d2d5"}
Nov 03 20:03:17.704309 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:17.704234    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9f5s5" event={"ID":"65008653-586d-4a76-a53f-7efe1f6d839b","Type":"ContainerStarted","Data":"8dd1760f632c5c28135a89fef44e119cf12f4fd22c927b2490d8a4995b99427a"}
Nov 03 20:03:18.223837 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:18.223542    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/51dffd76-5c30-440c-85ed-d81221663833-original-pull-secret\") pod \"global-pull-secret-syncer-j4jzv\" (UID: \"51dffd76-5c30-440c-85ed-d81221663833\") " pod="kube-system/global-pull-secret-syncer-j4jzv"
Nov 03 20:03:18.224055 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:18.223891    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c92a5d0-b34f-46a1-8f8f-a9ae3cb8d5a5-metrics-certs\") pod \"network-metrics-daemon-c947w\" (UID: \"4c92a5d0-b34f-46a1-8f8f-a9ae3cb8d5a5\") " pod="openshift-multus/network-metrics-daemon-c947w"
Nov 03 20:03:18.224156 ip-10-0-132-4 kubenswrapper[2577]: E1103 20:03:18.224060    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 20:03:18.224156 ip-10-0-132-4 kubenswrapper[2577]: E1103 20:03:18.224128    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c92a5d0-b34f-46a1-8f8f-a9ae3cb8d5a5-metrics-certs podName:4c92a5d0-b34f-46a1-8f8f-a9ae3cb8d5a5 nodeName:}" failed. No retries permitted until 2026-11-03 20:03:20.224107057 +0000 UTC m=+6.084219587 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4c92a5d0-b34f-46a1-8f8f-a9ae3cb8d5a5-metrics-certs") pod "network-metrics-daemon-c947w" (UID: "4c92a5d0-b34f-46a1-8f8f-a9ae3cb8d5a5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 20:03:18.224271 ip-10-0-132-4 kubenswrapper[2577]: E1103 20:03:18.224197    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 20:03:18.224271 ip-10-0-132-4 kubenswrapper[2577]: E1103 20:03:18.224231    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51dffd76-5c30-440c-85ed-d81221663833-original-pull-secret podName:51dffd76-5c30-440c-85ed-d81221663833 nodeName:}" failed. No retries permitted until 2026-11-03 20:03:20.224219989 +0000 UTC m=+6.084332521 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/51dffd76-5c30-440c-85ed-d81221663833-original-pull-secret") pod "global-pull-secret-syncer-j4jzv" (UID: "51dffd76-5c30-440c-85ed-d81221663833") : object "kube-system"/"original-pull-secret" not registered
Nov 03 20:03:18.325207 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:18.324522    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcxdr\" (UniqueName: \"kubernetes.io/projected/d71eaa2a-9659-4827-9dfc-c1fce02565cb-kube-api-access-zcxdr\") pod \"network-check-target-6dfr5\" (UID: \"d71eaa2a-9659-4827-9dfc-c1fce02565cb\") " pod="openshift-network-diagnostics/network-check-target-6dfr5"
Nov 03 20:03:18.325207 ip-10-0-132-4 kubenswrapper[2577]: E1103 20:03:18.324703    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 20:03:18.325207 ip-10-0-132-4 kubenswrapper[2577]: E1103 20:03:18.324729    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 20:03:18.325207 ip-10-0-132-4 kubenswrapper[2577]: E1103 20:03:18.324743    2577 projected.go:196] Error preparing data for projected volume kube-api-access-zcxdr for pod openshift-network-diagnostics/network-check-target-6dfr5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 20:03:18.325207 ip-10-0-132-4 kubenswrapper[2577]: E1103 20:03:18.324807    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d71eaa2a-9659-4827-9dfc-c1fce02565cb-kube-api-access-zcxdr podName:d71eaa2a-9659-4827-9dfc-c1fce02565cb nodeName:}" failed. No retries permitted until 2026-11-03 20:03:20.324787455 +0000 UTC m=+6.184899980 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-zcxdr" (UniqueName: "kubernetes.io/projected/d71eaa2a-9659-4827-9dfc-c1fce02565cb-kube-api-access-zcxdr") pod "network-check-target-6dfr5" (UID: "d71eaa2a-9659-4827-9dfc-c1fce02565cb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 20:03:18.544339 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:18.544187    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 19:58:16 +0000 UTC" deadline="2028-07-21 06:23:24.41079893 +0000 UTC"
Nov 03 20:03:18.544339 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:18.544252    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15010h20m5.866551151s"
Nov 03 20:03:18.617682 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:18.617647    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6dfr5"
Nov 03 20:03:18.617881 ip-10-0-132-4 kubenswrapper[2577]: E1103 20:03:18.617791    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6dfr5" podUID="d71eaa2a-9659-4827-9dfc-c1fce02565cb"
Nov 03 20:03:18.618196 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:18.618177    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j4jzv"
Nov 03 20:03:18.618307 ip-10-0-132-4 kubenswrapper[2577]: E1103 20:03:18.618287    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j4jzv" podUID="51dffd76-5c30-440c-85ed-d81221663833"
Nov 03 20:03:18.618382 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:18.618357    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c947w"
Nov 03 20:03:18.618452 ip-10-0-132-4 kubenswrapper[2577]: E1103 20:03:18.618434    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c947w" podUID="4c92a5d0-b34f-46a1-8f8f-a9ae3cb8d5a5"
Nov 03 20:03:18.720953 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:18.720876    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-4.ec2.internal" event={"ID":"5dff8c1e3d20edc13c5947130e8a5c14","Type":"ContainerStarted","Data":"f9e1beb267e03b7b0f87971a0c028bb780f4b92d5f92314d8290765e99268d5a"}
Nov 03 20:03:18.753746 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:18.753675    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-4.ec2.internal" podStartSLOduration=3.753654001 podStartE2EDuration="3.753654001s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:03:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:03:17.713392346 +0000 UTC m=+3.573504887" watchObservedRunningTime="2026-11-03 20:03:18.753654001 +0000 UTC m=+4.613766539"
Nov 03 20:03:19.726846 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:19.726811    2577 generic.go:356] "Generic (PLEG): container finished" podID="db895326-506d-479d-a526-6bf71be7f967" containerID="a701c1ac8d52e702cf58a7a8a4c2499cd44c210eb94b4dd1f0b6a71b821668a4" exitCode=0
Nov 03 20:03:19.727893 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:19.727864    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mhtmw" event={"ID":"db895326-506d-479d-a526-6bf71be7f967","Type":"ContainerDied","Data":"a701c1ac8d52e702cf58a7a8a4c2499cd44c210eb94b4dd1f0b6a71b821668a4"}
Nov 03 20:03:19.785685 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:19.785011    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-4.ec2.internal" podStartSLOduration=4.78499606 podStartE2EDuration="4.78499606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:03:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:03:18.754146824 +0000 UTC m=+4.614259360" watchObservedRunningTime="2026-11-03 20:03:19.78499606 +0000 UTC m=+5.645108595"
Nov 03 20:03:20.240386 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:20.240346    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/51dffd76-5c30-440c-85ed-d81221663833-original-pull-secret\") pod \"global-pull-secret-syncer-j4jzv\" (UID: \"51dffd76-5c30-440c-85ed-d81221663833\") " pod="kube-system/global-pull-secret-syncer-j4jzv"
Nov 03 20:03:20.240591 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:20.240426    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c92a5d0-b34f-46a1-8f8f-a9ae3cb8d5a5-metrics-certs\") pod \"network-metrics-daemon-c947w\" (UID: \"4c92a5d0-b34f-46a1-8f8f-a9ae3cb8d5a5\") " pod="openshift-multus/network-metrics-daemon-c947w"
Nov 03 20:03:20.240661 ip-10-0-132-4 kubenswrapper[2577]: E1103 20:03:20.240590    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 20:03:20.240661 ip-10-0-132-4 kubenswrapper[2577]: E1103 20:03:20.240652    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c92a5d0-b34f-46a1-8f8f-a9ae3cb8d5a5-metrics-certs podName:4c92a5d0-b34f-46a1-8f8f-a9ae3cb8d5a5 nodeName:}" failed. No retries permitted until 2026-11-03 20:03:24.240634874 +0000 UTC m=+10.100747392 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4c92a5d0-b34f-46a1-8f8f-a9ae3cb8d5a5-metrics-certs") pod "network-metrics-daemon-c947w" (UID: "4c92a5d0-b34f-46a1-8f8f-a9ae3cb8d5a5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 20:03:20.241105 ip-10-0-132-4 kubenswrapper[2577]: E1103 20:03:20.241060    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 20:03:20.241204 ip-10-0-132-4 kubenswrapper[2577]: E1103 20:03:20.241119    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51dffd76-5c30-440c-85ed-d81221663833-original-pull-secret podName:51dffd76-5c30-440c-85ed-d81221663833 nodeName:}" failed. No retries permitted until 2026-11-03 20:03:24.24110103 +0000 UTC m=+10.101213562 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/51dffd76-5c30-440c-85ed-d81221663833-original-pull-secret") pod "global-pull-secret-syncer-j4jzv" (UID: "51dffd76-5c30-440c-85ed-d81221663833") : object "kube-system"/"original-pull-secret" not registered
Nov 03 20:03:20.341039 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:20.340999    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcxdr\" (UniqueName: \"kubernetes.io/projected/d71eaa2a-9659-4827-9dfc-c1fce02565cb-kube-api-access-zcxdr\") pod \"network-check-target-6dfr5\" (UID: \"d71eaa2a-9659-4827-9dfc-c1fce02565cb\") " pod="openshift-network-diagnostics/network-check-target-6dfr5"
Nov 03 20:03:20.341223 ip-10-0-132-4 kubenswrapper[2577]: E1103 20:03:20.341190    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 20:03:20.341223 ip-10-0-132-4 kubenswrapper[2577]: E1103 20:03:20.341210    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 20:03:20.341223 ip-10-0-132-4 kubenswrapper[2577]: E1103 20:03:20.341224    2577 projected.go:196] Error preparing data for projected volume kube-api-access-zcxdr for pod openshift-network-diagnostics/network-check-target-6dfr5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 20:03:20.341377 ip-10-0-132-4 kubenswrapper[2577]: E1103 20:03:20.341282    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d71eaa2a-9659-4827-9dfc-c1fce02565cb-kube-api-access-zcxdr podName:d71eaa2a-9659-4827-9dfc-c1fce02565cb nodeName:}" failed. No retries permitted until 2026-11-03 20:03:24.341264399 +0000 UTC m=+10.201376930 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-zcxdr" (UniqueName: "kubernetes.io/projected/d71eaa2a-9659-4827-9dfc-c1fce02565cb-kube-api-access-zcxdr") pod "network-check-target-6dfr5" (UID: "d71eaa2a-9659-4827-9dfc-c1fce02565cb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 20:03:20.616341 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:20.616270    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j4jzv"
Nov 03 20:03:20.616341 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:20.616288    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c947w"
Nov 03 20:03:20.616557 ip-10-0-132-4 kubenswrapper[2577]: E1103 20:03:20.616412    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j4jzv" podUID="51dffd76-5c30-440c-85ed-d81221663833"
Nov 03 20:03:20.616557 ip-10-0-132-4 kubenswrapper[2577]: E1103 20:03:20.616515    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c947w" podUID="4c92a5d0-b34f-46a1-8f8f-a9ae3cb8d5a5"
Nov 03 20:03:20.616663 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:20.616571    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6dfr5"
Nov 03 20:03:20.616663 ip-10-0-132-4 kubenswrapper[2577]: E1103 20:03:20.616640    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6dfr5" podUID="d71eaa2a-9659-4827-9dfc-c1fce02565cb"
Nov 03 20:03:21.745534 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:21.745494    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-5p48g" event={"ID":"15b5070f-d0d3-4ad8-aa1d-93cc7b96003a","Type":"ContainerStarted","Data":"0eb869fd9b8fe782667a88a95f4fd1ce24ae4b0953982b3eb2c045030a7015cf"}
Nov 03 20:03:22.229115 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:22.229073    2577 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-5p48g"
Nov 03 20:03:22.229986 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:22.229717    2577 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-5p48g"
Nov 03 20:03:22.262658 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:22.262547    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-5p48g" podStartSLOduration=3.364906231 podStartE2EDuration="7.262527883s" totalImagesPullingTime="3.897621652s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:03:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:03:21.779505632 +0000 UTC m=+7.639618173" watchObservedRunningTime="2026-11-03 20:03:22.262527883 +0000 UTC m=+8.122640434"
Nov 03 20:03:22.616475 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:22.615971    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6dfr5"
Nov 03 20:03:22.616475 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:22.616039    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c947w"
Nov 03 20:03:22.616475 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:22.616051    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j4jzv"
Nov 03 20:03:22.616475 ip-10-0-132-4 kubenswrapper[2577]: E1103 20:03:22.616151    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j4jzv" podUID="51dffd76-5c30-440c-85ed-d81221663833"
Nov 03 20:03:22.616475 ip-10-0-132-4 kubenswrapper[2577]: E1103 20:03:22.616274    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6dfr5" podUID="d71eaa2a-9659-4827-9dfc-c1fce02565cb"
Nov 03 20:03:22.616475 ip-10-0-132-4 kubenswrapper[2577]: E1103 20:03:22.616364    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c947w" podUID="4c92a5d0-b34f-46a1-8f8f-a9ae3cb8d5a5"
Nov 03 20:03:22.751338 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:22.751305    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9f5s5" event={"ID":"65008653-586d-4a76-a53f-7efe1f6d839b","Type":"ContainerStarted","Data":"65357bfe580436b2c57969ed070abc39cb32058fc0d299ccf4efa82a6d143c1b"}
Nov 03 20:03:22.760978 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:22.760908    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-zzt8t" event={"ID":"1d4ce60c-e058-4c29-99e8-5f4f6eead744","Type":"ContainerStarted","Data":"167d64129b7433293129f321af718404cc06a64a3c796a4c7e20a8436ec2040e"}
Nov 03 20:03:23.763699 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:23.763361    2577 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 20:03:24.273111 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:24.273078    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c92a5d0-b34f-46a1-8f8f-a9ae3cb8d5a5-metrics-certs\") pod \"network-metrics-daemon-c947w\" (UID: \"4c92a5d0-b34f-46a1-8f8f-a9ae3cb8d5a5\") " pod="openshift-multus/network-metrics-daemon-c947w"
Nov 03 20:03:24.273220 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:24.273163    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/51dffd76-5c30-440c-85ed-d81221663833-original-pull-secret\") pod \"global-pull-secret-syncer-j4jzv\" (UID: \"51dffd76-5c30-440c-85ed-d81221663833\") " pod="kube-system/global-pull-secret-syncer-j4jzv"
Nov 03 20:03:24.273313 ip-10-0-132-4 kubenswrapper[2577]: E1103 20:03:24.273292    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 20:03:24.273384 ip-10-0-132-4 kubenswrapper[2577]: E1103 20:03:24.273345    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51dffd76-5c30-440c-85ed-d81221663833-original-pull-secret podName:51dffd76-5c30-440c-85ed-d81221663833 nodeName:}" failed. No retries permitted until 2026-11-03 20:03:32.273331496 +0000 UTC m=+18.133444010 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/51dffd76-5c30-440c-85ed-d81221663833-original-pull-secret") pod "global-pull-secret-syncer-j4jzv" (UID: "51dffd76-5c30-440c-85ed-d81221663833") : object "kube-system"/"original-pull-secret" not registered
Nov 03 20:03:24.273887 ip-10-0-132-4 kubenswrapper[2577]: E1103 20:03:24.273626    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 20:03:24.273887 ip-10-0-132-4 kubenswrapper[2577]: E1103 20:03:24.273674    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c92a5d0-b34f-46a1-8f8f-a9ae3cb8d5a5-metrics-certs podName:4c92a5d0-b34f-46a1-8f8f-a9ae3cb8d5a5 nodeName:}" failed. No retries permitted until 2026-11-03 20:03:32.273659374 +0000 UTC m=+18.133771908 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4c92a5d0-b34f-46a1-8f8f-a9ae3cb8d5a5-metrics-certs") pod "network-metrics-daemon-c947w" (UID: "4c92a5d0-b34f-46a1-8f8f-a9ae3cb8d5a5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 20:03:24.374975 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:24.374061    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcxdr\" (UniqueName: \"kubernetes.io/projected/d71eaa2a-9659-4827-9dfc-c1fce02565cb-kube-api-access-zcxdr\") pod \"network-check-target-6dfr5\" (UID: \"d71eaa2a-9659-4827-9dfc-c1fce02565cb\") " pod="openshift-network-diagnostics/network-check-target-6dfr5"
Nov 03 20:03:24.374975 ip-10-0-132-4 kubenswrapper[2577]: E1103 20:03:24.374255    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 20:03:24.374975 ip-10-0-132-4 kubenswrapper[2577]: E1103 20:03:24.374278    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 20:03:24.374975 ip-10-0-132-4 kubenswrapper[2577]: E1103 20:03:24.374290    2577 projected.go:196] Error preparing data for projected volume kube-api-access-zcxdr for pod openshift-network-diagnostics/network-check-target-6dfr5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 20:03:24.374975 ip-10-0-132-4 kubenswrapper[2577]: E1103 20:03:24.374348    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d71eaa2a-9659-4827-9dfc-c1fce02565cb-kube-api-access-zcxdr podName:d71eaa2a-9659-4827-9dfc-c1fce02565cb nodeName:}" failed. No retries permitted until 2026-11-03 20:03:32.374330314 +0000 UTC m=+18.234442847 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-zcxdr" (UniqueName: "kubernetes.io/projected/d71eaa2a-9659-4827-9dfc-c1fce02565cb-kube-api-access-zcxdr") pod "network-check-target-6dfr5" (UID: "d71eaa2a-9659-4827-9dfc-c1fce02565cb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 20:03:24.432340 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:24.432313    2577 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 03 20:03:24.616957 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:24.616927    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6dfr5"
Nov 03 20:03:24.617132 ip-10-0-132-4 kubenswrapper[2577]: E1103 20:03:24.617062    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6dfr5" podUID="d71eaa2a-9659-4827-9dfc-c1fce02565cb"
Nov 03 20:03:24.617469 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:24.617446    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j4jzv"
Nov 03 20:03:24.617562 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:24.617484    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c947w"
Nov 03 20:03:24.617621 ip-10-0-132-4 kubenswrapper[2577]: E1103 20:03:24.617559    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j4jzv" podUID="51dffd76-5c30-440c-85ed-d81221663833"
Nov 03 20:03:24.618549 ip-10-0-132-4 kubenswrapper[2577]: E1103 20:03:24.618523    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c947w" podUID="4c92a5d0-b34f-46a1-8f8f-a9ae3cb8d5a5"
Nov 03 20:03:24.704613 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:24.704526    2577 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T20:03:24.432336813Z","UUID":"c98fbfb4-05ee-40c4-b9bf-3c664ccf1912","Handler":null,"Name":"","Endpoint":""}
Nov 03 20:03:24.707974 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:24.707948    2577 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 03 20:03:24.708112 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:24.707982    2577 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 03 20:03:24.766585 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:24.766545    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-h6p95" event={"ID":"682463ee-d9c9-4e9b-82b7-f1116898debf","Type":"ContainerStarted","Data":"40d21ecbab92d56015ebc14d360210b19f5b2f874eaa478cae04f785834710a9"}
Nov 03 20:03:24.772654 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:24.772610    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9f5s5" event={"ID":"65008653-586d-4a76-a53f-7efe1f6d839b","Type":"ContainerStarted","Data":"6888005494f2fceaae88c32f6b3a50bd8413c80f15697a434d6729bd52ba1e29"}
Nov 03 20:03:24.776081 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:24.776048    2577 generic.go:356] "Generic (PLEG): container finished" podID="db895326-506d-479d-a526-6bf71be7f967" containerID="f3db6f558a5b35961c39f860bf189899d8c34ed67dbb5a8263c3eb7c4119e2f7" exitCode=0
Nov 03 20:03:24.776203 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:24.776100    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mhtmw" event={"ID":"db895326-506d-479d-a526-6bf71be7f967","Type":"ContainerDied","Data":"f3db6f558a5b35961c39f860bf189899d8c34ed67dbb5a8263c3eb7c4119e2f7"}
Nov 03 20:03:24.802123 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:24.802072    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-zzt8t" podStartSLOduration=4.806255989 podStartE2EDuration="9.802059207s" totalImagesPullingTime="4.995803218s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:03:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:03:22.793053532 +0000 UTC m=+8.653166068" watchObservedRunningTime="2026-11-03 20:03:24.802059207 +0000 UTC m=+10.662171742"
Nov 03 20:03:24.802543 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:24.802510    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-h6p95" podStartSLOduration=2.4091861 podStartE2EDuration="9.802502184s" totalImagesPullingTime="7.393316084s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:03:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:03:24.80186777 +0000 UTC m=+10.661980306" watchObservedRunningTime="2026-11-03 20:03:24.802502184 +0000 UTC m=+10.662614720"
Nov 03 20:03:25.780289 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:25.780012    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-t6dqw" event={"ID":"716cc9cf-d26a-481b-8114-169c6104ce56","Type":"ContainerStarted","Data":"386a2f82280e01eed6f201c6188b2634369b1cbcac13b98942292708d4e899fe"}
Nov 03 20:03:25.782762 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:25.782731    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9f5s5" event={"ID":"65008653-586d-4a76-a53f-7efe1f6d839b","Type":"ContainerStarted","Data":"a8751cff5ee54713352a2e8c08287dabc0bddb95ece27611046f8f865f2ffcc2"}
Nov 03 20:03:25.785312 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:25.785287    2577 generic.go:356] "Generic (PLEG): container finished" podID="db895326-506d-479d-a526-6bf71be7f967" containerID="18e7de51ee6637ad7c2c65b56530df6fba343d1141603e76d8231e46da4d68ed" exitCode=0
Nov 03 20:03:25.785426 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:25.785397    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mhtmw" event={"ID":"db895326-506d-479d-a526-6bf71be7f967","Type":"ContainerDied","Data":"18e7de51ee6637ad7c2c65b56530df6fba343d1141603e76d8231e46da4d68ed"}
Nov 03 20:03:25.813249 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:25.813202    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-t6dqw" podStartSLOduration=2.819666566 podStartE2EDuration="10.813185423s" totalImagesPullingTime="7.993518857s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:03:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:03:25.813005208 +0000 UTC m=+11.673117745" watchObservedRunningTime="2026-11-03 20:03:25.813185423 +0000 UTC m=+11.673297960"
Nov 03 20:03:26.615643 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:26.615598    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c947w"
Nov 03 20:03:26.615643 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:26.615620    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j4jzv"
Nov 03 20:03:26.615872 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:26.615695    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6dfr5"
Nov 03 20:03:26.615872 ip-10-0-132-4 kubenswrapper[2577]: E1103 20:03:26.615803    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6dfr5" podUID="d71eaa2a-9659-4827-9dfc-c1fce02565cb"
Nov 03 20:03:26.616242 ip-10-0-132-4 kubenswrapper[2577]: E1103 20:03:26.616217    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j4jzv" podUID="51dffd76-5c30-440c-85ed-d81221663833"
Nov 03 20:03:26.616328 ip-10-0-132-4 kubenswrapper[2577]: E1103 20:03:26.616302    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c947w" podUID="4c92a5d0-b34f-46a1-8f8f-a9ae3cb8d5a5"
Nov 03 20:03:26.790080 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:26.789997    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-9s4wk" event={"ID":"2073a8c8-6f07-4817-b7b3-468b327e2712","Type":"ContainerStarted","Data":"fc93d838c015b7bd111c846f9d8591367a87bd1ff0fe367d9c8f056e05facf26"}
Nov 03 20:03:26.821367 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:26.821311    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9f5s5" podStartSLOduration=3.614503946 podStartE2EDuration="11.821295885s" totalImagesPullingTime="8.206791939s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:03:15 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:03:25.883182064 +0000 UTC m=+11.743294601" watchObservedRunningTime="2026-11-03 20:03:26.821295885 +0000 UTC m=+12.681408442"
Nov 03 20:03:26.821623 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:26.821592    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-9s4wk" podStartSLOduration=3.88556807 podStartE2EDuration="11.821585169s" totalImagesPullingTime="7.936017099s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:03:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:03:26.821125331 +0000 UTC m=+12.681237868" watchObservedRunningTime="2026-11-03 20:03:26.821585169 +0000 UTC m=+12.681697704"
Nov 03 20:03:27.794337 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:27.794295    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-4dmkq" event={"ID":"f21cf70e-682f-490c-bda5-3954df150487","Type":"ContainerStarted","Data":"065e90e31cc12212e17e99458f0453fdcff20e48e2c0f19af3d4e4154f3fa80f"}
Nov 03 20:03:27.797080 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:27.797049    2577 generic.go:356] "Generic (PLEG): container finished" podID="db895326-506d-479d-a526-6bf71be7f967" containerID="214cfdccb0b4051ea1e1eed79c0796d2215b432938f46ad5c6bc09d398b15f6c" exitCode=0
Nov 03 20:03:27.797215 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:27.797168    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mhtmw" event={"ID":"db895326-506d-479d-a526-6bf71be7f967","Type":"ContainerDied","Data":"214cfdccb0b4051ea1e1eed79c0796d2215b432938f46ad5c6bc09d398b15f6c"}
Nov 03 20:03:27.832309 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:27.832248    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-4dmkq" podStartSLOduration=2.615638412 podStartE2EDuration="12.83222859s" totalImagesPullingTime="10.216590178s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:03:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:03:27.831767626 +0000 UTC m=+13.691880162" watchObservedRunningTime="2026-11-03 20:03:27.83222859 +0000 UTC m=+13.692341127"
Nov 03 20:03:28.615730 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:28.615697    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6dfr5"
Nov 03 20:03:28.615924 ip-10-0-132-4 kubenswrapper[2577]: E1103 20:03:28.615840    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6dfr5" podUID="d71eaa2a-9659-4827-9dfc-c1fce02565cb"
Nov 03 20:03:28.615924 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:28.615695    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j4jzv"
Nov 03 20:03:28.616049 ip-10-0-132-4 kubenswrapper[2577]: E1103 20:03:28.616022    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j4jzv" podUID="51dffd76-5c30-440c-85ed-d81221663833"
Nov 03 20:03:28.616049 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:28.615697    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c947w"
Nov 03 20:03:28.616225 ip-10-0-132-4 kubenswrapper[2577]: E1103 20:03:28.616134    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c947w" podUID="4c92a5d0-b34f-46a1-8f8f-a9ae3cb8d5a5"
Nov 03 20:03:29.804281 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:29.804241    2577 generic.go:356] "Generic (PLEG): container finished" podID="db895326-506d-479d-a526-6bf71be7f967" containerID="c3dc107a0c3b728fd541122d7dd467b9ed50d1642039d41bfe2634b0427a7687" exitCode=0
Nov 03 20:03:29.804972 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:29.804286    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mhtmw" event={"ID":"db895326-506d-479d-a526-6bf71be7f967","Type":"ContainerDied","Data":"c3dc107a0c3b728fd541122d7dd467b9ed50d1642039d41bfe2634b0427a7687"}
Nov 03 20:03:30.616340 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:30.616108    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j4jzv"
Nov 03 20:03:30.616577 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:30.616106    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c947w"
Nov 03 20:03:30.616577 ip-10-0-132-4 kubenswrapper[2577]: E1103 20:03:30.616452    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j4jzv" podUID="51dffd76-5c30-440c-85ed-d81221663833"
Nov 03 20:03:30.616577 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:30.616144    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6dfr5"
Nov 03 20:03:30.616577 ip-10-0-132-4 kubenswrapper[2577]: E1103 20:03:30.616527    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c947w" podUID="4c92a5d0-b34f-46a1-8f8f-a9ae3cb8d5a5"
Nov 03 20:03:30.616804 ip-10-0-132-4 kubenswrapper[2577]: E1103 20:03:30.616619    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6dfr5" podUID="d71eaa2a-9659-4827-9dfc-c1fce02565cb"
Nov 03 20:03:30.809098 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:30.809065    2577 generic.go:356] "Generic (PLEG): container finished" podID="db895326-506d-479d-a526-6bf71be7f967" containerID="f67c692909431ab5ef63756f63292003698c2a636cd7f4307477e9439029858a" exitCode=0
Nov 03 20:03:30.809531 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:30.809111    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mhtmw" event={"ID":"db895326-506d-479d-a526-6bf71be7f967","Type":"ContainerDied","Data":"f67c692909431ab5ef63756f63292003698c2a636cd7f4307477e9439029858a"}
Nov 03 20:03:31.815400 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:31.815297    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mhtmw" event={"ID":"db895326-506d-479d-a526-6bf71be7f967","Type":"ContainerStarted","Data":"5dea8fc26d053b0cfb371c09b1675f1a9e4496ed85bb4c9c9f6dcb760faaab11"}
Nov 03 20:03:31.818146 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:31.818125    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6sv7j" event={"ID":"9126a982-59c9-4cec-a665-5f1bc0ae3389","Type":"ContainerStarted","Data":"45eee5963edc90a62697e9ee6149ded18a8e99c374b32d04487a58442c52245a"}
Nov 03 20:03:31.818230 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:31.818185    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6sv7j" event={"ID":"9126a982-59c9-4cec-a665-5f1bc0ae3389","Type":"ContainerStarted","Data":"494f132601b6d060ac102ba018ee6211bf552c186be106f38f6cdb8c7edffeec"}
Nov 03 20:03:31.818230 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:31.818217    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6sv7j" event={"ID":"9126a982-59c9-4cec-a665-5f1bc0ae3389","Type":"ContainerStarted","Data":"6c1d0b603c2cbbf8ab9fc15cfe403c958f9c5a8380f266160b99de48b9f4bc09"}
Nov 03 20:03:31.818303 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:31.818276    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6sv7j" event={"ID":"9126a982-59c9-4cec-a665-5f1bc0ae3389","Type":"ContainerStarted","Data":"5b2d838ded11c8138a369fa5c043a88b0a2e2fcce56a35346cbd895849b2d80a"}
Nov 03 20:03:31.818331 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:31.818304    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6sv7j" event={"ID":"9126a982-59c9-4cec-a665-5f1bc0ae3389","Type":"ContainerStarted","Data":"0e93240f28733b2045fc6a323aaec073fa904d3e7ae5957a21750abd0670a4fd"}
Nov 03 20:03:31.818358 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:31.818334    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6sv7j" event={"ID":"9126a982-59c9-4cec-a665-5f1bc0ae3389","Type":"ContainerStarted","Data":"0a4f3a42f6a94ea17fa961e5383c9624123528fb267b5fbfc797a6ba5f08c900"}
Nov 03 20:03:31.871999 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:31.871946    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-mhtmw" podStartSLOduration=6.349128524 podStartE2EDuration="16.871931314s" totalImagesPullingTime="10.52280279s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:03:15 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:03:31.861292046 +0000 UTC m=+17.721404583" watchObservedRunningTime="2026-11-03 20:03:31.871931314 +0000 UTC m=+17.732043850"
Nov 03 20:03:31.954902 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:31.954874    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-5p48g"
Nov 03 20:03:31.955101 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:31.955007    2577 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 20:03:31.955498 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:31.955482    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-5p48g"
Nov 03 20:03:32.330200 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:32.330163    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c92a5d0-b34f-46a1-8f8f-a9ae3cb8d5a5-metrics-certs\") pod \"network-metrics-daemon-c947w\" (UID: \"4c92a5d0-b34f-46a1-8f8f-a9ae3cb8d5a5\") " pod="openshift-multus/network-metrics-daemon-c947w"
Nov 03 20:03:32.330347 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:32.330217    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/51dffd76-5c30-440c-85ed-d81221663833-original-pull-secret\") pod \"global-pull-secret-syncer-j4jzv\" (UID: \"51dffd76-5c30-440c-85ed-d81221663833\") " pod="kube-system/global-pull-secret-syncer-j4jzv"
Nov 03 20:03:32.330347 ip-10-0-132-4 kubenswrapper[2577]: E1103 20:03:32.330310    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 20:03:32.330404 ip-10-0-132-4 kubenswrapper[2577]: E1103 20:03:32.330370    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c92a5d0-b34f-46a1-8f8f-a9ae3cb8d5a5-metrics-certs podName:4c92a5d0-b34f-46a1-8f8f-a9ae3cb8d5a5 nodeName:}" failed. No retries permitted until 2026-11-03 20:03:48.330357176 +0000 UTC m=+34.190469694 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4c92a5d0-b34f-46a1-8f8f-a9ae3cb8d5a5-metrics-certs") pod "network-metrics-daemon-c947w" (UID: "4c92a5d0-b34f-46a1-8f8f-a9ae3cb8d5a5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 20:03:32.330404 ip-10-0-132-4 kubenswrapper[2577]: E1103 20:03:32.330318    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 20:03:32.330488 ip-10-0-132-4 kubenswrapper[2577]: E1103 20:03:32.330435    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51dffd76-5c30-440c-85ed-d81221663833-original-pull-secret podName:51dffd76-5c30-440c-85ed-d81221663833 nodeName:}" failed. No retries permitted until 2026-11-03 20:03:48.330425117 +0000 UTC m=+34.190537635 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/51dffd76-5c30-440c-85ed-d81221663833-original-pull-secret") pod "global-pull-secret-syncer-j4jzv" (UID: "51dffd76-5c30-440c-85ed-d81221663833") : object "kube-system"/"original-pull-secret" not registered
Nov 03 20:03:32.430814 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:32.430760    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcxdr\" (UniqueName: \"kubernetes.io/projected/d71eaa2a-9659-4827-9dfc-c1fce02565cb-kube-api-access-zcxdr\") pod \"network-check-target-6dfr5\" (UID: \"d71eaa2a-9659-4827-9dfc-c1fce02565cb\") " pod="openshift-network-diagnostics/network-check-target-6dfr5"
Nov 03 20:03:32.430962 ip-10-0-132-4 kubenswrapper[2577]: E1103 20:03:32.430920    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 20:03:32.430962 ip-10-0-132-4 kubenswrapper[2577]: E1103 20:03:32.430942    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 20:03:32.430962 ip-10-0-132-4 kubenswrapper[2577]: E1103 20:03:32.430951    2577 projected.go:196] Error preparing data for projected volume kube-api-access-zcxdr for pod openshift-network-diagnostics/network-check-target-6dfr5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 20:03:32.431076 ip-10-0-132-4 kubenswrapper[2577]: E1103 20:03:32.430999    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d71eaa2a-9659-4827-9dfc-c1fce02565cb-kube-api-access-zcxdr podName:d71eaa2a-9659-4827-9dfc-c1fce02565cb nodeName:}" failed. No retries permitted until 2026-11-03 20:03:48.430987013 +0000 UTC m=+34.291099548 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-zcxdr" (UniqueName: "kubernetes.io/projected/d71eaa2a-9659-4827-9dfc-c1fce02565cb-kube-api-access-zcxdr") pod "network-check-target-6dfr5" (UID: "d71eaa2a-9659-4827-9dfc-c1fce02565cb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 20:03:32.615169 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:32.615081    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6dfr5"
Nov 03 20:03:32.615169 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:32.615104    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j4jzv"
Nov 03 20:03:32.615169 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:32.615171    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c947w"
Nov 03 20:03:32.615425 ip-10-0-132-4 kubenswrapper[2577]: E1103 20:03:32.615257    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6dfr5" podUID="d71eaa2a-9659-4827-9dfc-c1fce02565cb"
Nov 03 20:03:32.615425 ip-10-0-132-4 kubenswrapper[2577]: E1103 20:03:32.615336    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c947w" podUID="4c92a5d0-b34f-46a1-8f8f-a9ae3cb8d5a5"
Nov 03 20:03:32.615509 ip-10-0-132-4 kubenswrapper[2577]: E1103 20:03:32.615436    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j4jzv" podUID="51dffd76-5c30-440c-85ed-d81221663833"
Nov 03 20:03:33.829405 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:33.829312    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6sv7j" event={"ID":"9126a982-59c9-4cec-a665-5f1bc0ae3389","Type":"ContainerStarted","Data":"ba9873d94510a4f8e0d797de2dd023bee96f3c285913edee22df555128e7e480"}
Nov 03 20:03:34.616768 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:34.616732    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6dfr5"
Nov 03 20:03:34.616933 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:34.616882    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j4jzv"
Nov 03 20:03:34.616933 ip-10-0-132-4 kubenswrapper[2577]: E1103 20:03:34.616895    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6dfr5" podUID="d71eaa2a-9659-4827-9dfc-c1fce02565cb"
Nov 03 20:03:34.617014 ip-10-0-132-4 kubenswrapper[2577]: E1103 20:03:34.616994    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j4jzv" podUID="51dffd76-5c30-440c-85ed-d81221663833"
Nov 03 20:03:34.617075 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:34.617063    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c947w"
Nov 03 20:03:34.617166 ip-10-0-132-4 kubenswrapper[2577]: E1103 20:03:34.617150    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c947w" podUID="4c92a5d0-b34f-46a1-8f8f-a9ae3cb8d5a5"
Nov 03 20:03:35.837813 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:35.837760    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6sv7j" event={"ID":"9126a982-59c9-4cec-a665-5f1bc0ae3389","Type":"ContainerStarted","Data":"7da2305a68f0818a713243649193cb96a6a190142586c0327e7c431082081ecc"}
Nov 03 20:03:35.838618 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:35.838032    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6sv7j"
Nov 03 20:03:35.852747 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:35.852715    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6sv7j"
Nov 03 20:03:35.907570 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:35.907482    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-6sv7j" podStartSLOduration=6.609765354 podStartE2EDuration="20.907470012s" totalImagesPullingTime="14.297704658s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:03:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:03:35.90729612 +0000 UTC m=+21.767408662" watchObservedRunningTime="2026-11-03 20:03:35.907470012 +0000 UTC m=+21.767582548"
Nov 03 20:03:36.615547 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:36.615515    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6dfr5"
Nov 03 20:03:36.615547 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:36.615537    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c947w"
Nov 03 20:03:36.615933 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:36.615595    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j4jzv"
Nov 03 20:03:36.615933 ip-10-0-132-4 kubenswrapper[2577]: E1103 20:03:36.615673    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6dfr5" podUID="d71eaa2a-9659-4827-9dfc-c1fce02565cb"
Nov 03 20:03:36.615933 ip-10-0-132-4 kubenswrapper[2577]: E1103 20:03:36.615802    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j4jzv" podUID="51dffd76-5c30-440c-85ed-d81221663833"
Nov 03 20:03:36.615933 ip-10-0-132-4 kubenswrapper[2577]: E1103 20:03:36.615910    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c947w" podUID="4c92a5d0-b34f-46a1-8f8f-a9ae3cb8d5a5"
Nov 03 20:03:36.840441 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:36.840414    2577 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 20:03:36.840873 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:36.840854    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6sv7j"
Nov 03 20:03:36.855631 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:36.855606    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6sv7j"
Nov 03 20:03:37.280899 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:37.280873    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-t6dqw_716cc9cf-d26a-481b-8114-169c6104ce56/dns-node-resolver/0.log"
Nov 03 20:03:37.842361 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:37.842333    2577 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 20:03:38.433210 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:38.433183    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-zzt8t_1d4ce60c-e058-4c29-99e8-5f4f6eead744/node-ca/0.log"
Nov 03 20:03:38.594214 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:38.594181    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5p48g_15b5070f-d0d3-4ad8-aa1d-93cc7b96003a/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:38.615225 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:38.615201    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j4jzv"
Nov 03 20:03:38.615317 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:38.615201    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6dfr5"
Nov 03 20:03:38.615351 ip-10-0-132-4 kubenswrapper[2577]: E1103 20:03:38.615330    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j4jzv" podUID="51dffd76-5c30-440c-85ed-d81221663833"
Nov 03 20:03:38.615391 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:38.615206    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c947w"
Nov 03 20:03:38.615421 ip-10-0-132-4 kubenswrapper[2577]: E1103 20:03:38.615381    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6dfr5" podUID="d71eaa2a-9659-4827-9dfc-c1fce02565cb"
Nov 03 20:03:38.615481 ip-10-0-132-4 kubenswrapper[2577]: E1103 20:03:38.615460    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c947w" podUID="4c92a5d0-b34f-46a1-8f8f-a9ae3cb8d5a5"
Nov 03 20:03:38.712799 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:38.712759    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5p48g_15b5070f-d0d3-4ad8-aa1d-93cc7b96003a/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:38.844649 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:38.844617    2577 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 20:03:39.428531 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:39.428494    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5p48g_15b5070f-d0d3-4ad8-aa1d-93cc7b96003a/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:40.615261 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:40.615224    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c947w"
Nov 03 20:03:40.615658 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:40.615224    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6dfr5"
Nov 03 20:03:40.615658 ip-10-0-132-4 kubenswrapper[2577]: E1103 20:03:40.615347    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c947w" podUID="4c92a5d0-b34f-46a1-8f8f-a9ae3cb8d5a5"
Nov 03 20:03:40.615658 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:40.615239    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j4jzv"
Nov 03 20:03:40.615658 ip-10-0-132-4 kubenswrapper[2577]: E1103 20:03:40.615403    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6dfr5" podUID="d71eaa2a-9659-4827-9dfc-c1fce02565cb"
Nov 03 20:03:40.615658 ip-10-0-132-4 kubenswrapper[2577]: E1103 20:03:40.615488    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j4jzv" podUID="51dffd76-5c30-440c-85ed-d81221663833"
Nov 03 20:03:42.615855 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:42.615817    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j4jzv"
Nov 03 20:03:42.615855 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:42.615842    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c947w"
Nov 03 20:03:42.615855 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:42.615847    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6dfr5"
Nov 03 20:03:42.616333 ip-10-0-132-4 kubenswrapper[2577]: E1103 20:03:42.615947    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j4jzv" podUID="51dffd76-5c30-440c-85ed-d81221663833"
Nov 03 20:03:42.616333 ip-10-0-132-4 kubenswrapper[2577]: E1103 20:03:42.616053    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c947w" podUID="4c92a5d0-b34f-46a1-8f8f-a9ae3cb8d5a5"
Nov 03 20:03:42.616333 ip-10-0-132-4 kubenswrapper[2577]: E1103 20:03:42.616108    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6dfr5" podUID="d71eaa2a-9659-4827-9dfc-c1fce02565cb"
Nov 03 20:03:44.616209 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:44.616008    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c947w"
Nov 03 20:03:44.616590 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:44.616067    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j4jzv"
Nov 03 20:03:44.616590 ip-10-0-132-4 kubenswrapper[2577]: E1103 20:03:44.616283    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c947w" podUID="4c92a5d0-b34f-46a1-8f8f-a9ae3cb8d5a5"
Nov 03 20:03:44.616590 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:44.616090    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6dfr5"
Nov 03 20:03:44.616590 ip-10-0-132-4 kubenswrapper[2577]: E1103 20:03:44.616361    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j4jzv" podUID="51dffd76-5c30-440c-85ed-d81221663833"
Nov 03 20:03:44.616590 ip-10-0-132-4 kubenswrapper[2577]: E1103 20:03:44.616431    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6dfr5" podUID="d71eaa2a-9659-4827-9dfc-c1fce02565cb"
Nov 03 20:03:46.230266 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:46.230229    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6sv7j"
Nov 03 20:03:46.230726 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:46.230441    2577 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 20:03:46.249727 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:46.249688    2577 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-6sv7j" podUID="9126a982-59c9-4cec-a665-5f1bc0ae3389" containerName="ovnkube-controller" probeResult="failure" output=""
Nov 03 20:03:46.258420 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:46.258387    2577 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-6sv7j" podUID="9126a982-59c9-4cec-a665-5f1bc0ae3389" containerName="ovnkube-controller" probeResult="failure" output=""
Nov 03 20:03:46.615691 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:46.615607    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c947w"
Nov 03 20:03:46.615691 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:46.615643    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j4jzv"
Nov 03 20:03:46.615900 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:46.615614    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6dfr5"
Nov 03 20:03:46.615900 ip-10-0-132-4 kubenswrapper[2577]: E1103 20:03:46.615718    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c947w" podUID="4c92a5d0-b34f-46a1-8f8f-a9ae3cb8d5a5"
Nov 03 20:03:46.615900 ip-10-0-132-4 kubenswrapper[2577]: E1103 20:03:46.615817    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6dfr5" podUID="d71eaa2a-9659-4827-9dfc-c1fce02565cb"
Nov 03 20:03:46.615900 ip-10-0-132-4 kubenswrapper[2577]: E1103 20:03:46.615894    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j4jzv" podUID="51dffd76-5c30-440c-85ed-d81221663833"
Nov 03 20:03:48.341816 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:48.341766    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c92a5d0-b34f-46a1-8f8f-a9ae3cb8d5a5-metrics-certs\") pod \"network-metrics-daemon-c947w\" (UID: \"4c92a5d0-b34f-46a1-8f8f-a9ae3cb8d5a5\") " pod="openshift-multus/network-metrics-daemon-c947w"
Nov 03 20:03:48.342343 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:48.341830    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/51dffd76-5c30-440c-85ed-d81221663833-original-pull-secret\") pod \"global-pull-secret-syncer-j4jzv\" (UID: \"51dffd76-5c30-440c-85ed-d81221663833\") " pod="kube-system/global-pull-secret-syncer-j4jzv"
Nov 03 20:03:48.342343 ip-10-0-132-4 kubenswrapper[2577]: E1103 20:03:48.341926    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 20:03:48.342343 ip-10-0-132-4 kubenswrapper[2577]: E1103 20:03:48.341939    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 20:03:48.342343 ip-10-0-132-4 kubenswrapper[2577]: E1103 20:03:48.341979    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51dffd76-5c30-440c-85ed-d81221663833-original-pull-secret podName:51dffd76-5c30-440c-85ed-d81221663833 nodeName:}" failed. No retries permitted until 2026-11-03 20:04:20.341965741 +0000 UTC m=+66.202078268 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/51dffd76-5c30-440c-85ed-d81221663833-original-pull-secret") pod "global-pull-secret-syncer-j4jzv" (UID: "51dffd76-5c30-440c-85ed-d81221663833") : object "kube-system"/"original-pull-secret" not registered
Nov 03 20:03:48.342343 ip-10-0-132-4 kubenswrapper[2577]: E1103 20:03:48.342010    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c92a5d0-b34f-46a1-8f8f-a9ae3cb8d5a5-metrics-certs podName:4c92a5d0-b34f-46a1-8f8f-a9ae3cb8d5a5 nodeName:}" failed. No retries permitted until 2026-11-03 20:04:20.341990495 +0000 UTC m=+66.202103022 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4c92a5d0-b34f-46a1-8f8f-a9ae3cb8d5a5-metrics-certs") pod "network-metrics-daemon-c947w" (UID: "4c92a5d0-b34f-46a1-8f8f-a9ae3cb8d5a5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 20:03:48.442237 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:48.442195    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcxdr\" (UniqueName: \"kubernetes.io/projected/d71eaa2a-9659-4827-9dfc-c1fce02565cb-kube-api-access-zcxdr\") pod \"network-check-target-6dfr5\" (UID: \"d71eaa2a-9659-4827-9dfc-c1fce02565cb\") " pod="openshift-network-diagnostics/network-check-target-6dfr5"
Nov 03 20:03:48.442410 ip-10-0-132-4 kubenswrapper[2577]: E1103 20:03:48.442367    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 20:03:48.442410 ip-10-0-132-4 kubenswrapper[2577]: E1103 20:03:48.442384    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 20:03:48.442410 ip-10-0-132-4 kubenswrapper[2577]: E1103 20:03:48.442393    2577 projected.go:196] Error preparing data for projected volume kube-api-access-zcxdr for pod openshift-network-diagnostics/network-check-target-6dfr5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 20:03:48.442502 ip-10-0-132-4 kubenswrapper[2577]: E1103 20:03:48.442461    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d71eaa2a-9659-4827-9dfc-c1fce02565cb-kube-api-access-zcxdr podName:d71eaa2a-9659-4827-9dfc-c1fce02565cb nodeName:}" failed. No retries permitted until 2026-11-03 20:04:20.442444885 +0000 UTC m=+66.302557411 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-zcxdr" (UniqueName: "kubernetes.io/projected/d71eaa2a-9659-4827-9dfc-c1fce02565cb-kube-api-access-zcxdr") pod "network-check-target-6dfr5" (UID: "d71eaa2a-9659-4827-9dfc-c1fce02565cb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 20:03:48.615241 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:48.615151    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6dfr5"
Nov 03 20:03:48.615423 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:48.615151    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c947w"
Nov 03 20:03:48.615423 ip-10-0-132-4 kubenswrapper[2577]: E1103 20:03:48.615266    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6dfr5" podUID="d71eaa2a-9659-4827-9dfc-c1fce02565cb"
Nov 03 20:03:48.615423 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:48.615285    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j4jzv"
Nov 03 20:03:48.615423 ip-10-0-132-4 kubenswrapper[2577]: E1103 20:03:48.615386    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c947w" podUID="4c92a5d0-b34f-46a1-8f8f-a9ae3cb8d5a5"
Nov 03 20:03:48.615567 ip-10-0-132-4 kubenswrapper[2577]: E1103 20:03:48.615478    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j4jzv" podUID="51dffd76-5c30-440c-85ed-d81221663833"
Nov 03 20:03:50.615012 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:50.614984    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j4jzv"
Nov 03 20:03:50.615012 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:50.614993    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c947w"
Nov 03 20:03:50.615522 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:50.614993    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6dfr5"
Nov 03 20:03:50.615522 ip-10-0-132-4 kubenswrapper[2577]: E1103 20:03:50.615077    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j4jzv" podUID="51dffd76-5c30-440c-85ed-d81221663833"
Nov 03 20:03:50.615522 ip-10-0-132-4 kubenswrapper[2577]: E1103 20:03:50.615158    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c947w" podUID="4c92a5d0-b34f-46a1-8f8f-a9ae3cb8d5a5"
Nov 03 20:03:50.615522 ip-10-0-132-4 kubenswrapper[2577]: E1103 20:03:50.615212    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6dfr5" podUID="d71eaa2a-9659-4827-9dfc-c1fce02565cb"
Nov 03 20:03:52.615402 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:52.615364    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j4jzv"
Nov 03 20:03:52.615402 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:52.615392    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c947w"
Nov 03 20:03:52.616021 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:52.615364    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6dfr5"
Nov 03 20:03:52.616021 ip-10-0-132-4 kubenswrapper[2577]: E1103 20:03:52.615469    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j4jzv" podUID="51dffd76-5c30-440c-85ed-d81221663833"
Nov 03 20:03:52.616021 ip-10-0-132-4 kubenswrapper[2577]: E1103 20:03:52.615555    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6dfr5" podUID="d71eaa2a-9659-4827-9dfc-c1fce02565cb"
Nov 03 20:03:52.616021 ip-10-0-132-4 kubenswrapper[2577]: E1103 20:03:52.615629    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c947w" podUID="4c92a5d0-b34f-46a1-8f8f-a9ae3cb8d5a5"
Nov 03 20:03:53.727948 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:53.726073    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-j4jzv"]
Nov 03 20:03:53.727948 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:53.726199    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j4jzv"
Nov 03 20:03:53.727948 ip-10-0-132-4 kubenswrapper[2577]: E1103 20:03:53.726313    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j4jzv" podUID="51dffd76-5c30-440c-85ed-d81221663833"
Nov 03 20:03:53.729938 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:53.729909    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-6dfr5"]
Nov 03 20:03:53.730062 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:53.729965    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-c947w"]
Nov 03 20:03:53.730062 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:53.730036    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c947w"
Nov 03 20:03:53.730167 ip-10-0-132-4 kubenswrapper[2577]: E1103 20:03:53.730119    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c947w" podUID="4c92a5d0-b34f-46a1-8f8f-a9ae3cb8d5a5"
Nov 03 20:03:53.730280 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:53.730264    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6dfr5"
Nov 03 20:03:53.730366 ip-10-0-132-4 kubenswrapper[2577]: E1103 20:03:53.730349    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6dfr5" podUID="d71eaa2a-9659-4827-9dfc-c1fce02565cb"
Nov 03 20:03:55.615667 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:55.615629    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c947w"
Nov 03 20:03:55.615667 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:55.615661    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6dfr5"
Nov 03 20:03:55.616208 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:55.615675    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j4jzv"
Nov 03 20:03:55.616208 ip-10-0-132-4 kubenswrapper[2577]: E1103 20:03:55.615744    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c947w" podUID="4c92a5d0-b34f-46a1-8f8f-a9ae3cb8d5a5"
Nov 03 20:03:55.616208 ip-10-0-132-4 kubenswrapper[2577]: E1103 20:03:55.615846    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6dfr5" podUID="d71eaa2a-9659-4827-9dfc-c1fce02565cb"
Nov 03 20:03:55.616208 ip-10-0-132-4 kubenswrapper[2577]: E1103 20:03:55.615938    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j4jzv" podUID="51dffd76-5c30-440c-85ed-d81221663833"
Nov 03 20:03:57.615746 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:57.615549    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c947w"
Nov 03 20:03:57.616182 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:57.615548    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6dfr5"
Nov 03 20:03:57.616182 ip-10-0-132-4 kubenswrapper[2577]: E1103 20:03:57.615854    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c947w" podUID="4c92a5d0-b34f-46a1-8f8f-a9ae3cb8d5a5"
Nov 03 20:03:57.616182 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:57.615575    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j4jzv"
Nov 03 20:03:57.616182 ip-10-0-132-4 kubenswrapper[2577]: E1103 20:03:57.615928    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6dfr5" podUID="d71eaa2a-9659-4827-9dfc-c1fce02565cb"
Nov 03 20:03:57.616182 ip-10-0-132-4 kubenswrapper[2577]: E1103 20:03:57.615975    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j4jzv" podUID="51dffd76-5c30-440c-85ed-d81221663833"
Nov 03 20:03:57.942724 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:57.942703    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-4.ec2.internal" event="NodeReady"
Nov 03 20:03:57.942905 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:57.942857    2577 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Nov 03 20:03:57.991740 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:57.991710    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-7zl2f"]
Nov 03 20:03:57.993606 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:57.993588    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7zl2f"
Nov 03 20:03:57.997681 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:57.997658    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-5jnts"]
Nov 03 20:03:57.999275 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:57.999257    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-5jnts"
Nov 03 20:03:58.006339 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:58.006315    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 03 20:03:58.006491 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:58.006471    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-v94zj\""
Nov 03 20:03:58.006491 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:58.006483    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 03 20:03:58.012527 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:58.012506    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-8bwhz"]
Nov 03 20:03:58.015488 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:58.015470    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-7zl2f"]
Nov 03 20:03:58.015576 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:58.015559    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-8bwhz"
Nov 03 20:03:58.018817 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:58.018794    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-5jnts"]
Nov 03 20:03:58.033640 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:58.033618    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-8bwhz"]
Nov 03 20:03:58.106918 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:58.106883    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 03 20:03:58.106918 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:58.106886    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 03 20:03:58.106918 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:58.106919    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 03 20:03:58.107219 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:58.106980    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-5kpvl\""
Nov 03 20:03:58.107219 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:58.107073    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 03 20:03:58.107219 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:58.107099    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 03 20:03:58.107219 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:58.107143    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 03 20:03:58.107219 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:58.107157    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-86559\""
Nov 03 20:03:58.107219 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:58.107198    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 03 20:03:58.109415 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:58.109397    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szdn8\" (UniqueName: \"kubernetes.io/projected/884cbb80-b3f8-4f90-bd55-328d17943cf3-kube-api-access-szdn8\") pod \"insights-runtime-extractor-8bwhz\" (UID: \"884cbb80-b3f8-4f90-bd55-328d17943cf3\") " pod="openshift-insights/insights-runtime-extractor-8bwhz"
Nov 03 20:03:58.109502 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:58.109430    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-df59d\" (UniqueName: \"kubernetes.io/projected/3f681106-32d4-4550-a00d-1f9c52060b57-kube-api-access-df59d\") pod \"ingress-canary-5jnts\" (UID: \"3f681106-32d4-4550-a00d-1f9c52060b57\") " pod="openshift-ingress-canary/ingress-canary-5jnts"
Nov 03 20:03:58.109502 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:58.109457    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56z7v\" (UniqueName: \"kubernetes.io/projected/2fd3c650-2f4c-404e-ab06-c50612af72ce-kube-api-access-56z7v\") pod \"dns-default-7zl2f\" (UID: \"2fd3c650-2f4c-404e-ab06-c50612af72ce\") " pod="openshift-dns/dns-default-7zl2f"
Nov 03 20:03:58.109569 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:58.109509    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/884cbb80-b3f8-4f90-bd55-328d17943cf3-data-volume\") pod \"insights-runtime-extractor-8bwhz\" (UID: \"884cbb80-b3f8-4f90-bd55-328d17943cf3\") " pod="openshift-insights/insights-runtime-extractor-8bwhz"
Nov 03 20:03:58.109615 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:58.109600    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fd3c650-2f4c-404e-ab06-c50612af72ce-tmp-dir\") pod \"dns-default-7zl2f\" (UID: \"2fd3c650-2f4c-404e-ab06-c50612af72ce\") " pod="openshift-dns/dns-default-7zl2f"
Nov 03 20:03:58.109648 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:58.109624    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2fd3c650-2f4c-404e-ab06-c50612af72ce-config-volume\") pod \"dns-default-7zl2f\" (UID: \"2fd3c650-2f4c-404e-ab06-c50612af72ce\") " pod="openshift-dns/dns-default-7zl2f"
Nov 03 20:03:58.109690 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:58.109649    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2fd3c650-2f4c-404e-ab06-c50612af72ce-metrics-tls\") pod \"dns-default-7zl2f\" (UID: \"2fd3c650-2f4c-404e-ab06-c50612af72ce\") " pod="openshift-dns/dns-default-7zl2f"
Nov 03 20:03:58.109690 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:58.109669    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/884cbb80-b3f8-4f90-bd55-328d17943cf3-crio-socket\") pod \"insights-runtime-extractor-8bwhz\" (UID: \"884cbb80-b3f8-4f90-bd55-328d17943cf3\") " pod="openshift-insights/insights-runtime-extractor-8bwhz"
Nov 03 20:03:58.109756 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:58.109710    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/884cbb80-b3f8-4f90-bd55-328d17943cf3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8bwhz\" (UID: \"884cbb80-b3f8-4f90-bd55-328d17943cf3\") " pod="openshift-insights/insights-runtime-extractor-8bwhz"
Nov 03 20:03:58.109818 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:58.109806    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/884cbb80-b3f8-4f90-bd55-328d17943cf3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8bwhz\" (UID: \"884cbb80-b3f8-4f90-bd55-328d17943cf3\") " pod="openshift-insights/insights-runtime-extractor-8bwhz"
Nov 03 20:03:58.109854 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:58.109826    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3f681106-32d4-4550-a00d-1f9c52060b57-cert\") pod \"ingress-canary-5jnts\" (UID: \"3f681106-32d4-4550-a00d-1f9c52060b57\") " pod="openshift-ingress-canary/ingress-canary-5jnts"
Nov 03 20:03:58.210878 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:58.210792    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2fd3c650-2f4c-404e-ab06-c50612af72ce-metrics-tls\") pod \"dns-default-7zl2f\" (UID: \"2fd3c650-2f4c-404e-ab06-c50612af72ce\") " pod="openshift-dns/dns-default-7zl2f"
Nov 03 20:03:58.210878 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:58.210840    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/884cbb80-b3f8-4f90-bd55-328d17943cf3-crio-socket\") pod \"insights-runtime-extractor-8bwhz\" (UID: \"884cbb80-b3f8-4f90-bd55-328d17943cf3\") " pod="openshift-insights/insights-runtime-extractor-8bwhz"
Nov 03 20:03:58.210878 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:58.210868    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/884cbb80-b3f8-4f90-bd55-328d17943cf3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8bwhz\" (UID: \"884cbb80-b3f8-4f90-bd55-328d17943cf3\") " pod="openshift-insights/insights-runtime-extractor-8bwhz"
Nov 03 20:03:58.211164 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:58.210906    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/884cbb80-b3f8-4f90-bd55-328d17943cf3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8bwhz\" (UID: \"884cbb80-b3f8-4f90-bd55-328d17943cf3\") " pod="openshift-insights/insights-runtime-extractor-8bwhz"
Nov 03 20:03:58.211164 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:58.210927    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3f681106-32d4-4550-a00d-1f9c52060b57-cert\") pod \"ingress-canary-5jnts\" (UID: \"3f681106-32d4-4550-a00d-1f9c52060b57\") " pod="openshift-ingress-canary/ingress-canary-5jnts"
Nov 03 20:03:58.211164 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:58.210952    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szdn8\" (UniqueName: \"kubernetes.io/projected/884cbb80-b3f8-4f90-bd55-328d17943cf3-kube-api-access-szdn8\") pod \"insights-runtime-extractor-8bwhz\" (UID: \"884cbb80-b3f8-4f90-bd55-328d17943cf3\") " pod="openshift-insights/insights-runtime-extractor-8bwhz"
Nov 03 20:03:58.211164 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:58.210980    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-df59d\" (UniqueName: \"kubernetes.io/projected/3f681106-32d4-4550-a00d-1f9c52060b57-kube-api-access-df59d\") pod \"ingress-canary-5jnts\" (UID: \"3f681106-32d4-4550-a00d-1f9c52060b57\") " pod="openshift-ingress-canary/ingress-canary-5jnts"
Nov 03 20:03:58.211164 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:58.211004    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56z7v\" (UniqueName: \"kubernetes.io/projected/2fd3c650-2f4c-404e-ab06-c50612af72ce-kube-api-access-56z7v\") pod \"dns-default-7zl2f\" (UID: \"2fd3c650-2f4c-404e-ab06-c50612af72ce\") " pod="openshift-dns/dns-default-7zl2f"
Nov 03 20:03:58.211164 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:58.211027    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/884cbb80-b3f8-4f90-bd55-328d17943cf3-data-volume\") pod \"insights-runtime-extractor-8bwhz\" (UID: \"884cbb80-b3f8-4f90-bd55-328d17943cf3\") " pod="openshift-insights/insights-runtime-extractor-8bwhz"
Nov 03 20:03:58.211164 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:58.211087    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fd3c650-2f4c-404e-ab06-c50612af72ce-tmp-dir\") pod \"dns-default-7zl2f\" (UID: \"2fd3c650-2f4c-404e-ab06-c50612af72ce\") " pod="openshift-dns/dns-default-7zl2f"
Nov 03 20:03:58.211164 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:58.211162    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2fd3c650-2f4c-404e-ab06-c50612af72ce-config-volume\") pod \"dns-default-7zl2f\" (UID: \"2fd3c650-2f4c-404e-ab06-c50612af72ce\") " pod="openshift-dns/dns-default-7zl2f"
Nov 03 20:03:58.211582 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:58.211363    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/884cbb80-b3f8-4f90-bd55-328d17943cf3-crio-socket\") pod \"insights-runtime-extractor-8bwhz\" (UID: \"884cbb80-b3f8-4f90-bd55-328d17943cf3\") " pod="openshift-insights/insights-runtime-extractor-8bwhz"
Nov 03 20:03:58.211639 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:58.211629    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/884cbb80-b3f8-4f90-bd55-328d17943cf3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8bwhz\" (UID: \"884cbb80-b3f8-4f90-bd55-328d17943cf3\") " pod="openshift-insights/insights-runtime-extractor-8bwhz"
Nov 03 20:03:58.211697 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:58.211640    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/884cbb80-b3f8-4f90-bd55-328d17943cf3-data-volume\") pod \"insights-runtime-extractor-8bwhz\" (UID: \"884cbb80-b3f8-4f90-bd55-328d17943cf3\") " pod="openshift-insights/insights-runtime-extractor-8bwhz"
Nov 03 20:03:58.211751 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:58.211729    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2fd3c650-2f4c-404e-ab06-c50612af72ce-config-volume\") pod \"dns-default-7zl2f\" (UID: \"2fd3c650-2f4c-404e-ab06-c50612af72ce\") " pod="openshift-dns/dns-default-7zl2f"
Nov 03 20:03:58.211868 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:58.211849    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fd3c650-2f4c-404e-ab06-c50612af72ce-tmp-dir\") pod \"dns-default-7zl2f\" (UID: \"2fd3c650-2f4c-404e-ab06-c50612af72ce\") " pod="openshift-dns/dns-default-7zl2f"
Nov 03 20:03:58.215016 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:58.214991    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/884cbb80-b3f8-4f90-bd55-328d17943cf3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8bwhz\" (UID: \"884cbb80-b3f8-4f90-bd55-328d17943cf3\") " pod="openshift-insights/insights-runtime-extractor-8bwhz"
Nov 03 20:03:58.215113 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:58.215023    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2fd3c650-2f4c-404e-ab06-c50612af72ce-metrics-tls\") pod \"dns-default-7zl2f\" (UID: \"2fd3c650-2f4c-404e-ab06-c50612af72ce\") " pod="openshift-dns/dns-default-7zl2f"
Nov 03 20:03:58.215113 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:58.215099    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3f681106-32d4-4550-a00d-1f9c52060b57-cert\") pod \"ingress-canary-5jnts\" (UID: \"3f681106-32d4-4550-a00d-1f9c52060b57\") " pod="openshift-ingress-canary/ingress-canary-5jnts"
Nov 03 20:03:58.227911 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:58.227884    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-szdn8\" (UniqueName: \"kubernetes.io/projected/884cbb80-b3f8-4f90-bd55-328d17943cf3-kube-api-access-szdn8\") pod \"insights-runtime-extractor-8bwhz\" (UID: \"884cbb80-b3f8-4f90-bd55-328d17943cf3\") " pod="openshift-insights/insights-runtime-extractor-8bwhz"
Nov 03 20:03:58.228605 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:58.228587    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56z7v\" (UniqueName: \"kubernetes.io/projected/2fd3c650-2f4c-404e-ab06-c50612af72ce-kube-api-access-56z7v\") pod \"dns-default-7zl2f\" (UID: \"2fd3c650-2f4c-404e-ab06-c50612af72ce\") " pod="openshift-dns/dns-default-7zl2f"
Nov 03 20:03:58.228683 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:58.228666    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-df59d\" (UniqueName: \"kubernetes.io/projected/3f681106-32d4-4550-a00d-1f9c52060b57-kube-api-access-df59d\") pod \"ingress-canary-5jnts\" (UID: \"3f681106-32d4-4550-a00d-1f9c52060b57\") " pod="openshift-ingress-canary/ingress-canary-5jnts"
Nov 03 20:03:58.306655 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:58.306611    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7zl2f"
Nov 03 20:03:58.311237 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:58.311213    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-5jnts"
Nov 03 20:03:58.328474 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:58.328445    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-8bwhz"
Nov 03 20:03:58.621806 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:58.621718    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-5jnts"]
Nov 03 20:03:58.627002 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:58.626977    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f681106_32d4_4550_a00d_1f9c52060b57.slice/crio-ee3ebf1d7f01d0a2963b1d18ebd247697e393917f93326d8b167e966ef5c8601 WatchSource:0}: Error finding container ee3ebf1d7f01d0a2963b1d18ebd247697e393917f93326d8b167e966ef5c8601: Status 404 returned error can't find the container with id ee3ebf1d7f01d0a2963b1d18ebd247697e393917f93326d8b167e966ef5c8601
Nov 03 20:03:58.649281 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:58.649256    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-8bwhz"]
Nov 03 20:03:58.652976 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:58.652952    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-7zl2f"]
Nov 03 20:03:58.653558 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:58.653533    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod884cbb80_b3f8_4f90_bd55_328d17943cf3.slice/crio-47a7d0ed1c1bb8bcaef8b72693485b6259b9f0511abffeae75ba9fb88dae4ece WatchSource:0}: Error finding container 47a7d0ed1c1bb8bcaef8b72693485b6259b9f0511abffeae75ba9fb88dae4ece: Status 404 returned error can't find the container with id 47a7d0ed1c1bb8bcaef8b72693485b6259b9f0511abffeae75ba9fb88dae4ece
Nov 03 20:03:58.659996 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:03:58.657321    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fd3c650_2f4c_404e_ab06_c50612af72ce.slice/crio-a7dae436e7bb325f83307dc7270e2a3b50e7675bfe3214d187fdaa4b63e06161 WatchSource:0}: Error finding container a7dae436e7bb325f83307dc7270e2a3b50e7675bfe3214d187fdaa4b63e06161: Status 404 returned error can't find the container with id a7dae436e7bb325f83307dc7270e2a3b50e7675bfe3214d187fdaa4b63e06161
Nov 03 20:03:58.889820 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:58.889674    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8bwhz" event={"ID":"884cbb80-b3f8-4f90-bd55-328d17943cf3","Type":"ContainerStarted","Data":"22ac6171469c0eccb20e673500d3991e7d3ef4385b71147e1f624b1bf07f4e13"}
Nov 03 20:03:58.889820 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:58.889763    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8bwhz" event={"ID":"884cbb80-b3f8-4f90-bd55-328d17943cf3","Type":"ContainerStarted","Data":"47a7d0ed1c1bb8bcaef8b72693485b6259b9f0511abffeae75ba9fb88dae4ece"}
Nov 03 20:03:58.890872 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:58.890846    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-5jnts" event={"ID":"3f681106-32d4-4550-a00d-1f9c52060b57","Type":"ContainerStarted","Data":"ee3ebf1d7f01d0a2963b1d18ebd247697e393917f93326d8b167e966ef5c8601"}
Nov 03 20:03:58.891969 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:58.891947    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7zl2f" event={"ID":"2fd3c650-2f4c-404e-ab06-c50612af72ce","Type":"ContainerStarted","Data":"a7dae436e7bb325f83307dc7270e2a3b50e7675bfe3214d187fdaa4b63e06161"}
Nov 03 20:03:59.615980 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:59.615904    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c947w"
Nov 03 20:03:59.616342 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:59.616326    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j4jzv"
Nov 03 20:03:59.616701 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:59.616683    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6dfr5"
Nov 03 20:03:59.629282 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:59.629253    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 03 20:03:59.639381 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:59.639340    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 03 20:03:59.639492 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:59.639431    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-q2t72\""
Nov 03 20:03:59.639551 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:59.639504    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-n7xhm\""
Nov 03 20:03:59.639601 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:59.639566    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 03 20:03:59.639665 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:59.639651    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 03 20:03:59.639736 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:59.639723    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-89f7f\""
Nov 03 20:03:59.898551 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:03:59.898471    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8bwhz" event={"ID":"884cbb80-b3f8-4f90-bd55-328d17943cf3","Type":"ContainerStarted","Data":"6e12f5bc67943bff22bd2b18545efba3f379fb5f20592c75113630e3f6e2f419"}
Nov 03 20:04:00.275479 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:00.275435    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-rwhnf"]
Nov 03 20:04:00.278482 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:00.277716    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-rwhnf"
Nov 03 20:04:00.292468 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:00.292145    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Nov 03 20:04:00.292468 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:00.292253    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Nov 03 20:04:00.292468 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:00.292375    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-kk6pp\""
Nov 03 20:04:00.302223 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:00.302159    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 03 20:04:00.302223 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:00.302159    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 03 20:04:00.302223 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:00.302163    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Nov 03 20:04:00.303197 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:00.303168    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-rwhnf"]
Nov 03 20:04:00.313790 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:00.313744    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-j46kr"]
Nov 03 20:04:00.317506 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:00.317477    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-j46kr"
Nov 03 20:04:00.337617 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:00.336438    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-q8b6s"]
Nov 03 20:04:00.339169 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:00.338794    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-q8b6s"
Nov 03 20:04:00.355996 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:00.355956    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-q8b6s"]
Nov 03 20:04:00.392529 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:00.392493    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Nov 03 20:04:00.392529 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:00.392521    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Nov 03 20:04:00.392529 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:00.392535    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Nov 03 20:04:00.392853 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:00.392507    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Nov 03 20:04:00.392853 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:00.392663    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-89ghp\""
Nov 03 20:04:00.392853 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:00.392679    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-tmbxc\""
Nov 03 20:04:00.392853 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:00.392719    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Nov 03 20:04:00.392990 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:00.392918    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Nov 03 20:04:00.432719 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:00.432686    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ffe074b1-e2a0-403b-9f81-581fdb4d2775-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-q8b6s\" (UID: \"ffe074b1-e2a0-403b-9f81-581fdb4d2775\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-q8b6s"
Nov 03 20:04:00.432719 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:00.432723    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ffe074b1-e2a0-403b-9f81-581fdb4d2775-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-q8b6s\" (UID: \"ffe074b1-e2a0-403b-9f81-581fdb4d2775\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-q8b6s"
Nov 03 20:04:00.432932 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:00.432739    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/ffe074b1-e2a0-403b-9f81-581fdb4d2775-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-q8b6s\" (UID: \"ffe074b1-e2a0-403b-9f81-581fdb4d2775\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-q8b6s"
Nov 03 20:04:00.432932 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:00.432756    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/74d4808e-24ce-4c47-b4ce-e1188eeea6f1-sys\") pod \"node-exporter-j46kr\" (UID: \"74d4808e-24ce-4c47-b4ce-e1188eeea6f1\") " pod="openshift-monitoring/node-exporter-j46kr"
Nov 03 20:04:00.432932 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:00.432786    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/74d4808e-24ce-4c47-b4ce-e1188eeea6f1-node-exporter-wtmp\") pod \"node-exporter-j46kr\" (UID: \"74d4808e-24ce-4c47-b4ce-e1188eeea6f1\") " pod="openshift-monitoring/node-exporter-j46kr"
Nov 03 20:04:00.432932 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:00.432806    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/74d4808e-24ce-4c47-b4ce-e1188eeea6f1-node-exporter-accelerators-collector-config\") pod \"node-exporter-j46kr\" (UID: \"74d4808e-24ce-4c47-b4ce-e1188eeea6f1\") " pod="openshift-monitoring/node-exporter-j46kr"
Nov 03 20:04:00.432932 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:00.432822    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/74d4808e-24ce-4c47-b4ce-e1188eeea6f1-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-j46kr\" (UID: \"74d4808e-24ce-4c47-b4ce-e1188eeea6f1\") " pod="openshift-monitoring/node-exporter-j46kr"
Nov 03 20:04:00.432932 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:00.432842    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f547t\" (UniqueName: \"kubernetes.io/projected/74d4808e-24ce-4c47-b4ce-e1188eeea6f1-kube-api-access-f547t\") pod \"node-exporter-j46kr\" (UID: \"74d4808e-24ce-4c47-b4ce-e1188eeea6f1\") " pod="openshift-monitoring/node-exporter-j46kr"
Nov 03 20:04:00.432932 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:00.432875    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cb9bef9b-c7cd-410e-8c1e-1b27474daf81-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-rwhnf\" (UID: \"cb9bef9b-c7cd-410e-8c1e-1b27474daf81\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-rwhnf"
Nov 03 20:04:00.432932 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:00.432895    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4g29t\" (UniqueName: \"kubernetes.io/projected/ffe074b1-e2a0-403b-9f81-581fdb4d2775-kube-api-access-4g29t\") pod \"kube-state-metrics-5dc8d7d89b-q8b6s\" (UID: \"ffe074b1-e2a0-403b-9f81-581fdb4d2775\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-q8b6s"
Nov 03 20:04:00.432932 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:00.432909    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/74d4808e-24ce-4c47-b4ce-e1188eeea6f1-node-exporter-textfile\") pod \"node-exporter-j46kr\" (UID: \"74d4808e-24ce-4c47-b4ce-e1188eeea6f1\") " pod="openshift-monitoring/node-exporter-j46kr"
Nov 03 20:04:00.432932 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:00.432934    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cb9bef9b-c7cd-410e-8c1e-1b27474daf81-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-rwhnf\" (UID: \"cb9bef9b-c7cd-410e-8c1e-1b27474daf81\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-rwhnf"
Nov 03 20:04:00.433256 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:00.432952    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/74d4808e-24ce-4c47-b4ce-e1188eeea6f1-metrics-client-ca\") pod \"node-exporter-j46kr\" (UID: \"74d4808e-24ce-4c47-b4ce-e1188eeea6f1\") " pod="openshift-monitoring/node-exporter-j46kr"
Nov 03 20:04:00.433256 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:00.432970    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ffe074b1-e2a0-403b-9f81-581fdb4d2775-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-q8b6s\" (UID: \"ffe074b1-e2a0-403b-9f81-581fdb4d2775\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-q8b6s"
Nov 03 20:04:00.433256 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:00.433004    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/cb9bef9b-c7cd-410e-8c1e-1b27474daf81-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-rwhnf\" (UID: \"cb9bef9b-c7cd-410e-8c1e-1b27474daf81\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-rwhnf"
Nov 03 20:04:00.433256 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:00.433025    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ndt8\" (UniqueName: \"kubernetes.io/projected/cb9bef9b-c7cd-410e-8c1e-1b27474daf81-kube-api-access-4ndt8\") pod \"openshift-state-metrics-7bd7489957-rwhnf\" (UID: \"cb9bef9b-c7cd-410e-8c1e-1b27474daf81\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-rwhnf"
Nov 03 20:04:00.433256 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:00.433044    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/ffe074b1-e2a0-403b-9f81-581fdb4d2775-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-q8b6s\" (UID: \"ffe074b1-e2a0-403b-9f81-581fdb4d2775\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-q8b6s"
Nov 03 20:04:00.433256 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:00.433062    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/74d4808e-24ce-4c47-b4ce-e1188eeea6f1-root\") pod \"node-exporter-j46kr\" (UID: \"74d4808e-24ce-4c47-b4ce-e1188eeea6f1\") " pod="openshift-monitoring/node-exporter-j46kr"
Nov 03 20:04:00.433256 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:00.433077    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/74d4808e-24ce-4c47-b4ce-e1188eeea6f1-node-exporter-tls\") pod \"node-exporter-j46kr\" (UID: \"74d4808e-24ce-4c47-b4ce-e1188eeea6f1\") " pod="openshift-monitoring/node-exporter-j46kr"
Nov 03 20:04:00.534085 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:00.534052    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4g29t\" (UniqueName: \"kubernetes.io/projected/ffe074b1-e2a0-403b-9f81-581fdb4d2775-kube-api-access-4g29t\") pod \"kube-state-metrics-5dc8d7d89b-q8b6s\" (UID: \"ffe074b1-e2a0-403b-9f81-581fdb4d2775\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-q8b6s"
Nov 03 20:04:00.534212 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:00.534096    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/74d4808e-24ce-4c47-b4ce-e1188eeea6f1-node-exporter-textfile\") pod \"node-exporter-j46kr\" (UID: \"74d4808e-24ce-4c47-b4ce-e1188eeea6f1\") " pod="openshift-monitoring/node-exporter-j46kr"
Nov 03 20:04:00.534212 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:00.534132    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cb9bef9b-c7cd-410e-8c1e-1b27474daf81-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-rwhnf\" (UID: \"cb9bef9b-c7cd-410e-8c1e-1b27474daf81\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-rwhnf"
Nov 03 20:04:00.534212 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:00.534165    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/74d4808e-24ce-4c47-b4ce-e1188eeea6f1-metrics-client-ca\") pod \"node-exporter-j46kr\" (UID: \"74d4808e-24ce-4c47-b4ce-e1188eeea6f1\") " pod="openshift-monitoring/node-exporter-j46kr"
Nov 03 20:04:00.534212 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:00.534193    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ffe074b1-e2a0-403b-9f81-581fdb4d2775-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-q8b6s\" (UID: \"ffe074b1-e2a0-403b-9f81-581fdb4d2775\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-q8b6s"
Nov 03 20:04:00.534344 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:00.534233    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/cb9bef9b-c7cd-410e-8c1e-1b27474daf81-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-rwhnf\" (UID: \"cb9bef9b-c7cd-410e-8c1e-1b27474daf81\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-rwhnf"
Nov 03 20:04:00.534344 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:00.534263    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ndt8\" (UniqueName: \"kubernetes.io/projected/cb9bef9b-c7cd-410e-8c1e-1b27474daf81-kube-api-access-4ndt8\") pod \"openshift-state-metrics-7bd7489957-rwhnf\" (UID: \"cb9bef9b-c7cd-410e-8c1e-1b27474daf81\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-rwhnf"
Nov 03 20:04:00.534344 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:00.534290    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/ffe074b1-e2a0-403b-9f81-581fdb4d2775-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-q8b6s\" (UID: \"ffe074b1-e2a0-403b-9f81-581fdb4d2775\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-q8b6s"
Nov 03 20:04:00.534344 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:00.534318    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/74d4808e-24ce-4c47-b4ce-e1188eeea6f1-root\") pod \"node-exporter-j46kr\" (UID: \"74d4808e-24ce-4c47-b4ce-e1188eeea6f1\") " pod="openshift-monitoring/node-exporter-j46kr"
Nov 03 20:04:00.534471 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:00.534346    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/74d4808e-24ce-4c47-b4ce-e1188eeea6f1-node-exporter-tls\") pod \"node-exporter-j46kr\" (UID: \"74d4808e-24ce-4c47-b4ce-e1188eeea6f1\") " pod="openshift-monitoring/node-exporter-j46kr"
Nov 03 20:04:00.534471 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:00.534393    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ffe074b1-e2a0-403b-9f81-581fdb4d2775-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-q8b6s\" (UID: \"ffe074b1-e2a0-403b-9f81-581fdb4d2775\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-q8b6s"
Nov 03 20:04:00.534471 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:00.534423    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ffe074b1-e2a0-403b-9f81-581fdb4d2775-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-q8b6s\" (UID: \"ffe074b1-e2a0-403b-9f81-581fdb4d2775\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-q8b6s"
Nov 03 20:04:00.534471 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:00.534452    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/ffe074b1-e2a0-403b-9f81-581fdb4d2775-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-q8b6s\" (UID: \"ffe074b1-e2a0-403b-9f81-581fdb4d2775\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-q8b6s"
Nov 03 20:04:00.534584 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:00.534476    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/74d4808e-24ce-4c47-b4ce-e1188eeea6f1-sys\") pod \"node-exporter-j46kr\" (UID: \"74d4808e-24ce-4c47-b4ce-e1188eeea6f1\") " pod="openshift-monitoring/node-exporter-j46kr"
Nov 03 20:04:00.534584 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:00.534498    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/74d4808e-24ce-4c47-b4ce-e1188eeea6f1-node-exporter-wtmp\") pod \"node-exporter-j46kr\" (UID: \"74d4808e-24ce-4c47-b4ce-e1188eeea6f1\") " pod="openshift-monitoring/node-exporter-j46kr"
Nov 03 20:04:00.534584 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:00.534528    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/74d4808e-24ce-4c47-b4ce-e1188eeea6f1-node-exporter-accelerators-collector-config\") pod \"node-exporter-j46kr\" (UID: \"74d4808e-24ce-4c47-b4ce-e1188eeea6f1\") " pod="openshift-monitoring/node-exporter-j46kr"
Nov 03 20:04:00.534584 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:00.534555    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/74d4808e-24ce-4c47-b4ce-e1188eeea6f1-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-j46kr\" (UID: \"74d4808e-24ce-4c47-b4ce-e1188eeea6f1\") " pod="openshift-monitoring/node-exporter-j46kr"
Nov 03 20:04:00.534695 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:00.534589    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f547t\" (UniqueName: \"kubernetes.io/projected/74d4808e-24ce-4c47-b4ce-e1188eeea6f1-kube-api-access-f547t\") pod \"node-exporter-j46kr\" (UID: \"74d4808e-24ce-4c47-b4ce-e1188eeea6f1\") " pod="openshift-monitoring/node-exporter-j46kr"
Nov 03 20:04:00.534695 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:00.534633    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cb9bef9b-c7cd-410e-8c1e-1b27474daf81-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-rwhnf\" (UID: \"cb9bef9b-c7cd-410e-8c1e-1b27474daf81\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-rwhnf"
Nov 03 20:04:00.535629 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:00.535603    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cb9bef9b-c7cd-410e-8c1e-1b27474daf81-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-rwhnf\" (UID: \"cb9bef9b-c7cd-410e-8c1e-1b27474daf81\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-rwhnf"
Nov 03 20:04:00.535921 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:00.535897    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/74d4808e-24ce-4c47-b4ce-e1188eeea6f1-node-exporter-textfile\") pod \"node-exporter-j46kr\" (UID: \"74d4808e-24ce-4c47-b4ce-e1188eeea6f1\") " pod="openshift-monitoring/node-exporter-j46kr"
Nov 03 20:04:00.544110 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:00.537740    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/74d4808e-24ce-4c47-b4ce-e1188eeea6f1-node-exporter-accelerators-collector-config\") pod \"node-exporter-j46kr\" (UID: \"74d4808e-24ce-4c47-b4ce-e1188eeea6f1\") " pod="openshift-monitoring/node-exporter-j46kr"
Nov 03 20:04:00.546237 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:00.544647    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ffe074b1-e2a0-403b-9f81-581fdb4d2775-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-q8b6s\" (UID: \"ffe074b1-e2a0-403b-9f81-581fdb4d2775\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-q8b6s"
Nov 03 20:04:00.546237 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:00.545219    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cb9bef9b-c7cd-410e-8c1e-1b27474daf81-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-rwhnf\" (UID: \"cb9bef9b-c7cd-410e-8c1e-1b27474daf81\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-rwhnf"
Nov 03 20:04:00.546237 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:00.545879    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/74d4808e-24ce-4c47-b4ce-e1188eeea6f1-metrics-client-ca\") pod \"node-exporter-j46kr\" (UID: \"74d4808e-24ce-4c47-b4ce-e1188eeea6f1\") " pod="openshift-monitoring/node-exporter-j46kr"
Nov 03 20:04:00.546795 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:00.546752    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ffe074b1-e2a0-403b-9f81-581fdb4d2775-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-q8b6s\" (UID: \"ffe074b1-e2a0-403b-9f81-581fdb4d2775\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-q8b6s"
Nov 03 20:04:00.550028 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:00.547131    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/74d4808e-24ce-4c47-b4ce-e1188eeea6f1-node-exporter-wtmp\") pod \"node-exporter-j46kr\" (UID: \"74d4808e-24ce-4c47-b4ce-e1188eeea6f1\") " pod="openshift-monitoring/node-exporter-j46kr"
Nov 03 20:04:00.550028 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:00.547200    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/74d4808e-24ce-4c47-b4ce-e1188eeea6f1-sys\") pod \"node-exporter-j46kr\" (UID: \"74d4808e-24ce-4c47-b4ce-e1188eeea6f1\") " pod="openshift-monitoring/node-exporter-j46kr"
Nov 03 20:04:00.550028 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:00.547285    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/74d4808e-24ce-4c47-b4ce-e1188eeea6f1-root\") pod \"node-exporter-j46kr\" (UID: \"74d4808e-24ce-4c47-b4ce-e1188eeea6f1\") " pod="openshift-monitoring/node-exporter-j46kr"
Nov 03 20:04:00.550028 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:00.549340    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/ffe074b1-e2a0-403b-9f81-581fdb4d2775-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-q8b6s\" (UID: \"ffe074b1-e2a0-403b-9f81-581fdb4d2775\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-q8b6s"
Nov 03 20:04:00.550028 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:00.549583    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/74d4808e-24ce-4c47-b4ce-e1188eeea6f1-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-j46kr\" (UID: \"74d4808e-24ce-4c47-b4ce-e1188eeea6f1\") " pod="openshift-monitoring/node-exporter-j46kr"
Nov 03 20:04:00.553796 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:00.536616    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/ffe074b1-e2a0-403b-9f81-581fdb4d2775-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-q8b6s\" (UID: \"ffe074b1-e2a0-403b-9f81-581fdb4d2775\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-q8b6s"
Nov 03 20:04:00.553796 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:00.552829    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/cb9bef9b-c7cd-410e-8c1e-1b27474daf81-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-rwhnf\" (UID: \"cb9bef9b-c7cd-410e-8c1e-1b27474daf81\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-rwhnf"
Nov 03 20:04:00.559323 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:00.559294    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4g29t\" (UniqueName: \"kubernetes.io/projected/ffe074b1-e2a0-403b-9f81-581fdb4d2775-kube-api-access-4g29t\") pod \"kube-state-metrics-5dc8d7d89b-q8b6s\" (UID: \"ffe074b1-e2a0-403b-9f81-581fdb4d2775\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-q8b6s"
Nov 03 20:04:00.559585 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:00.559564    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/74d4808e-24ce-4c47-b4ce-e1188eeea6f1-node-exporter-tls\") pod \"node-exporter-j46kr\" (UID: \"74d4808e-24ce-4c47-b4ce-e1188eeea6f1\") " pod="openshift-monitoring/node-exporter-j46kr"
Nov 03 20:04:00.564098 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:00.564031    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ffe074b1-e2a0-403b-9f81-581fdb4d2775-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-q8b6s\" (UID: \"ffe074b1-e2a0-403b-9f81-581fdb4d2775\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-q8b6s"
Nov 03 20:04:00.565974 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:00.565949    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ndt8\" (UniqueName: \"kubernetes.io/projected/cb9bef9b-c7cd-410e-8c1e-1b27474daf81-kube-api-access-4ndt8\") pod \"openshift-state-metrics-7bd7489957-rwhnf\" (UID: \"cb9bef9b-c7cd-410e-8c1e-1b27474daf81\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-rwhnf"
Nov 03 20:04:00.581398 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:00.581376    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f547t\" (UniqueName: \"kubernetes.io/projected/74d4808e-24ce-4c47-b4ce-e1188eeea6f1-kube-api-access-f547t\") pod \"node-exporter-j46kr\" (UID: \"74d4808e-24ce-4c47-b4ce-e1188eeea6f1\") " pod="openshift-monitoring/node-exporter-j46kr"
Nov 03 20:04:00.592625 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:00.592598    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-rwhnf"
Nov 03 20:04:00.630468 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:00.630259    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-j46kr"
Nov 03 20:04:00.647948 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:04:00.647913    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74d4808e_24ce_4c47_b4ce_e1188eeea6f1.slice/crio-74d96638209d41f05199b24b4b3143d5e94c1b40a389dde0dcfbd1ceef3266b7 WatchSource:0}: Error finding container 74d96638209d41f05199b24b4b3143d5e94c1b40a389dde0dcfbd1ceef3266b7: Status 404 returned error can't find the container with id 74d96638209d41f05199b24b4b3143d5e94c1b40a389dde0dcfbd1ceef3266b7
Nov 03 20:04:00.657577 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:00.654673    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-q8b6s"
Nov 03 20:04:00.903165 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:00.903110    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7zl2f" event={"ID":"2fd3c650-2f4c-404e-ab06-c50612af72ce","Type":"ContainerStarted","Data":"e0602b68c64c5ee4f8d5c39466f91ee32da9c8faf3c16cfe873f51a6c8fb48e5"}
Nov 03 20:04:00.903165 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:00.903181    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7zl2f" event={"ID":"2fd3c650-2f4c-404e-ab06-c50612af72ce","Type":"ContainerStarted","Data":"93a47b4d2d4948525f8cde4d265d7caf4b1dd7af8127fe77fd41067672f955c4"}
Nov 03 20:04:00.903427 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:00.903228    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-7zl2f"
Nov 03 20:04:00.904745 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:00.904721    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-j46kr" event={"ID":"74d4808e-24ce-4c47-b4ce-e1188eeea6f1","Type":"ContainerStarted","Data":"74d96638209d41f05199b24b4b3143d5e94c1b40a389dde0dcfbd1ceef3266b7"}
Nov 03 20:04:00.913665 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:00.913638    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-rwhnf"]
Nov 03 20:04:00.921108 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:04:00.921080    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb9bef9b_c7cd_410e_8c1e_1b27474daf81.slice/crio-212fa88bc72490cb2e7aee7e40dbef30ea7a30c81b02dce4efa80ce99a6a0225 WatchSource:0}: Error finding container 212fa88bc72490cb2e7aee7e40dbef30ea7a30c81b02dce4efa80ce99a6a0225: Status 404 returned error can't find the container with id 212fa88bc72490cb2e7aee7e40dbef30ea7a30c81b02dce4efa80ce99a6a0225
Nov 03 20:04:00.938146 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:00.937575    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-7zl2f" podStartSLOduration=2.084021408 podStartE2EDuration="3.937557763s" totalImagesPullingTime="1.853536355s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:03:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:04:00.936638276 +0000 UTC m=+46.796750811" watchObservedRunningTime="2026-11-03 20:04:00.937557763 +0000 UTC m=+46.797670301"
Nov 03 20:04:00.970221 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:00.970193    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-q8b6s"]
Nov 03 20:04:01.646117 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:01.646095    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 20:04:01.654528 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:01.652941    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:04:01.665833 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:01.665799    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Nov 03 20:04:01.665939 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:01.665871    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Nov 03 20:04:01.665939 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:01.665874    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Nov 03 20:04:01.665939 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:01.665878    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Nov 03 20:04:01.665939 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:01.665931    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Nov 03 20:04:01.666108 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:01.666082    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-rnjcd\""
Nov 03 20:04:01.666215 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:01.666168    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Nov 03 20:04:01.666215 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:01.666199    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Nov 03 20:04:01.666330 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:01.666246    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Nov 03 20:04:01.666908 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:01.666887    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Nov 03 20:04:01.687879 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:01.687852    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 20:04:01.745475 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:01.745440    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f7d30630-43af-4157-b90f-d7eddef25085-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f7d30630-43af-4157-b90f-d7eddef25085\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:04:01.745614 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:01.745491    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f7d30630-43af-4157-b90f-d7eddef25085-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f7d30630-43af-4157-b90f-d7eddef25085\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:04:01.745614 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:01.745523    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f7d30630-43af-4157-b90f-d7eddef25085-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f7d30630-43af-4157-b90f-d7eddef25085\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:04:01.745614 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:01.745568    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f7d30630-43af-4157-b90f-d7eddef25085-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f7d30630-43af-4157-b90f-d7eddef25085\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:04:01.745715 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:01.745616    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f7d30630-43af-4157-b90f-d7eddef25085-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f7d30630-43af-4157-b90f-d7eddef25085\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:04:01.745715 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:01.745640    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f7d30630-43af-4157-b90f-d7eddef25085-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f7d30630-43af-4157-b90f-d7eddef25085\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:04:01.745715 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:01.745684    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f7d30630-43af-4157-b90f-d7eddef25085-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f7d30630-43af-4157-b90f-d7eddef25085\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:04:01.745841 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:01.745710    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f7d30630-43af-4157-b90f-d7eddef25085-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f7d30630-43af-4157-b90f-d7eddef25085\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:04:01.745841 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:01.745751    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z67jw\" (UniqueName: \"kubernetes.io/projected/f7d30630-43af-4157-b90f-d7eddef25085-kube-api-access-z67jw\") pod \"alertmanager-main-0\" (UID: \"f7d30630-43af-4157-b90f-d7eddef25085\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:04:01.745841 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:01.745834    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f7d30630-43af-4157-b90f-d7eddef25085-web-config\") pod \"alertmanager-main-0\" (UID: \"f7d30630-43af-4157-b90f-d7eddef25085\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:04:01.745990 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:01.745866    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f7d30630-43af-4157-b90f-d7eddef25085-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f7d30630-43af-4157-b90f-d7eddef25085\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:04:01.745990 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:01.745902    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f7d30630-43af-4157-b90f-d7eddef25085-config-volume\") pod \"alertmanager-main-0\" (UID: \"f7d30630-43af-4157-b90f-d7eddef25085\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:04:01.745990 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:01.745967    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f7d30630-43af-4157-b90f-d7eddef25085-config-out\") pod \"alertmanager-main-0\" (UID: \"f7d30630-43af-4157-b90f-d7eddef25085\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:04:01.847380 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:01.847341    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f7d30630-43af-4157-b90f-d7eddef25085-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f7d30630-43af-4157-b90f-d7eddef25085\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:04:01.847551 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:01.847389    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f7d30630-43af-4157-b90f-d7eddef25085-config-volume\") pod \"alertmanager-main-0\" (UID: \"f7d30630-43af-4157-b90f-d7eddef25085\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:04:01.847551 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:01.847428    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f7d30630-43af-4157-b90f-d7eddef25085-config-out\") pod \"alertmanager-main-0\" (UID: \"f7d30630-43af-4157-b90f-d7eddef25085\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:04:01.847551 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:01.847460    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f7d30630-43af-4157-b90f-d7eddef25085-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f7d30630-43af-4157-b90f-d7eddef25085\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:04:01.847551 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:01.847484    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f7d30630-43af-4157-b90f-d7eddef25085-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f7d30630-43af-4157-b90f-d7eddef25085\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:04:01.847551 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:01.847505    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f7d30630-43af-4157-b90f-d7eddef25085-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f7d30630-43af-4157-b90f-d7eddef25085\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:04:01.847551 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:01.847530    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f7d30630-43af-4157-b90f-d7eddef25085-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f7d30630-43af-4157-b90f-d7eddef25085\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:04:01.847823 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:01.847565    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f7d30630-43af-4157-b90f-d7eddef25085-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f7d30630-43af-4157-b90f-d7eddef25085\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:04:01.847823 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:01.847580    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f7d30630-43af-4157-b90f-d7eddef25085-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f7d30630-43af-4157-b90f-d7eddef25085\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:04:01.847823 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:01.847611    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f7d30630-43af-4157-b90f-d7eddef25085-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f7d30630-43af-4157-b90f-d7eddef25085\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:04:01.847823 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:01.847629    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f7d30630-43af-4157-b90f-d7eddef25085-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f7d30630-43af-4157-b90f-d7eddef25085\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:04:01.847823 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:01.847645    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z67jw\" (UniqueName: \"kubernetes.io/projected/f7d30630-43af-4157-b90f-d7eddef25085-kube-api-access-z67jw\") pod \"alertmanager-main-0\" (UID: \"f7d30630-43af-4157-b90f-d7eddef25085\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:04:01.847823 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:01.847658    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f7d30630-43af-4157-b90f-d7eddef25085-web-config\") pod \"alertmanager-main-0\" (UID: \"f7d30630-43af-4157-b90f-d7eddef25085\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:04:01.851005 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:01.850220    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f7d30630-43af-4157-b90f-d7eddef25085-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f7d30630-43af-4157-b90f-d7eddef25085\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:04:01.851005 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:01.850296    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f7d30630-43af-4157-b90f-d7eddef25085-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f7d30630-43af-4157-b90f-d7eddef25085\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:04:01.851005 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:01.850631    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f7d30630-43af-4157-b90f-d7eddef25085-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f7d30630-43af-4157-b90f-d7eddef25085\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:04:01.868311 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:01.868284    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f7d30630-43af-4157-b90f-d7eddef25085-web-config\") pod \"alertmanager-main-0\" (UID: \"f7d30630-43af-4157-b90f-d7eddef25085\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:04:01.871489 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:01.870428    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f7d30630-43af-4157-b90f-d7eddef25085-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f7d30630-43af-4157-b90f-d7eddef25085\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:04:01.871489 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:01.870578    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f7d30630-43af-4157-b90f-d7eddef25085-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f7d30630-43af-4157-b90f-d7eddef25085\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:04:01.871489 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:01.870681    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f7d30630-43af-4157-b90f-d7eddef25085-config-out\") pod \"alertmanager-main-0\" (UID: \"f7d30630-43af-4157-b90f-d7eddef25085\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:04:01.871489 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:01.870939    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f7d30630-43af-4157-b90f-d7eddef25085-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f7d30630-43af-4157-b90f-d7eddef25085\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:04:01.871489 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:01.871014    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f7d30630-43af-4157-b90f-d7eddef25085-config-volume\") pod \"alertmanager-main-0\" (UID: \"f7d30630-43af-4157-b90f-d7eddef25085\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:04:01.871489 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:01.871133    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f7d30630-43af-4157-b90f-d7eddef25085-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f7d30630-43af-4157-b90f-d7eddef25085\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:04:01.871489 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:01.871381    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f7d30630-43af-4157-b90f-d7eddef25085-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f7d30630-43af-4157-b90f-d7eddef25085\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:04:01.871489 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:01.871456    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z67jw\" (UniqueName: \"kubernetes.io/projected/f7d30630-43af-4157-b90f-d7eddef25085-kube-api-access-z67jw\") pod \"alertmanager-main-0\" (UID: \"f7d30630-43af-4157-b90f-d7eddef25085\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:04:01.872456 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:01.872433    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f7d30630-43af-4157-b90f-d7eddef25085-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f7d30630-43af-4157-b90f-d7eddef25085\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:04:01.911402 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:01.911369    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8bwhz" event={"ID":"884cbb80-b3f8-4f90-bd55-328d17943cf3","Type":"ContainerStarted","Data":"842c94311d675726febea5cd6198eb4803500c116c32b5fcb02a5fd90522f558"}
Nov 03 20:04:01.914961 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:01.914925    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-rwhnf" event={"ID":"cb9bef9b-c7cd-410e-8c1e-1b27474daf81","Type":"ContainerStarted","Data":"228b416a30d32c336ca4f7a8b287a4a5fdbcee11fc72e99c7bf3312d3b142821"}
Nov 03 20:04:01.915066 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:01.914996    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-rwhnf" event={"ID":"cb9bef9b-c7cd-410e-8c1e-1b27474daf81","Type":"ContainerStarted","Data":"ce0c3a1ca353b2c89a8721e0810d85de828c5332c16ec6c003db01cf17778150"}
Nov 03 20:04:01.915066 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:01.915023    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-rwhnf" event={"ID":"cb9bef9b-c7cd-410e-8c1e-1b27474daf81","Type":"ContainerStarted","Data":"212fa88bc72490cb2e7aee7e40dbef30ea7a30c81b02dce4efa80ce99a6a0225"}
Nov 03 20:04:01.916858 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:01.916828    2577 generic.go:356] "Generic (PLEG): container finished" podID="74d4808e-24ce-4c47-b4ce-e1188eeea6f1" containerID="4e6d31cdfaa90ea53ffda45d72fa7ea49e5905a25dc84b2607056358cf3863ed" exitCode=0
Nov 03 20:04:01.917157 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:01.917102    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-j46kr" event={"ID":"74d4808e-24ce-4c47-b4ce-e1188eeea6f1","Type":"ContainerDied","Data":"4e6d31cdfaa90ea53ffda45d72fa7ea49e5905a25dc84b2607056358cf3863ed"}
Nov 03 20:04:01.921563 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:01.921214    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-q8b6s" event={"ID":"ffe074b1-e2a0-403b-9f81-581fdb4d2775","Type":"ContainerStarted","Data":"5f9800b7bb6c494e33253ca718f90909053640f14dee0e4169958b54fb015d21"}
Nov 03 20:04:01.947457 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:01.946291    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-8bwhz" podStartSLOduration=2.81790935 podStartE2EDuration="4.946272759s" totalImagesPullingTime="2.128363409s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:03:57 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:04:01.945435251 +0000 UTC m=+47.805547787" watchObservedRunningTime="2026-11-03 20:04:01.946272759 +0000 UTC m=+47.806385298"
Nov 03 20:04:01.982641 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:01.982581    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:04:02.321930 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:02.321901    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 20:04:02.325391 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:04:02.325290    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7d30630_43af_4157_b90f_d7eddef25085.slice/crio-8ef0ef34d0b75303f53cea650dd4280a76b4f1e5813cd730e128a5fd3b3b726e WatchSource:0}: Error finding container 8ef0ef34d0b75303f53cea650dd4280a76b4f1e5813cd730e128a5fd3b3b726e: Status 404 returned error can't find the container with id 8ef0ef34d0b75303f53cea650dd4280a76b4f1e5813cd730e128a5fd3b3b726e
Nov 03 20:04:02.486957 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:02.486918    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-5bd4bfb68d-9449x"]
Nov 03 20:04:02.489879 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:02.489854    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5bd4bfb68d-9449x"
Nov 03 20:04:02.510653 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:02.510615    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5bd4bfb68d-9449x"]
Nov 03 20:04:02.512616 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:02.512591    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Nov 03 20:04:02.512910 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:02.512879    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Nov 03 20:04:02.512910 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:02.512909    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-m22xr\""
Nov 03 20:04:02.513075 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:02.512891    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Nov 03 20:04:02.513075 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:02.512973    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Nov 03 20:04:02.513075 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:02.512977    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Nov 03 20:04:02.513075 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:02.512996    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-6ldnaap773utg\""
Nov 03 20:04:02.656454 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:02.656361    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/59f6a4d6-afe0-4c96-a54a-d8e6b5c66a38-metrics-client-ca\") pod \"thanos-querier-5bd4bfb68d-9449x\" (UID: \"59f6a4d6-afe0-4c96-a54a-d8e6b5c66a38\") " pod="openshift-monitoring/thanos-querier-5bd4bfb68d-9449x"
Nov 03 20:04:02.656454 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:02.656409    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/59f6a4d6-afe0-4c96-a54a-d8e6b5c66a38-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5bd4bfb68d-9449x\" (UID: \"59f6a4d6-afe0-4c96-a54a-d8e6b5c66a38\") " pod="openshift-monitoring/thanos-querier-5bd4bfb68d-9449x"
Nov 03 20:04:02.656931 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:02.656523    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/59f6a4d6-afe0-4c96-a54a-d8e6b5c66a38-secret-grpc-tls\") pod \"thanos-querier-5bd4bfb68d-9449x\" (UID: \"59f6a4d6-afe0-4c96-a54a-d8e6b5c66a38\") " pod="openshift-monitoring/thanos-querier-5bd4bfb68d-9449x"
Nov 03 20:04:02.656931 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:02.656561    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/59f6a4d6-afe0-4c96-a54a-d8e6b5c66a38-tmp\") pod \"thanos-querier-5bd4bfb68d-9449x\" (UID: \"59f6a4d6-afe0-4c96-a54a-d8e6b5c66a38\") " pod="openshift-monitoring/thanos-querier-5bd4bfb68d-9449x"
Nov 03 20:04:02.656931 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:02.656658    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/59f6a4d6-afe0-4c96-a54a-d8e6b5c66a38-secret-thanos-querier-tls\") pod \"thanos-querier-5bd4bfb68d-9449x\" (UID: \"59f6a4d6-afe0-4c96-a54a-d8e6b5c66a38\") " pod="openshift-monitoring/thanos-querier-5bd4bfb68d-9449x"
Nov 03 20:04:02.656931 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:02.656698    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7hn2\" (UniqueName: \"kubernetes.io/projected/59f6a4d6-afe0-4c96-a54a-d8e6b5c66a38-kube-api-access-w7hn2\") pod \"thanos-querier-5bd4bfb68d-9449x\" (UID: \"59f6a4d6-afe0-4c96-a54a-d8e6b5c66a38\") " pod="openshift-monitoring/thanos-querier-5bd4bfb68d-9449x"
Nov 03 20:04:02.656931 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:02.656733    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/59f6a4d6-afe0-4c96-a54a-d8e6b5c66a38-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5bd4bfb68d-9449x\" (UID: \"59f6a4d6-afe0-4c96-a54a-d8e6b5c66a38\") " pod="openshift-monitoring/thanos-querier-5bd4bfb68d-9449x"
Nov 03 20:04:02.656931 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:02.656769    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/59f6a4d6-afe0-4c96-a54a-d8e6b5c66a38-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5bd4bfb68d-9449x\" (UID: \"59f6a4d6-afe0-4c96-a54a-d8e6b5c66a38\") " pod="openshift-monitoring/thanos-querier-5bd4bfb68d-9449x"
Nov 03 20:04:02.656931 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:02.656836    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/59f6a4d6-afe0-4c96-a54a-d8e6b5c66a38-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5bd4bfb68d-9449x\" (UID: \"59f6a4d6-afe0-4c96-a54a-d8e6b5c66a38\") " pod="openshift-monitoring/thanos-querier-5bd4bfb68d-9449x"
Nov 03 20:04:02.757882 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:02.757842    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/59f6a4d6-afe0-4c96-a54a-d8e6b5c66a38-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5bd4bfb68d-9449x\" (UID: \"59f6a4d6-afe0-4c96-a54a-d8e6b5c66a38\") " pod="openshift-monitoring/thanos-querier-5bd4bfb68d-9449x"
Nov 03 20:04:02.758081 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:02.757918    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/59f6a4d6-afe0-4c96-a54a-d8e6b5c66a38-secret-grpc-tls\") pod \"thanos-querier-5bd4bfb68d-9449x\" (UID: \"59f6a4d6-afe0-4c96-a54a-d8e6b5c66a38\") " pod="openshift-monitoring/thanos-querier-5bd4bfb68d-9449x"
Nov 03 20:04:02.758081 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:02.757947    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/59f6a4d6-afe0-4c96-a54a-d8e6b5c66a38-tmp\") pod \"thanos-querier-5bd4bfb68d-9449x\" (UID: \"59f6a4d6-afe0-4c96-a54a-d8e6b5c66a38\") " pod="openshift-monitoring/thanos-querier-5bd4bfb68d-9449x"
Nov 03 20:04:02.758081 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:02.757989    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/59f6a4d6-afe0-4c96-a54a-d8e6b5c66a38-secret-thanos-querier-tls\") pod \"thanos-querier-5bd4bfb68d-9449x\" (UID: \"59f6a4d6-afe0-4c96-a54a-d8e6b5c66a38\") " pod="openshift-monitoring/thanos-querier-5bd4bfb68d-9449x"
Nov 03 20:04:02.758081 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:02.758012    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7hn2\" (UniqueName: \"kubernetes.io/projected/59f6a4d6-afe0-4c96-a54a-d8e6b5c66a38-kube-api-access-w7hn2\") pod \"thanos-querier-5bd4bfb68d-9449x\" (UID: \"59f6a4d6-afe0-4c96-a54a-d8e6b5c66a38\") " pod="openshift-monitoring/thanos-querier-5bd4bfb68d-9449x"
Nov 03 20:04:02.758081 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:02.758038    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/59f6a4d6-afe0-4c96-a54a-d8e6b5c66a38-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5bd4bfb68d-9449x\" (UID: \"59f6a4d6-afe0-4c96-a54a-d8e6b5c66a38\") " pod="openshift-monitoring/thanos-querier-5bd4bfb68d-9449x"
Nov 03 20:04:02.758081 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:02.758062    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/59f6a4d6-afe0-4c96-a54a-d8e6b5c66a38-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5bd4bfb68d-9449x\" (UID: \"59f6a4d6-afe0-4c96-a54a-d8e6b5c66a38\") " pod="openshift-monitoring/thanos-querier-5bd4bfb68d-9449x"
Nov 03 20:04:02.758362 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:02.758095    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/59f6a4d6-afe0-4c96-a54a-d8e6b5c66a38-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5bd4bfb68d-9449x\" (UID: \"59f6a4d6-afe0-4c96-a54a-d8e6b5c66a38\") " pod="openshift-monitoring/thanos-querier-5bd4bfb68d-9449x"
Nov 03 20:04:02.758362 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:02.758173    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/59f6a4d6-afe0-4c96-a54a-d8e6b5c66a38-metrics-client-ca\") pod \"thanos-querier-5bd4bfb68d-9449x\" (UID: \"59f6a4d6-afe0-4c96-a54a-d8e6b5c66a38\") " pod="openshift-monitoring/thanos-querier-5bd4bfb68d-9449x"
Nov 03 20:04:02.758869 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:02.758816    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/59f6a4d6-afe0-4c96-a54a-d8e6b5c66a38-tmp\") pod \"thanos-querier-5bd4bfb68d-9449x\" (UID: \"59f6a4d6-afe0-4c96-a54a-d8e6b5c66a38\") " pod="openshift-monitoring/thanos-querier-5bd4bfb68d-9449x"
Nov 03 20:04:02.759033 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:02.759010    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/59f6a4d6-afe0-4c96-a54a-d8e6b5c66a38-metrics-client-ca\") pod \"thanos-querier-5bd4bfb68d-9449x\" (UID: \"59f6a4d6-afe0-4c96-a54a-d8e6b5c66a38\") " pod="openshift-monitoring/thanos-querier-5bd4bfb68d-9449x"
Nov 03 20:04:02.761214 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:02.761190    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/59f6a4d6-afe0-4c96-a54a-d8e6b5c66a38-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5bd4bfb68d-9449x\" (UID: \"59f6a4d6-afe0-4c96-a54a-d8e6b5c66a38\") " pod="openshift-monitoring/thanos-querier-5bd4bfb68d-9449x"
Nov 03 20:04:02.763201 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:02.763176    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/59f6a4d6-afe0-4c96-a54a-d8e6b5c66a38-secret-grpc-tls\") pod \"thanos-querier-5bd4bfb68d-9449x\" (UID: \"59f6a4d6-afe0-4c96-a54a-d8e6b5c66a38\") " pod="openshift-monitoring/thanos-querier-5bd4bfb68d-9449x"
Nov 03 20:04:02.763749 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:02.763667    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/59f6a4d6-afe0-4c96-a54a-d8e6b5c66a38-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5bd4bfb68d-9449x\" (UID: \"59f6a4d6-afe0-4c96-a54a-d8e6b5c66a38\") " pod="openshift-monitoring/thanos-querier-5bd4bfb68d-9449x"
Nov 03 20:04:02.763749 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:02.763692    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/59f6a4d6-afe0-4c96-a54a-d8e6b5c66a38-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5bd4bfb68d-9449x\" (UID: \"59f6a4d6-afe0-4c96-a54a-d8e6b5c66a38\") " pod="openshift-monitoring/thanos-querier-5bd4bfb68d-9449x"
Nov 03 20:04:02.763932 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:02.763762    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/59f6a4d6-afe0-4c96-a54a-d8e6b5c66a38-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5bd4bfb68d-9449x\" (UID: \"59f6a4d6-afe0-4c96-a54a-d8e6b5c66a38\") " pod="openshift-monitoring/thanos-querier-5bd4bfb68d-9449x"
Nov 03 20:04:02.765632 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:02.765607    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/59f6a4d6-afe0-4c96-a54a-d8e6b5c66a38-secret-thanos-querier-tls\") pod \"thanos-querier-5bd4bfb68d-9449x\" (UID: \"59f6a4d6-afe0-4c96-a54a-d8e6b5c66a38\") " pod="openshift-monitoring/thanos-querier-5bd4bfb68d-9449x"
Nov 03 20:04:02.775741 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:02.775712    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7hn2\" (UniqueName: \"kubernetes.io/projected/59f6a4d6-afe0-4c96-a54a-d8e6b5c66a38-kube-api-access-w7hn2\") pod \"thanos-querier-5bd4bfb68d-9449x\" (UID: \"59f6a4d6-afe0-4c96-a54a-d8e6b5c66a38\") " pod="openshift-monitoring/thanos-querier-5bd4bfb68d-9449x"
Nov 03 20:04:02.807455 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:02.807419    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5bd4bfb68d-9449x"
Nov 03 20:04:02.926199 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:02.926112    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-5jnts" event={"ID":"3f681106-32d4-4550-a00d-1f9c52060b57","Type":"ContainerStarted","Data":"f0eb46bc7c7cca048ccd3196eb756ebba543ab89207120a4fb3b456e5ddb4ba8"}
Nov 03 20:04:02.929022 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:02.928500    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-j46kr" event={"ID":"74d4808e-24ce-4c47-b4ce-e1188eeea6f1","Type":"ContainerStarted","Data":"5e533a0f26d62247c576792d48e28c1119c1009df0015851ee9d260fe30d1a53"}
Nov 03 20:04:02.929022 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:02.928565    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-j46kr" event={"ID":"74d4808e-24ce-4c47-b4ce-e1188eeea6f1","Type":"ContainerStarted","Data":"2cf5b58f6bcf7e3449b181f8dcbb0a3086c4c3727237c4d59841a75f65cc1f7c"}
Nov 03 20:04:02.930832 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:02.930797    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-q8b6s" event={"ID":"ffe074b1-e2a0-403b-9f81-581fdb4d2775","Type":"ContainerStarted","Data":"456ee8d509cfaf5c74098ba0a4507e3b138fe9820c8686372c92ee7412655e61"}
Nov 03 20:04:02.930944 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:02.930870    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-q8b6s" event={"ID":"ffe074b1-e2a0-403b-9f81-581fdb4d2775","Type":"ContainerStarted","Data":"8c904c9a0096d9289225cdd6eb4347c0d04e06858202ad199a6e7b004f7054fa"}
Nov 03 20:04:02.930944 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:02.930901    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-q8b6s" event={"ID":"ffe074b1-e2a0-403b-9f81-581fdb4d2775","Type":"ContainerStarted","Data":"b07e5ac0234feb788fe9fc36c82279077162422273d430595919ad9652b479cd"}
Nov 03 20:04:02.933451 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:02.932996    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-rwhnf" event={"ID":"cb9bef9b-c7cd-410e-8c1e-1b27474daf81","Type":"ContainerStarted","Data":"fff039b90674c21ca39cd5da1b359be906ddcf487004112b8d7786d4fa404a80"}
Nov 03 20:04:02.934317 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:02.934292    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f7d30630-43af-4157-b90f-d7eddef25085","Type":"ContainerStarted","Data":"8ef0ef34d0b75303f53cea650dd4280a76b4f1e5813cd730e128a5fd3b3b726e"}
Nov 03 20:04:02.961627 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:02.961572    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-5jnts" podStartSLOduration=2.7067938529999997 podStartE2EDuration="5.961559141s" totalImagesPullingTime="3.254765288s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:03:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:04:02.961265727 +0000 UTC m=+48.821378264" watchObservedRunningTime="2026-11-03 20:04:02.961559141 +0000 UTC m=+48.821671677"
Nov 03 20:04:02.999881 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:02.999826    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-j46kr" podStartSLOduration=1.938037437 podStartE2EDuration="2.999812984s" totalImagesPullingTime="974.957187ms" totalInitContainerRuntime="86.81836ms" isStatefulPod=true podCreationTimestamp="2026-11-03 20:04:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:04:02.998757981 +0000 UTC m=+48.858870516" watchObservedRunningTime="2026-11-03 20:04:02.999812984 +0000 UTC m=+48.859925520"
Nov 03 20:04:03.034272 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:03.034218    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-rwhnf" podStartSLOduration=1.957093148 podStartE2EDuration="3.034203345s" totalImagesPullingTime="1.077110197s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:04:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:04:03.033762872 +0000 UTC m=+48.893875410" watchObservedRunningTime="2026-11-03 20:04:03.034203345 +0000 UTC m=+48.894315910"
Nov 03 20:04:03.069569 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:03.069505    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-q8b6s" podStartSLOduration=1.7353619569999998 podStartE2EDuration="3.069488595s" totalImagesPullingTime="1.334126638s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:04:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:04:03.068028902 +0000 UTC m=+48.928141441" watchObservedRunningTime="2026-11-03 20:04:03.069488595 +0000 UTC m=+48.929601180"
Nov 03 20:04:03.118532 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:03.118496    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5bd4bfb68d-9449x"]
Nov 03 20:04:03.123166 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:04:03.123132    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59f6a4d6_afe0_4c96_a54a_d8e6b5c66a38.slice/crio-c7cefb72ae55f2db75ebd2c7f2795449854b5f8f786ac9b0aa02eb6fb0ab5b45 WatchSource:0}: Error finding container c7cefb72ae55f2db75ebd2c7f2795449854b5f8f786ac9b0aa02eb6fb0ab5b45: Status 404 returned error can't find the container with id c7cefb72ae55f2db75ebd2c7f2795449854b5f8f786ac9b0aa02eb6fb0ab5b45
Nov 03 20:04:03.939645 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:03.939596    2577 generic.go:356] "Generic (PLEG): container finished" podID="f7d30630-43af-4157-b90f-d7eddef25085" containerID="785eedffac63800711a89514aec53147c740e3e22adcbae196ce94713ef27782" exitCode=0
Nov 03 20:04:03.940151 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:03.939971    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f7d30630-43af-4157-b90f-d7eddef25085","Type":"ContainerDied","Data":"785eedffac63800711a89514aec53147c740e3e22adcbae196ce94713ef27782"}
Nov 03 20:04:03.941878 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:03.941725    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5bd4bfb68d-9449x" event={"ID":"59f6a4d6-afe0-4c96-a54a-d8e6b5c66a38","Type":"ContainerStarted","Data":"c7cefb72ae55f2db75ebd2c7f2795449854b5f8f786ac9b0aa02eb6fb0ab5b45"}
Nov 03 20:04:04.783025 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:04.782989    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-654b7fcdc8-l6l9f"]
Nov 03 20:04:04.786587 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:04.786561    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-654b7fcdc8-l6l9f"
Nov 03 20:04:04.811593 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:04.808823    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-654b7fcdc8-l6l9f"]
Nov 03 20:04:04.817309 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:04.817281    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Nov 03 20:04:04.817462 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:04.817406    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Nov 03 20:04:04.817607 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:04.817583    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-mbtbz\""
Nov 03 20:04:04.817720 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:04.817622    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Nov 03 20:04:04.817720 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:04.817668    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Nov 03 20:04:04.817720 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:04.817713    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-ao94rttoovh0l\""
Nov 03 20:04:04.881999 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:04.881923    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wq4p\" (UniqueName: \"kubernetes.io/projected/d5d14caa-6661-4146-b888-59c3518216d4-kube-api-access-8wq4p\") pod \"metrics-server-654b7fcdc8-l6l9f\" (UID: \"d5d14caa-6661-4146-b888-59c3518216d4\") " pod="openshift-monitoring/metrics-server-654b7fcdc8-l6l9f"
Nov 03 20:04:04.882178 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:04.882067    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/d5d14caa-6661-4146-b888-59c3518216d4-secret-metrics-server-tls\") pod \"metrics-server-654b7fcdc8-l6l9f\" (UID: \"d5d14caa-6661-4146-b888-59c3518216d4\") " pod="openshift-monitoring/metrics-server-654b7fcdc8-l6l9f"
Nov 03 20:04:04.882178 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:04.882135    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/d5d14caa-6661-4146-b888-59c3518216d4-secret-metrics-server-client-certs\") pod \"metrics-server-654b7fcdc8-l6l9f\" (UID: \"d5d14caa-6661-4146-b888-59c3518216d4\") " pod="openshift-monitoring/metrics-server-654b7fcdc8-l6l9f"
Nov 03 20:04:04.882178 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:04.882167    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/d5d14caa-6661-4146-b888-59c3518216d4-audit-log\") pod \"metrics-server-654b7fcdc8-l6l9f\" (UID: \"d5d14caa-6661-4146-b888-59c3518216d4\") " pod="openshift-monitoring/metrics-server-654b7fcdc8-l6l9f"
Nov 03 20:04:04.882332 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:04.882200    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5d14caa-6661-4146-b888-59c3518216d4-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-654b7fcdc8-l6l9f\" (UID: \"d5d14caa-6661-4146-b888-59c3518216d4\") " pod="openshift-monitoring/metrics-server-654b7fcdc8-l6l9f"
Nov 03 20:04:04.882332 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:04.882219    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/d5d14caa-6661-4146-b888-59c3518216d4-client-ca-bundle\") pod \"metrics-server-654b7fcdc8-l6l9f\" (UID: \"d5d14caa-6661-4146-b888-59c3518216d4\") " pod="openshift-monitoring/metrics-server-654b7fcdc8-l6l9f"
Nov 03 20:04:04.882332 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:04.882253    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/d5d14caa-6661-4146-b888-59c3518216d4-metrics-server-audit-profiles\") pod \"metrics-server-654b7fcdc8-l6l9f\" (UID: \"d5d14caa-6661-4146-b888-59c3518216d4\") " pod="openshift-monitoring/metrics-server-654b7fcdc8-l6l9f"
Nov 03 20:04:04.983004 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:04.982966    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wq4p\" (UniqueName: \"kubernetes.io/projected/d5d14caa-6661-4146-b888-59c3518216d4-kube-api-access-8wq4p\") pod \"metrics-server-654b7fcdc8-l6l9f\" (UID: \"d5d14caa-6661-4146-b888-59c3518216d4\") " pod="openshift-monitoring/metrics-server-654b7fcdc8-l6l9f"
Nov 03 20:04:04.983497 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:04.983050    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/d5d14caa-6661-4146-b888-59c3518216d4-secret-metrics-server-tls\") pod \"metrics-server-654b7fcdc8-l6l9f\" (UID: \"d5d14caa-6661-4146-b888-59c3518216d4\") " pod="openshift-monitoring/metrics-server-654b7fcdc8-l6l9f"
Nov 03 20:04:04.983497 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:04.983096    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/d5d14caa-6661-4146-b888-59c3518216d4-secret-metrics-server-client-certs\") pod \"metrics-server-654b7fcdc8-l6l9f\" (UID: \"d5d14caa-6661-4146-b888-59c3518216d4\") " pod="openshift-monitoring/metrics-server-654b7fcdc8-l6l9f"
Nov 03 20:04:04.983497 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:04.983127    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/d5d14caa-6661-4146-b888-59c3518216d4-audit-log\") pod \"metrics-server-654b7fcdc8-l6l9f\" (UID: \"d5d14caa-6661-4146-b888-59c3518216d4\") " pod="openshift-monitoring/metrics-server-654b7fcdc8-l6l9f"
Nov 03 20:04:04.983497 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:04.983182    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5d14caa-6661-4146-b888-59c3518216d4-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-654b7fcdc8-l6l9f\" (UID: \"d5d14caa-6661-4146-b888-59c3518216d4\") " pod="openshift-monitoring/metrics-server-654b7fcdc8-l6l9f"
Nov 03 20:04:04.983497 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:04.983211    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/d5d14caa-6661-4146-b888-59c3518216d4-client-ca-bundle\") pod \"metrics-server-654b7fcdc8-l6l9f\" (UID: \"d5d14caa-6661-4146-b888-59c3518216d4\") " pod="openshift-monitoring/metrics-server-654b7fcdc8-l6l9f"
Nov 03 20:04:04.983497 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:04.983248    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/d5d14caa-6661-4146-b888-59c3518216d4-metrics-server-audit-profiles\") pod \"metrics-server-654b7fcdc8-l6l9f\" (UID: \"d5d14caa-6661-4146-b888-59c3518216d4\") " pod="openshift-monitoring/metrics-server-654b7fcdc8-l6l9f"
Nov 03 20:04:04.984442 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:04.984416    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/d5d14caa-6661-4146-b888-59c3518216d4-metrics-server-audit-profiles\") pod \"metrics-server-654b7fcdc8-l6l9f\" (UID: \"d5d14caa-6661-4146-b888-59c3518216d4\") " pod="openshift-monitoring/metrics-server-654b7fcdc8-l6l9f"
Nov 03 20:04:04.984752 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:04.984722    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/d5d14caa-6661-4146-b888-59c3518216d4-audit-log\") pod \"metrics-server-654b7fcdc8-l6l9f\" (UID: \"d5d14caa-6661-4146-b888-59c3518216d4\") " pod="openshift-monitoring/metrics-server-654b7fcdc8-l6l9f"
Nov 03 20:04:04.985652 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:04.985618    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5d14caa-6661-4146-b888-59c3518216d4-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-654b7fcdc8-l6l9f\" (UID: \"d5d14caa-6661-4146-b888-59c3518216d4\") " pod="openshift-monitoring/metrics-server-654b7fcdc8-l6l9f"
Nov 03 20:04:04.987393 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:04.987371    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/d5d14caa-6661-4146-b888-59c3518216d4-secret-metrics-server-client-certs\") pod \"metrics-server-654b7fcdc8-l6l9f\" (UID: \"d5d14caa-6661-4146-b888-59c3518216d4\") " pod="openshift-monitoring/metrics-server-654b7fcdc8-l6l9f"
Nov 03 20:04:04.988241 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:04.988221    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/d5d14caa-6661-4146-b888-59c3518216d4-secret-metrics-server-tls\") pod \"metrics-server-654b7fcdc8-l6l9f\" (UID: \"d5d14caa-6661-4146-b888-59c3518216d4\") " pod="openshift-monitoring/metrics-server-654b7fcdc8-l6l9f"
Nov 03 20:04:04.988334 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:04.988290    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/d5d14caa-6661-4146-b888-59c3518216d4-client-ca-bundle\") pod \"metrics-server-654b7fcdc8-l6l9f\" (UID: \"d5d14caa-6661-4146-b888-59c3518216d4\") " pod="openshift-monitoring/metrics-server-654b7fcdc8-l6l9f"
Nov 03 20:04:05.000908 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:05.000881    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wq4p\" (UniqueName: \"kubernetes.io/projected/d5d14caa-6661-4146-b888-59c3518216d4-kube-api-access-8wq4p\") pod \"metrics-server-654b7fcdc8-l6l9f\" (UID: \"d5d14caa-6661-4146-b888-59c3518216d4\") " pod="openshift-monitoring/metrics-server-654b7fcdc8-l6l9f"
Nov 03 20:04:05.100514 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:05.100425    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-654b7fcdc8-l6l9f"
Nov 03 20:04:05.427820 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:05.427796    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-654b7fcdc8-l6l9f"]
Nov 03 20:04:05.436880 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:04:05.434941    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5d14caa_6661_4146_b888_59c3518216d4.slice/crio-6fba5ae523e88c172564490952767809562c59f6a0322d27757e10868226a8c0 WatchSource:0}: Error finding container 6fba5ae523e88c172564490952767809562c59f6a0322d27757e10868226a8c0: Status 404 returned error can't find the container with id 6fba5ae523e88c172564490952767809562c59f6a0322d27757e10868226a8c0
Nov 03 20:04:05.956903 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:05.956633    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f7d30630-43af-4157-b90f-d7eddef25085","Type":"ContainerStarted","Data":"960a709deb8999b7ce32bdc8df50230a2b1c396c9f4b7fec4565ce238051471c"}
Nov 03 20:04:05.956903 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:05.956708    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f7d30630-43af-4157-b90f-d7eddef25085","Type":"ContainerStarted","Data":"014278ced716c2a52d7b544d736e420472f4440af29a73b56a93341b99fc7d51"}
Nov 03 20:04:05.956903 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:05.956748    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f7d30630-43af-4157-b90f-d7eddef25085","Type":"ContainerStarted","Data":"6bbb13e30ca02316f2656a078c3631aa8a4bee6760823627343e629d39e7dda7"}
Nov 03 20:04:05.956903 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:05.956803    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f7d30630-43af-4157-b90f-d7eddef25085","Type":"ContainerStarted","Data":"26307d79ee47d0ad8409c23eb27f0d2e6e77b80f6bcc80c48a111753adc9ab9c"}
Nov 03 20:04:05.956903 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:05.956851    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f7d30630-43af-4157-b90f-d7eddef25085","Type":"ContainerStarted","Data":"0ff023e230ba0d28d23d2743e9b0aba7c73e86105e55e4944b6cb93cc8a89c71"}
Nov 03 20:04:05.957972 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:05.957941    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-654b7fcdc8-l6l9f" event={"ID":"d5d14caa-6661-4146-b888-59c3518216d4","Type":"ContainerStarted","Data":"6fba5ae523e88c172564490952767809562c59f6a0322d27757e10868226a8c0"}
Nov 03 20:04:05.960158 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:05.960136    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5bd4bfb68d-9449x" event={"ID":"59f6a4d6-afe0-4c96-a54a-d8e6b5c66a38","Type":"ContainerStarted","Data":"dd5dd52df1d7a4fc8c3630482357e06329126b582ba797787b1f72183e5e168c"}
Nov 03 20:04:05.960245 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:05.960197    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5bd4bfb68d-9449x" event={"ID":"59f6a4d6-afe0-4c96-a54a-d8e6b5c66a38","Type":"ContainerStarted","Data":"e5dcac08f7ead22be0b471015e93798f146b2a0d6925b345dd1f380e47f17ab5"}
Nov 03 20:04:05.960245 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:05.960231    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5bd4bfb68d-9449x" event={"ID":"59f6a4d6-afe0-4c96-a54a-d8e6b5c66a38","Type":"ContainerStarted","Data":"0748b9022b6e4fdbdc0c052e3d2ee1fb2950f026a2512f81c51220c74bc48432"}
Nov 03 20:04:06.967491 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:06.967450    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f7d30630-43af-4157-b90f-d7eddef25085","Type":"ContainerStarted","Data":"e2504655bc35f9bba51ad6d1bfb0b7293d1790ba3ebcd18215ab07aea2386a41"}
Nov 03 20:04:06.969147 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:06.969119    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-654b7fcdc8-l6l9f" event={"ID":"d5d14caa-6661-4146-b888-59c3518216d4","Type":"ContainerStarted","Data":"db8f4cb4e559dc41ca19fbff175f4baa7558af35bcfd123c75ca21073d3506a4"}
Nov 03 20:04:06.972056 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:06.972033    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5bd4bfb68d-9449x" event={"ID":"59f6a4d6-afe0-4c96-a54a-d8e6b5c66a38","Type":"ContainerStarted","Data":"c6de2fac38daf1fa1949b21868b0208b42fda457831db8897066833f6f27f156"}
Nov 03 20:04:06.972167 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:06.972093    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5bd4bfb68d-9449x" event={"ID":"59f6a4d6-afe0-4c96-a54a-d8e6b5c66a38","Type":"ContainerStarted","Data":"c38ce81dea3d9f4214ae6ccf53c8325af6b824ca0a9cefe723f1e214ebf17a25"}
Nov 03 20:04:06.972167 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:06.972135    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5bd4bfb68d-9449x" event={"ID":"59f6a4d6-afe0-4c96-a54a-d8e6b5c66a38","Type":"ContainerStarted","Data":"5dec394b61af8fdb80611de2ab3af96ac744218ae21340140ce38c2e1861387c"}
Nov 03 20:04:06.972265 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:06.972207    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-5bd4bfb68d-9449x"
Nov 03 20:04:07.029009 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:07.028955    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.921467457 podStartE2EDuration="6.028941727s" totalImagesPullingTime="3.093629287s" totalInitContainerRuntime="13.844983ms" isStatefulPod=false podCreationTimestamp="2026-11-03 20:04:01 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:04:07.018652338 +0000 UTC m=+52.878764874" watchObservedRunningTime="2026-11-03 20:04:07.028941727 +0000 UTC m=+52.889054262"
Nov 03 20:04:07.053210 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:07.053101    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-654b7fcdc8-l6l9f" podStartSLOduration=1.7079653700000001 podStartE2EDuration="3.053083968s" totalImagesPullingTime="1.345118598s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:04:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:04:07.05248929 +0000 UTC m=+52.912601826" watchObservedRunningTime="2026-11-03 20:04:07.053083968 +0000 UTC m=+52.913196505"
Nov 03 20:04:07.091189 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:07.091123    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-5bd4bfb68d-9449x" podStartSLOduration=1.9007073559999998 podStartE2EDuration="5.0911012s" totalImagesPullingTime="3.190393844s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:04:02 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:04:07.089994404 +0000 UTC m=+52.950106940" watchObservedRunningTime="2026-11-03 20:04:07.0911012 +0000 UTC m=+52.951213740"
Nov 03 20:04:10.923896 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:10.923863    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-7zl2f"
Nov 03 20:04:12.984822 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:12.984769    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-5bd4bfb68d-9449x"
Nov 03 20:04:16.260022 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:16.259994    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6sv7j"
Nov 03 20:04:20.415673 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:20.415628    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c92a5d0-b34f-46a1-8f8f-a9ae3cb8d5a5-metrics-certs\") pod \"network-metrics-daemon-c947w\" (UID: \"4c92a5d0-b34f-46a1-8f8f-a9ae3cb8d5a5\") " pod="openshift-multus/network-metrics-daemon-c947w"
Nov 03 20:04:20.416107 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:20.415681    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/51dffd76-5c30-440c-85ed-d81221663833-original-pull-secret\") pod \"global-pull-secret-syncer-j4jzv\" (UID: \"51dffd76-5c30-440c-85ed-d81221663833\") " pod="kube-system/global-pull-secret-syncer-j4jzv"
Nov 03 20:04:20.436838 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:20.436810    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 03 20:04:20.436838 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:20.436815    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 03 20:04:20.449102 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:20.449071    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/51dffd76-5c30-440c-85ed-d81221663833-original-pull-secret\") pod \"global-pull-secret-syncer-j4jzv\" (UID: \"51dffd76-5c30-440c-85ed-d81221663833\") " pod="kube-system/global-pull-secret-syncer-j4jzv"
Nov 03 20:04:20.449222 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:20.449162    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c92a5d0-b34f-46a1-8f8f-a9ae3cb8d5a5-metrics-certs\") pod \"network-metrics-daemon-c947w\" (UID: \"4c92a5d0-b34f-46a1-8f8f-a9ae3cb8d5a5\") " pod="openshift-multus/network-metrics-daemon-c947w"
Nov 03 20:04:20.516704 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:20.516664    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcxdr\" (UniqueName: \"kubernetes.io/projected/d71eaa2a-9659-4827-9dfc-c1fce02565cb-kube-api-access-zcxdr\") pod \"network-check-target-6dfr5\" (UID: \"d71eaa2a-9659-4827-9dfc-c1fce02565cb\") " pod="openshift-network-diagnostics/network-check-target-6dfr5"
Nov 03 20:04:20.528142 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:20.528119    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 03 20:04:20.548260 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:20.548238    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 03 20:04:20.559630 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:20.559601    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcxdr\" (UniqueName: \"kubernetes.io/projected/d71eaa2a-9659-4827-9dfc-c1fce02565cb-kube-api-access-zcxdr\") pod \"network-check-target-6dfr5\" (UID: \"d71eaa2a-9659-4827-9dfc-c1fce02565cb\") " pod="openshift-network-diagnostics/network-check-target-6dfr5"
Nov 03 20:04:20.643389 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:20.643359    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-89f7f\""
Nov 03 20:04:20.649455 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:20.649439    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-q2t72\""
Nov 03 20:04:20.652598 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:20.652583    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c947w"
Nov 03 20:04:20.657013 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:20.656955    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-n7xhm\""
Nov 03 20:04:20.659056 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:20.659036    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j4jzv"
Nov 03 20:04:20.666098 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:20.666032    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6dfr5"
Nov 03 20:04:20.952433 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:20.952402    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-c947w"]
Nov 03 20:04:20.960258 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:04:20.960226    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c92a5d0_b34f_46a1_8f8f_a9ae3cb8d5a5.slice/crio-61df4200b766b2af80af533f595a1e8304287c36ff3f42949b1bce70bb3f12e5 WatchSource:0}: Error finding container 61df4200b766b2af80af533f595a1e8304287c36ff3f42949b1bce70bb3f12e5: Status 404 returned error can't find the container with id 61df4200b766b2af80af533f595a1e8304287c36ff3f42949b1bce70bb3f12e5
Nov 03 20:04:20.965349 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:20.965324    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-j4jzv"]
Nov 03 20:04:20.969045 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:04:20.969019    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51dffd76_5c30_440c_85ed_d81221663833.slice/crio-39b3b6fef3084099c1fad9f5b7009f36f870612c1f58ece11a394def4ed6264f WatchSource:0}: Error finding container 39b3b6fef3084099c1fad9f5b7009f36f870612c1f58ece11a394def4ed6264f: Status 404 returned error can't find the container with id 39b3b6fef3084099c1fad9f5b7009f36f870612c1f58ece11a394def4ed6264f
Nov 03 20:04:20.979007 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:20.978981    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-6dfr5"]
Nov 03 20:04:20.982638 ip-10-0-132-4 kubenswrapper[2577]: W1103 20:04:20.982611    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd71eaa2a_9659_4827_9dfc_c1fce02565cb.slice/crio-c93ea4ea8fbd9cc0e9b74d6c46c7ae00477df057982cf51ccc955b5da5867803 WatchSource:0}: Error finding container c93ea4ea8fbd9cc0e9b74d6c46c7ae00477df057982cf51ccc955b5da5867803: Status 404 returned error can't find the container with id c93ea4ea8fbd9cc0e9b74d6c46c7ae00477df057982cf51ccc955b5da5867803
Nov 03 20:04:21.021827 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:21.021788    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-6dfr5" event={"ID":"d71eaa2a-9659-4827-9dfc-c1fce02565cb","Type":"ContainerStarted","Data":"c93ea4ea8fbd9cc0e9b74d6c46c7ae00477df057982cf51ccc955b5da5867803"}
Nov 03 20:04:21.022929 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:21.022898    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-j4jzv" event={"ID":"51dffd76-5c30-440c-85ed-d81221663833","Type":"ContainerStarted","Data":"39b3b6fef3084099c1fad9f5b7009f36f870612c1f58ece11a394def4ed6264f"}
Nov 03 20:04:21.024015 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:21.023995    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-c947w" event={"ID":"4c92a5d0-b34f-46a1-8f8f-a9ae3cb8d5a5","Type":"ContainerStarted","Data":"61df4200b766b2af80af533f595a1e8304287c36ff3f42949b1bce70bb3f12e5"}
Nov 03 20:04:23.035295 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:23.035252    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-c947w" event={"ID":"4c92a5d0-b34f-46a1-8f8f-a9ae3cb8d5a5","Type":"ContainerStarted","Data":"fb3a54d57ec68a3d1a0f8fabfb6caf31d92d6c0ade85d6f860a651eef4527048"}
Nov 03 20:04:23.036036 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:23.035326    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-c947w" event={"ID":"4c92a5d0-b34f-46a1-8f8f-a9ae3cb8d5a5","Type":"ContainerStarted","Data":"30d65f9025211158000a5c3fb02f1c6fc379d079ca965502f708a9d66979d4cf"}
Nov 03 20:04:23.071269 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:23.071205    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-c947w" podStartSLOduration=67.009701692 podStartE2EDuration="1m8.071185453s" totalImagesPullingTime="1.061483761s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:03:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:04:23.068463198 +0000 UTC m=+68.928575737" watchObservedRunningTime="2026-11-03 20:04:23.071185453 +0000 UTC m=+68.931297993"
Nov 03 20:04:25.046114 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:25.046070    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-6dfr5" event={"ID":"d71eaa2a-9659-4827-9dfc-c1fce02565cb","Type":"ContainerStarted","Data":"7e28cf903ccac41f50cf937af696d45cb2ede12e5e3e95256dcc07c3c2367866"}
Nov 03 20:04:25.046546 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:25.046230    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-6dfr5"
Nov 03 20:04:25.078791 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:25.078728    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-6dfr5" podStartSLOduration=66.94127676 podStartE2EDuration="1m10.078715597s" totalImagesPullingTime="3.137438837s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:03:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:04:25.07818803 +0000 UTC m=+70.938300565" watchObservedRunningTime="2026-11-03 20:04:25.078715597 +0000 UTC m=+70.938828133"
Nov 03 20:04:25.101651 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:25.101610    2577 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-654b7fcdc8-l6l9f"
Nov 03 20:04:25.101818 ip-10-0-132-4 kubenswrapper[2577]: I1103 20:04:25.101682    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-654b7fcdc8-l6l9f"
Nov 03 20:04:25.788950 ip-10-0-132-4 systemd[1]: Stopping Kubernetes Kubelet...
Nov 03 20:04:25.798459 ip-10-0-132-4 systemd[1]: kubelet.service: Deactivated successfully.
Nov 03 20:04:25.798662 ip-10-0-132-4 systemd[1]: Stopped Kubernetes Kubelet.
Nov 03 20:04:25.798867 ip-10-0-132-4 systemd[1]: kubelet.service: Consumed 4.288s CPU time.
Nov 03 20:04:25.813688 ip-10-0-132-4 systemd[1]: Starting Kubernetes Kubelet...
Nov 03 20:04:25.878955 ip-10-0-132-4 kubenswrapper[5570]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 20:04:25.878955 ip-10-0-132-4 kubenswrapper[5570]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 03 20:04:25.878955 ip-10-0-132-4 kubenswrapper[5570]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 20:04:25.878955 ip-10-0-132-4 kubenswrapper[5570]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 20:04:25.881241 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.881153    5570 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 20:04:25.881241 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.881240    5570 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 20:04:25.881241 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.881244    5570 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 20:04:25.881350 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.881247    5570 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 20:04:25.881350 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.881251    5570 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 20:04:25.881350 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.881254    5570 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 20:04:25.881350 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.881256    5570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 20:04:25.881350 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.881259    5570 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 20:04:25.881350 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.881262    5570 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 20:04:25.881350 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.881265    5570 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 20:04:25.881350 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.881268    5570 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 20:04:25.881350 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.881270    5570 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 20:04:25.881350 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.881273    5570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 20:04:25.881350 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.881276    5570 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 20:04:25.881350 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.881279    5570 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 20:04:25.881350 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.881282    5570 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 20:04:25.881350 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.881285    5570 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 20:04:25.881350 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.881288    5570 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 20:04:25.881350 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.881290    5570 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 20:04:25.881350 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.881293    5570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 20:04:25.881350 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.881296    5570 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 20:04:25.881350 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.881298    5570 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 20:04:25.881856 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.881301    5570 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 20:04:25.881856 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.881303    5570 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 20:04:25.881856 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.881306    5570 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 20:04:25.881856 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.881308    5570 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 20:04:25.881856 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.881311    5570 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 20:04:25.881856 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.881314    5570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 20:04:25.881856 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.881321    5570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 20:04:25.881856 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.881324    5570 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 20:04:25.881856 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.881326    5570 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 20:04:25.881856 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.881329    5570 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 20:04:25.881856 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.881331    5570 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 20:04:25.881856 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.881334    5570 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 20:04:25.881856 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.881338    5570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 20:04:25.881856 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.881341    5570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 20:04:25.881856 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.881343    5570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 20:04:25.881856 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.881347    5570 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 20:04:25.881856 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.881349    5570 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 20:04:25.881856 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.881352    5570 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 20:04:25.881856 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.881355    5570 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 20:04:25.881856 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.881358    5570 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 20:04:25.882324 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.881361    5570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 20:04:25.882324 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.881363    5570 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 20:04:25.882324 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.881366    5570 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 20:04:25.882324 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.881369    5570 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 20:04:25.882324 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.881371    5570 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 20:04:25.882324 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.881374    5570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 20:04:25.882324 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.881376    5570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 20:04:25.882324 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.881379    5570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 20:04:25.882324 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.881381    5570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 20:04:25.882324 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.881384    5570 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 20:04:25.882324 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.881387    5570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 20:04:25.882324 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.881389    5570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 20:04:25.882324 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.881392    5570 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 20:04:25.882324 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.881395    5570 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 20:04:25.882324 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.881398    5570 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 20:04:25.882324 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.881400    5570 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 20:04:25.882324 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.881403    5570 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 20:04:25.882324 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.881405    5570 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 20:04:25.882324 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.881408    5570 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 20:04:25.882324 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.881412    5570 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 20:04:25.882803 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.881414    5570 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 20:04:25.882803 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.881417    5570 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 20:04:25.882803 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.881420    5570 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 20:04:25.882803 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.881422    5570 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 20:04:25.882803 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.881425    5570 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 20:04:25.882803 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.881428    5570 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 20:04:25.882803 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.881431    5570 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 20:04:25.882803 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.881433    5570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 20:04:25.882803 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.881435    5570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 20:04:25.882803 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.881443    5570 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 20:04:25.882803 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.881446    5570 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 20:04:25.882803 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.881450    5570 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 20:04:25.882803 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.881453    5570 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 20:04:25.882803 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.881457    5570 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 20:04:25.882803 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.881460    5570 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 20:04:25.882803 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.881463    5570 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 20:04:25.882803 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.881465    5570 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 20:04:25.882803 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.881468    5570 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 20:04:25.882803 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.881471    5570 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 20:04:25.882803 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.881473    5570 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 20:04:25.883269 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.881476    5570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 20:04:25.883269 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.881479    5570 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 20:04:25.883269 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.881482    5570 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 20:04:25.883269 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.881484    5570 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 20:04:25.883269 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.881487    5570 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 20:04:25.883269 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.881490    5570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 20:04:25.883269 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.881493    5570 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 20:04:25.883269 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.881495    5570 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 20:04:25.883269 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.881498    5570 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 20:04:25.883269 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.881501    5570 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 20:04:25.883269 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.881503    5570 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 20:04:25.883269 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.881506    5570 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 20:04:25.883269 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.881511    5570 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 20:04:25.883269 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.881515    5570 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 20:04:25.883269 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.881519    5570 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 20:04:25.883269 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.881522    5570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 20:04:25.883269 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.881525    5570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 20:04:25.883269 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.881527    5570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 20:04:25.883269 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.881530    5570 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 20:04:25.883746 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.881532    5570 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 20:04:25.883746 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.881535    5570 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 20:04:25.883746 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.881539    5570 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 20:04:25.883746 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.881542    5570 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 20:04:25.883746 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.881545    5570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 20:04:25.883746 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.882330    5570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 20:04:25.883746 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.882338    5570 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 20:04:25.883746 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.882341    5570 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 20:04:25.883746 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.882344    5570 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 20:04:25.883746 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.882347    5570 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 20:04:25.883746 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.882349    5570 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 20:04:25.883746 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.882352    5570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 20:04:25.883746 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.882355    5570 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 20:04:25.883746 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.882357    5570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 20:04:25.883746 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.882360    5570 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 20:04:25.883746 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.882363    5570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 20:04:25.883746 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.882365    5570 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 20:04:25.883746 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.882368    5570 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 20:04:25.883746 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.882370    5570 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 20:04:25.884227 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.882373    5570 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 20:04:25.884227 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.882376    5570 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 20:04:25.884227 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.882378    5570 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 20:04:25.884227 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.882381    5570 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 20:04:25.884227 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.882383    5570 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 20:04:25.884227 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.882386    5570 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 20:04:25.884227 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.882390    5570 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 20:04:25.884227 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.882393    5570 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 20:04:25.884227 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.882396    5570 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 20:04:25.884227 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.882398    5570 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 20:04:25.884227 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.882401    5570 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 20:04:25.884227 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.882404    5570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 20:04:25.884227 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.882406    5570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 20:04:25.884227 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.882409    5570 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 20:04:25.884227 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.882411    5570 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 20:04:25.884227 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.882415    5570 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 20:04:25.884227 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.882417    5570 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 20:04:25.884227 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.882420    5570 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 20:04:25.884227 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.882424    5570 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 20:04:25.884686 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.882428    5570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 20:04:25.884686 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.882431    5570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 20:04:25.884686 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.882435    5570 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 20:04:25.884686 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.882437    5570 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 20:04:25.884686 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.882440    5570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 20:04:25.884686 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.882443    5570 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 20:04:25.884686 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.882446    5570 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 20:04:25.884686 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.882448    5570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 20:04:25.884686 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.882451    5570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 20:04:25.884686 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.882453    5570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 20:04:25.884686 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.882456    5570 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 20:04:25.884686 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.882458    5570 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 20:04:25.884686 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.882461    5570 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 20:04:25.884686 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.882463    5570 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 20:04:25.884686 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.882466    5570 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 20:04:25.884686 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.882468    5570 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 20:04:25.884686 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.882474    5570 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 20:04:25.884686 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.882477    5570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 20:04:25.884686 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.882480    5570 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 20:04:25.885151 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.882483    5570 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 20:04:25.885151 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.882486    5570 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 20:04:25.885151 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.882494    5570 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 20:04:25.885151 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.882498    5570 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 20:04:25.885151 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.882500    5570 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 20:04:25.885151 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.882503    5570 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 20:04:25.885151 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.882505    5570 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 20:04:25.885151 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.882508    5570 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 20:04:25.885151 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.882511    5570 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 20:04:25.885151 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.882514    5570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 20:04:25.885151 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.882516    5570 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 20:04:25.885151 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.882519    5570 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 20:04:25.885151 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.882521    5570 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 20:04:25.885151 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.882524    5570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 20:04:25.885151 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.882527    5570 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 20:04:25.885151 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.882529    5570 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 20:04:25.885151 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.882532    5570 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 20:04:25.885151 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.882534    5570 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 20:04:25.885151 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.882537    5570 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 20:04:25.885151 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.882539    5570 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 20:04:25.885638 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.882542    5570 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 20:04:25.885638 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.882544    5570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 20:04:25.885638 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.882547    5570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 20:04:25.885638 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.882549    5570 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 20:04:25.885638 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.882552    5570 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 20:04:25.885638 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.882554    5570 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 20:04:25.885638 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.882557    5570 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 20:04:25.885638 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.882559    5570 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 20:04:25.885638 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.882562    5570 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 20:04:25.885638 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.882564    5570 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 20:04:25.885638 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.882567    5570 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 20:04:25.885638 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.882570    5570 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 20:04:25.885638 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.882573    5570 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 20:04:25.885638 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.882576    5570 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 20:04:25.885638 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.882579    5570 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 20:04:25.885638 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.882588    5570 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 20:04:25.885638 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.882591    5570 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 20:04:25.885638 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.882594    5570 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 20:04:25.885638 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.882597    5570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 20:04:25.885638 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.882601    5570 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 20:04:25.886151 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.882604    5570 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 20:04:25.886151 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.882607    5570 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 20:04:25.886151 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.882609    5570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 20:04:25.886151 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.882612    5570 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 20:04:25.886151 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.882615    5570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 20:04:25.886151 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.882617    5570 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 20:04:25.886151 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.882620    5570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 20:04:25.886151 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.882622    5570 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 20:04:25.886151 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.882625    5570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 20:04:25.886151 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.882628    5570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 20:04:25.886151 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.882631    5570 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 20:04:25.886151 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.882633    5570 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 20:04:25.886151 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.882635    5570 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 20:04:25.886151 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.882638    5570 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 20:04:25.886151 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.883207    5570 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 20:04:25.886151 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.883213    5570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 20:04:25.886151 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.883216    5570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 20:04:25.886151 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.883218    5570 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 20:04:25.886151 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.883221    5570 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 20:04:25.886151 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.883225    5570 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 20:04:25.886656 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.883227    5570 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 20:04:25.886656 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.883230    5570 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 20:04:25.886656 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.883233    5570 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 20:04:25.886656 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.883236    5570 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 20:04:25.886656 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.883239    5570 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 20:04:25.886656 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.883242    5570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 20:04:25.886656 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.883244    5570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 20:04:25.886656 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.883255    5570 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 20:04:25.886656 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.883258    5570 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 20:04:25.886656 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.883261    5570 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 20:04:25.886656 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.883263    5570 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 20:04:25.886656 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.883266    5570 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 20:04:25.886656 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.883269    5570 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 20:04:25.886656 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.883271    5570 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 20:04:25.886656 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.883274    5570 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 20:04:25.886656 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.883277    5570 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 20:04:25.886656 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.883280    5570 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 20:04:25.886656 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.883282    5570 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 20:04:25.886656 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.883285    5570 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 20:04:25.887125 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.883287    5570 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 20:04:25.887125 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.883290    5570 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 20:04:25.887125 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.883293    5570 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 20:04:25.887125 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.883295    5570 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 20:04:25.887125 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.883298    5570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 20:04:25.887125 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.883300    5570 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 20:04:25.887125 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.883303    5570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 20:04:25.887125 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.883305    5570 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 20:04:25.887125 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.883308    5570 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 20:04:25.887125 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.883311    5570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 20:04:25.887125 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.883313    5570 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 20:04:25.887125 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.883316    5570 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 20:04:25.887125 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.883318    5570 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 20:04:25.887125 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.883321    5570 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 20:04:25.887125 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.883323    5570 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 20:04:25.887125 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.883328    5570 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 20:04:25.887125 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.883331    5570 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 20:04:25.887125 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.883334    5570 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 20:04:25.887125 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.883337    5570 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 20:04:25.887125 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.883339    5570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 20:04:25.887638 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.883342    5570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 20:04:25.887638 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.883350    5570 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 20:04:25.887638 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.883353    5570 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 20:04:25.887638 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.883356    5570 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 20:04:25.887638 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.883358    5570 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 20:04:25.887638 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.883361    5570 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 20:04:25.887638 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.883363    5570 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 20:04:25.887638 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.883366    5570 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 20:04:25.887638 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.883369    5570 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 20:04:25.887638 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.883371    5570 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 20:04:25.887638 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.883374    5570 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 20:04:25.887638 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.883377    5570 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 20:04:25.887638 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.883379    5570 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 20:04:25.887638 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.883382    5570 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 20:04:25.887638 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.883385    5570 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 20:04:25.887638 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.883388    5570 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 20:04:25.887638 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.883392    5570 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 20:04:25.887638 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.883396    5570 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 20:04:25.887638 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.883399    5570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 20:04:25.888159 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.883402    5570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 20:04:25.888159 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.883405    5570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 20:04:25.888159 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.883407    5570 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 20:04:25.888159 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.883410    5570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 20:04:25.888159 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.883413    5570 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 20:04:25.888159 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.883415    5570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 20:04:25.888159 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.883418    5570 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 20:04:25.888159 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.883421    5570 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 20:04:25.888159 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.883427    5570 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 20:04:25.888159 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.883429    5570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 20:04:25.888159 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.883433    5570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 20:04:25.888159 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.883435    5570 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 20:04:25.888159 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.883438    5570 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 20:04:25.888159 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.883440    5570 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 20:04:25.888159 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.883443    5570 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 20:04:25.888159 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.883456    5570 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 20:04:25.888159 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.883459    5570 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 20:04:25.888159 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.883462    5570 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 20:04:25.888159 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.883464    5570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 20:04:25.888159 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.883467    5570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 20:04:25.888651 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.883469    5570 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 20:04:25.888651 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.883472    5570 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 20:04:25.888651 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.883474    5570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 20:04:25.888651 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.883477    5570 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 20:04:25.888651 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.883479    5570 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 20:04:25.888651 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.883482    5570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 20:04:25.888651 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.883484    5570 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 20:04:25.888651 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.883487    5570 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 20:04:25.888651 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.883489    5570 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 20:04:25.888651 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.883491    5570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 20:04:25.888651 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.883494    5570 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 20:04:25.888651 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.883496    5570 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 20:04:25.888651 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.883499    5570 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 20:04:25.888651 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.883501    5570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 20:04:25.888651 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.883504    5570 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 20:04:25.888651 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.883506    5570 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 20:04:25.888651 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.883509    5570 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 20:04:25.888651 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.883511    5570 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 20:04:25.888651 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.883514    5570 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 20:04:25.888651 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.883517    5570 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 20:04:25.889411 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.883521    5570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 20:04:25.889411 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.883524    5570 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 20:04:25.889411 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.883623    5570 flags.go:64] FLAG: --address="0.0.0.0"
Nov 03 20:04:25.889411 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.883634    5570 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 03 20:04:25.889411 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.883646    5570 flags.go:64] FLAG: --anonymous-auth="true"
Nov 03 20:04:25.889411 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.883651    5570 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 03 20:04:25.889411 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.883656    5570 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 03 20:04:25.889411 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.883659    5570 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 03 20:04:25.889411 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.883669    5570 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 03 20:04:25.889411 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.883674    5570 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 03 20:04:25.889411 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.883677    5570 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 03 20:04:25.889411 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.883680    5570 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 03 20:04:25.889411 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.883684    5570 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 03 20:04:25.889411 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.883687    5570 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 03 20:04:25.889411 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.883690    5570 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 03 20:04:25.889411 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.883693    5570 flags.go:64] FLAG: --cgroup-root=""
Nov 03 20:04:25.889411 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.883696    5570 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 03 20:04:25.889411 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.883699    5570 flags.go:64] FLAG: --client-ca-file=""
Nov 03 20:04:25.889411 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.883702    5570 flags.go:64] FLAG: --cloud-provider="external"
Nov 03 20:04:25.889411 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.883705    5570 flags.go:64] FLAG: --cluster-dns="[]"
Nov 03 20:04:25.889411 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.883715    5570 flags.go:64] FLAG: --cluster-domain=""
Nov 03 20:04:25.889411 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.883718    5570 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 03 20:04:25.889411 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.883722    5570 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 03 20:04:25.890076 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.883725    5570 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 03 20:04:25.890076 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.883728    5570 flags.go:64] FLAG: --container-log-max-files="5"
Nov 03 20:04:25.890076 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.883732    5570 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 03 20:04:25.890076 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.883735    5570 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 03 20:04:25.890076 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.883738    5570 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 03 20:04:25.890076 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.883741    5570 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 03 20:04:25.890076 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.883745    5570 flags.go:64] FLAG: --contention-profiling="false"
Nov 03 20:04:25.890076 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.883748    5570 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 03 20:04:25.890076 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.883751    5570 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 03 20:04:25.890076 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.883756    5570 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 03 20:04:25.890076 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.883759    5570 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 03 20:04:25.890076 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.883764    5570 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 03 20:04:25.890076 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.883768    5570 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 03 20:04:25.890076 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.883789    5570 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 03 20:04:25.890076 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.883793    5570 flags.go:64] FLAG: --enable-load-reader="false"
Nov 03 20:04:25.890076 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.883796    5570 flags.go:64] FLAG: --enable-server="true"
Nov 03 20:04:25.890076 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.883799    5570 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 03 20:04:25.890076 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.883806    5570 flags.go:64] FLAG: --event-burst="100"
Nov 03 20:04:25.890076 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.883815    5570 flags.go:64] FLAG: --event-qps="50"
Nov 03 20:04:25.890076 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.883819    5570 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 03 20:04:25.890076 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.883822    5570 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 03 20:04:25.890076 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.883825    5570 flags.go:64] FLAG: --eviction-hard=""
Nov 03 20:04:25.890076 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.883830    5570 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 03 20:04:25.890076 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.883833    5570 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 03 20:04:25.890651 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.883836    5570 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 03 20:04:25.890651 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.883839    5570 flags.go:64] FLAG: --eviction-soft=""
Nov 03 20:04:25.890651 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.883842    5570 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 03 20:04:25.890651 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.883845    5570 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 03 20:04:25.890651 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.883849    5570 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 03 20:04:25.890651 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.883852    5570 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 03 20:04:25.890651 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.883855    5570 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 03 20:04:25.890651 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.883858    5570 flags.go:64] FLAG: --fail-swap-on="true"
Nov 03 20:04:25.890651 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.883860    5570 flags.go:64] FLAG: --feature-gates=""
Nov 03 20:04:25.890651 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.883865    5570 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 03 20:04:25.890651 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.883868    5570 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 03 20:04:25.890651 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.883871    5570 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 03 20:04:25.890651 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.883874    5570 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 03 20:04:25.890651 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.883877    5570 flags.go:64] FLAG: --healthz-port="10248"
Nov 03 20:04:25.890651 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.883881    5570 flags.go:64] FLAG: --help="false"
Nov 03 20:04:25.890651 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.883884    5570 flags.go:64] FLAG: --hostname-override="ip-10-0-132-4.ec2.internal"
Nov 03 20:04:25.890651 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.883887    5570 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 03 20:04:25.890651 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.883892    5570 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 03 20:04:25.890651 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.883896    5570 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 03 20:04:25.890651 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.883899    5570 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 03 20:04:25.890651 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.883903    5570 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 03 20:04:25.890651 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.883906    5570 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 03 20:04:25.890651 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.883909    5570 flags.go:64] FLAG: --image-service-endpoint=""
Nov 03 20:04:25.891319 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.883912    5570 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 03 20:04:25.891319 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.883915    5570 flags.go:64] FLAG: --kube-api-burst="100"
Nov 03 20:04:25.891319 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.883918    5570 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 03 20:04:25.891319 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.883921    5570 flags.go:64] FLAG: --kube-api-qps="50"
Nov 03 20:04:25.891319 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.883930    5570 flags.go:64] FLAG: --kube-reserved=""
Nov 03 20:04:25.891319 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.883934    5570 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 03 20:04:25.891319 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.883937    5570 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 03 20:04:25.891319 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.883940    5570 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 03 20:04:25.891319 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.883943    5570 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 03 20:04:25.891319 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.883946    5570 flags.go:64] FLAG: --lock-file=""
Nov 03 20:04:25.891319 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.883948    5570 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 03 20:04:25.891319 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.883951    5570 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 03 20:04:25.891319 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.883954    5570 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 03 20:04:25.891319 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.883960    5570 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 03 20:04:25.891319 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.883963    5570 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 03 20:04:25.891319 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.883966    5570 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 03 20:04:25.891319 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.883968    5570 flags.go:64] FLAG: --logging-format="text"
Nov 03 20:04:25.891319 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.883971    5570 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 03 20:04:25.891319 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.883975    5570 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 03 20:04:25.891319 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.883978    5570 flags.go:64] FLAG: --manifest-url=""
Nov 03 20:04:25.891319 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.883981    5570 flags.go:64] FLAG: --manifest-url-header=""
Nov 03 20:04:25.891319 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.883985    5570 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 03 20:04:25.891319 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.883989    5570 flags.go:64] FLAG: --max-open-files="1000000"
Nov 03 20:04:25.891319 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.883992    5570 flags.go:64] FLAG: --max-pods="110"
Nov 03 20:04:25.891319 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.883995    5570 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 03 20:04:25.891975 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.883998    5570 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 03 20:04:25.891975 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.884004    5570 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 03 20:04:25.891975 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.884007    5570 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 03 20:04:25.891975 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.884009    5570 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 03 20:04:25.891975 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.884012    5570 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 03 20:04:25.891975 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.884015    5570 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 03 20:04:25.891975 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.884022    5570 flags.go:64] FLAG: --node-status-max-images="50"
Nov 03 20:04:25.891975 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.884026    5570 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 03 20:04:25.891975 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.884029    5570 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 03 20:04:25.891975 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.884032    5570 flags.go:64] FLAG: --pod-cidr=""
Nov 03 20:04:25.891975 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.884035    5570 flags.go:64] FLAG: --pod-manifest-path=""
Nov 03 20:04:25.891975 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.884038    5570 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 03 20:04:25.891975 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.884047    5570 flags.go:64] FLAG: --pods-per-core="0"
Nov 03 20:04:25.891975 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.884050    5570 flags.go:64] FLAG: --port="10250"
Nov 03 20:04:25.891975 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.884053    5570 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 03 20:04:25.891975 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.884056    5570 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-098aef875b2b4ec37"
Nov 03 20:04:25.891975 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.884059    5570 flags.go:64] FLAG: --qos-reserved=""
Nov 03 20:04:25.891975 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.884062    5570 flags.go:64] FLAG: --read-only-port="10255"
Nov 03 20:04:25.891975 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.884065    5570 flags.go:64] FLAG: --register-node="true"
Nov 03 20:04:25.891975 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.884068    5570 flags.go:64] FLAG: --register-with-taints=""
Nov 03 20:04:25.891975 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.884072    5570 flags.go:64] FLAG: --registry-burst="10"
Nov 03 20:04:25.891975 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.884075    5570 flags.go:64] FLAG: --registry-qps="5"
Nov 03 20:04:25.891975 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.884078    5570 flags.go:64] FLAG: --reserved-cpus=""
Nov 03 20:04:25.891975 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.884081    5570 flags.go:64] FLAG: --reserved-memory=""
Nov 03 20:04:25.891975 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.884085    5570 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 03 20:04:25.891975 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.884088    5570 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 03 20:04:25.892614 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.884091    5570 flags.go:64] FLAG: --rotate-certificates="false"
Nov 03 20:04:25.892614 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.884094    5570 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 03 20:04:25.892614 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.884097    5570 flags.go:64] FLAG: --runonce="false"
Nov 03 20:04:25.892614 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.884099    5570 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 03 20:04:25.892614 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.884103    5570 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 03 20:04:25.892614 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.884106    5570 flags.go:64] FLAG: --seccomp-default="false"
Nov 03 20:04:25.892614 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.884109    5570 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 03 20:04:25.892614 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.884114    5570 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 03 20:04:25.892614 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.884117    5570 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 03 20:04:25.892614 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.884120    5570 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 03 20:04:25.892614 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.884123    5570 flags.go:64] FLAG: --storage-driver-password="root"
Nov 03 20:04:25.892614 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.884126    5570 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 03 20:04:25.892614 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.884129    5570 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 03 20:04:25.892614 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.884132    5570 flags.go:64] FLAG: --storage-driver-user="root"
Nov 03 20:04:25.892614 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.884135    5570 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 03 20:04:25.892614 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.884138    5570 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 03 20:04:25.892614 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.884141    5570 flags.go:64] FLAG: --system-cgroups=""
Nov 03 20:04:25.892614 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.884144    5570 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 03 20:04:25.892614 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.884149    5570 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 03 20:04:25.892614 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.884159    5570 flags.go:64] FLAG: --tls-cert-file=""
Nov 03 20:04:25.892614 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.884163    5570 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 03 20:04:25.892614 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.884170    5570 flags.go:64] FLAG: --tls-min-version=""
Nov 03 20:04:25.892614 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.884173    5570 flags.go:64] FLAG: --tls-private-key-file=""
Nov 03 20:04:25.892614 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.884176    5570 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 03 20:04:25.892614 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.884178    5570 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 03 20:04:25.893228 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.884181    5570 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 03 20:04:25.893228 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.884184    5570 flags.go:64] FLAG: --v="2"
Nov 03 20:04:25.893228 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.884189    5570 flags.go:64] FLAG: --version="false"
Nov 03 20:04:25.893228 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.884193    5570 flags.go:64] FLAG: --vmodule=""
Nov 03 20:04:25.893228 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.884197    5570 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 03 20:04:25.893228 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.884201    5570 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 03 20:04:25.893228 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884370    5570 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 20:04:25.893228 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884374    5570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 20:04:25.893228 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884377    5570 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 20:04:25.893228 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884380    5570 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 20:04:25.893228 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884383    5570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 20:04:25.893228 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884386    5570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 20:04:25.893228 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884389    5570 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 20:04:25.893228 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884391    5570 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 20:04:25.893228 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884396    5570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 20:04:25.893228 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884399    5570 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 20:04:25.893228 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884401    5570 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 20:04:25.893228 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884404    5570 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 20:04:25.893228 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884407    5570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 20:04:25.893228 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884409    5570 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 20:04:25.893228 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884412    5570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 20:04:25.893758 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884414    5570 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 20:04:25.893758 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884417    5570 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 20:04:25.893758 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884420    5570 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 20:04:25.893758 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884422    5570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 20:04:25.893758 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884425    5570 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 20:04:25.893758 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884427    5570 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 20:04:25.893758 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884431    5570 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 20:04:25.893758 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884434    5570 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 20:04:25.893758 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884436    5570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 20:04:25.893758 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884439    5570 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 20:04:25.893758 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884441    5570 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 20:04:25.893758 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884444    5570 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 20:04:25.893758 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884446    5570 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 20:04:25.893758 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884449    5570 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 20:04:25.893758 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884451    5570 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 20:04:25.893758 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884454    5570 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 20:04:25.893758 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884456    5570 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 20:04:25.893758 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884459    5570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 20:04:25.893758 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884461    5570 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 20:04:25.894227 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884464    5570 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 20:04:25.894227 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884466    5570 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 20:04:25.894227 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884469    5570 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 20:04:25.894227 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884472    5570 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 20:04:25.894227 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884474    5570 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 20:04:25.894227 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884477    5570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 20:04:25.894227 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884481    5570 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 20:04:25.894227 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884483    5570 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 20:04:25.894227 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884486    5570 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 20:04:25.894227 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884488    5570 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 20:04:25.894227 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884491    5570 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 20:04:25.894227 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884494    5570 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 20:04:25.894227 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884496    5570 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 20:04:25.894227 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884499    5570 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 20:04:25.894227 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884501    5570 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 20:04:25.894227 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884504    5570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 20:04:25.894227 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884506    5570 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 20:04:25.894227 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884509    5570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 20:04:25.894227 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884511    5570 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 20:04:25.894227 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884514    5570 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 20:04:25.894702 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884523    5570 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 20:04:25.894702 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884525    5570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 20:04:25.894702 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884528    5570 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 20:04:25.894702 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884530    5570 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 20:04:25.894702 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884533    5570 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 20:04:25.894702 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884537    5570 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 20:04:25.894702 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884541    5570 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 20:04:25.894702 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884544    5570 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 20:04:25.894702 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884547    5570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 20:04:25.894702 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884550    5570 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 20:04:25.894702 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884554    5570 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 20:04:25.894702 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884558    5570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 20:04:25.894702 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884561    5570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 20:04:25.894702 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884565    5570 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 20:04:25.894702 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884568    5570 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 20:04:25.894702 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884570    5570 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 20:04:25.894702 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884573    5570 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 20:04:25.894702 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884576    5570 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 20:04:25.894702 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884580    5570 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 20:04:25.895194 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884582    5570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 20:04:25.895194 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884585    5570 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 20:04:25.895194 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884588    5570 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 20:04:25.895194 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884590    5570 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 20:04:25.895194 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884593    5570 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 20:04:25.895194 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884595    5570 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 20:04:25.895194 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884598    5570 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 20:04:25.895194 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884600    5570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 20:04:25.895194 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884603    5570 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 20:04:25.895194 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884605    5570 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 20:04:25.895194 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884608    5570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 20:04:25.895194 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884610    5570 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 20:04:25.895194 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884613    5570 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 20:04:25.895194 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884615    5570 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 20:04:25.895194 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884623    5570 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 20:04:25.895194 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884626    5570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 20:04:25.895194 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884629    5570 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 20:04:25.895194 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884631    5570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 20:04:25.895194 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884634    5570 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 20:04:25.895194 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884636    5570 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 20:04:25.895757 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884639    5570 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 20:04:25.895757 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884641    5570 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 20:04:25.895757 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884644    5570 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 20:04:25.895757 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884647    5570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 20:04:25.895757 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884649    5570 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 20:04:25.895757 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884652    5570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 20:04:25.895757 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884655    5570 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 20:04:25.895757 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884657    5570 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 20:04:25.895757 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884660    5570 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 20:04:25.895757 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884662    5570 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 20:04:25.895757 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884665    5570 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 20:04:25.895757 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884669    5570 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 20:04:25.895757 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884671    5570 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 20:04:25.895757 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.884697    5570 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 20:04:25.895757 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884735    5570 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 20:04:25.895757 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884740    5570 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 20:04:25.896437 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884743    5570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 20:04:25.896437 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884745    5570 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 20:04:25.896437 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884748    5570 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 20:04:25.896437 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884750    5570 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 20:04:25.896437 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884753    5570 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 20:04:25.896437 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884755    5570 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 20:04:25.896437 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884758    5570 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 20:04:25.896437 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884761    5570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 20:04:25.896437 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884763    5570 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 20:04:25.896437 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884767    5570 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 20:04:25.896437 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884783    5570 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 20:04:25.896437 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884798    5570 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 20:04:25.896437 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884801    5570 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 20:04:25.896437 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884804    5570 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 20:04:25.896437 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884806    5570 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 20:04:25.896437 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884809    5570 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 20:04:25.896437 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884812    5570 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 20:04:25.896437 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884814    5570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 20:04:25.896437 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884817    5570 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 20:04:25.896437 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884819    5570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 20:04:25.896947 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884822    5570 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 20:04:25.896947 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884825    5570 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 20:04:25.896947 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884828    5570 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 20:04:25.896947 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884830    5570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 20:04:25.896947 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884833    5570 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 20:04:25.896947 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884836    5570 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 20:04:25.896947 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884838    5570 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 20:04:25.896947 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884843    5570 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 20:04:25.896947 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884847    5570 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 20:04:25.896947 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884850    5570 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 20:04:25.896947 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884853    5570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 20:04:25.896947 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884855    5570 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 20:04:25.896947 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884859    5570 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 20:04:25.896947 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884863    5570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 20:04:25.896947 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884866    5570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 20:04:25.896947 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884868    5570 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 20:04:25.896947 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884871    5570 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 20:04:25.896947 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884874    5570 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 20:04:25.896947 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884876    5570 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 20:04:25.897415 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884879    5570 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 20:04:25.897415 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884881    5570 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 20:04:25.897415 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884885    5570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 20:04:25.897415 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884888    5570 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 20:04:25.897415 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884891    5570 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 20:04:25.897415 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884899    5570 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 20:04:25.897415 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884902    5570 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 20:04:25.897415 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884905    5570 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 20:04:25.897415 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884907    5570 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 20:04:25.897415 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884910    5570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 20:04:25.897415 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884913    5570 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 20:04:25.897415 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884915    5570 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 20:04:25.897415 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884918    5570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 20:04:25.897415 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884920    5570 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 20:04:25.897415 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884923    5570 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 20:04:25.897415 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884925    5570 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 20:04:25.897415 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884928    5570 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 20:04:25.897415 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884930    5570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 20:04:25.897415 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884933    5570 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 20:04:25.897415 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884936    5570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 20:04:25.898052 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884939    5570 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 20:04:25.898052 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884942    5570 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 20:04:25.898052 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884945    5570 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 20:04:25.898052 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884947    5570 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 20:04:25.898052 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884950    5570 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 20:04:25.898052 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884953    5570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 20:04:25.898052 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884955    5570 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 20:04:25.898052 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884958    5570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 20:04:25.898052 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884961    5570 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 20:04:25.898052 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884964    5570 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 20:04:25.898052 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884966    5570 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 20:04:25.898052 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884969    5570 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 20:04:25.898052 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884971    5570 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 20:04:25.898052 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884974    5570 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 20:04:25.898052 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884977    5570 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 20:04:25.898052 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884980    5570 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 20:04:25.898052 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884982    5570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 20:04:25.898052 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884985    5570 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 20:04:25.898052 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884995    5570 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 20:04:25.898052 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.884998    5570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 20:04:25.898717 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.885001    5570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 20:04:25.898717 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.885003    5570 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 20:04:25.898717 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.885006    5570 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 20:04:25.898717 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.885008    5570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 20:04:25.898717 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.885011    5570 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 20:04:25.898717 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.885013    5570 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 20:04:25.898717 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.885016    5570 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 20:04:25.898717 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.885019    5570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 20:04:25.898717 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.885021    5570 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 20:04:25.898717 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.885024    5570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 20:04:25.898717 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.885026    5570 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 20:04:25.898717 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.885029    5570 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 20:04:25.898717 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.885033    5570 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 20:04:25.898717 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.885036    5570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 20:04:25.898717 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.885038    5570 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 20:04:25.898717 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.885041    5570 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 20:04:25.898717 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.885044    5570 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 20:04:25.898717 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.885046    5570 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 20:04:25.898717 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.885048    5570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 20:04:25.899278 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.885051    5570 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 20:04:25.899278 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.885053    5570 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 20:04:25.899278 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.885056    5570 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 20:04:25.899278 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.885058    5570 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 20:04:25.899278 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.885061    5570 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 20:04:25.899278 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.885064    5570 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 20:04:25.899278 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.890210    5570 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 03 20:04:25.899278 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.890228    5570 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 03 20:04:25.899278 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.890247    5570 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 03 20:04:25.899278 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.890252    5570 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 03 20:04:25.899278 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890313    5570 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 20:04:25.899278 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890318    5570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 20:04:25.899278 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890322    5570 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 20:04:25.899278 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890325    5570 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 20:04:25.899278 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890327    5570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 20:04:25.899278 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890332    5570 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 20:04:25.899278 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890335    5570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 20:04:25.899278 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890339    5570 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 20:04:25.899278 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890342    5570 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 20:04:25.899766 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890346    5570 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 20:04:25.899766 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890348    5570 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 20:04:25.899766 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890351    5570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 20:04:25.899766 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890353    5570 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 20:04:25.899766 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890358    5570 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 20:04:25.899766 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890361    5570 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 20:04:25.899766 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890364    5570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 20:04:25.899766 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890366    5570 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 20:04:25.899766 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890369    5570 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 20:04:25.899766 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890371    5570 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 20:04:25.899766 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890376    5570 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 20:04:25.899766 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890380    5570 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 20:04:25.899766 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890383    5570 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 20:04:25.899766 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890386    5570 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 20:04:25.899766 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890389    5570 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 20:04:25.899766 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890392    5570 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 20:04:25.899766 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890395    5570 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 20:04:25.899766 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890397    5570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 20:04:25.899766 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890401    5570 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 20:04:25.900348 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890404    5570 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 20:04:25.900348 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890406    5570 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 20:04:25.900348 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890409    5570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 20:04:25.900348 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890412    5570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 20:04:25.900348 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890415    5570 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 20:04:25.900348 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890418    5570 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 20:04:25.900348 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890420    5570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 20:04:25.900348 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890423    5570 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 20:04:25.900348 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890425    5570 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 20:04:25.900348 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890428    5570 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 20:04:25.900348 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890431    5570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 20:04:25.900348 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890433    5570 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 20:04:25.900348 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890436    5570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 20:04:25.900348 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890439    5570 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 20:04:25.900348 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890441    5570 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 20:04:25.900348 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890444    5570 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 20:04:25.900348 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890446    5570 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 20:04:25.900348 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890449    5570 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 20:04:25.900348 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890452    5570 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 20:04:25.900348 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890455    5570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 20:04:25.900844 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890458    5570 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 20:04:25.900844 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890461    5570 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 20:04:25.900844 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890463    5570 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 20:04:25.900844 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890466    5570 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 20:04:25.900844 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890469    5570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 20:04:25.900844 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890472    5570 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 20:04:25.900844 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890474    5570 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 20:04:25.900844 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890477    5570 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 20:04:25.900844 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890480    5570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 20:04:25.900844 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890482    5570 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 20:04:25.900844 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890485    5570 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 20:04:25.900844 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890488    5570 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 20:04:25.900844 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890491    5570 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 20:04:25.900844 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890493    5570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 20:04:25.900844 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890495    5570 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 20:04:25.900844 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890499    5570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 20:04:25.900844 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890501    5570 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 20:04:25.900844 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890504    5570 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 20:04:25.900844 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890506    5570 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 20:04:25.901308 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890509    5570 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 20:04:25.901308 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890511    5570 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 20:04:25.901308 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890514    5570 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 20:04:25.901308 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890516    5570 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 20:04:25.901308 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890519    5570 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 20:04:25.901308 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890522    5570 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 20:04:25.901308 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890524    5570 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 20:04:25.901308 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890527    5570 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 20:04:25.901308 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890529    5570 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 20:04:25.901308 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890532    5570 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 20:04:25.901308 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890534    5570 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 20:04:25.901308 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890537    5570 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 20:04:25.901308 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890540    5570 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 20:04:25.901308 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890543    5570 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 20:04:25.901308 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890545    5570 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 20:04:25.901308 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890548    5570 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 20:04:25.901308 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890550    5570 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 20:04:25.901308 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890553    5570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 20:04:25.901308 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890556    5570 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 20:04:25.901797 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890558    5570 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 20:04:25.901797 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890561    5570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 20:04:25.901797 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890563    5570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 20:04:25.901797 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890566    5570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 20:04:25.901797 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890568    5570 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 20:04:25.901797 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890571    5570 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 20:04:25.901797 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890574    5570 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 20:04:25.901797 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890576    5570 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 20:04:25.901797 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890579    5570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 20:04:25.901797 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890582    5570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 20:04:25.901797 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890584    5570 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 20:04:25.901797 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890587    5570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 20:04:25.901797 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890589    5570 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 20:04:25.901797 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890592    5570 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 20:04:25.901797 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890594    5570 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 20:04:25.901797 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890597    5570 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 20:04:25.901797 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890600    5570 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 20:04:25.901797 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890602    5570 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 20:04:25.901797 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890605    5570 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 20:04:25.901797 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890608    5570 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 20:04:25.902279 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.890635    5570 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 20:04:25.902279 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890800    5570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 20:04:25.902279 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890807    5570 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 20:04:25.902279 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890810    5570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 20:04:25.902279 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890813    5570 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 20:04:25.902279 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890816    5570 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 20:04:25.902279 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890819    5570 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 20:04:25.902279 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890822    5570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 20:04:25.902279 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890825    5570 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 20:04:25.902279 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890827    5570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 20:04:25.902279 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890830    5570 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 20:04:25.902279 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890833    5570 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 20:04:25.902279 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890835    5570 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 20:04:25.902279 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890838    5570 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 20:04:25.902279 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890841    5570 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 20:04:25.902279 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890843    5570 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 20:04:25.902657 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890847    5570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 20:04:25.902657 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890849    5570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 20:04:25.902657 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890852    5570 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 20:04:25.902657 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890855    5570 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 20:04:25.902657 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890857    5570 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 20:04:25.902657 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890860    5570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 20:04:25.902657 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890862    5570 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 20:04:25.902657 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890865    5570 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 20:04:25.902657 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890868    5570 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 20:04:25.902657 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890870    5570 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 20:04:25.902657 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890873    5570 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 20:04:25.902657 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890875    5570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 20:04:25.902657 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890878    5570 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 20:04:25.902657 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890881    5570 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 20:04:25.902657 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890884    5570 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 20:04:25.902657 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890887    5570 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 20:04:25.902657 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890889    5570 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 20:04:25.902657 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890892    5570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 20:04:25.902657 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890894    5570 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 20:04:25.902657 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890897    5570 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 20:04:25.903148 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890900    5570 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 20:04:25.903148 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890902    5570 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 20:04:25.903148 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890905    5570 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 20:04:25.903148 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890908    5570 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 20:04:25.903148 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890911    5570 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 20:04:25.903148 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890913    5570 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 20:04:25.903148 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890916    5570 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 20:04:25.903148 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890918    5570 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 20:04:25.903148 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890921    5570 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 20:04:25.903148 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890923    5570 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 20:04:25.903148 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890926    5570 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 20:04:25.903148 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890930    5570 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 20:04:25.903148 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890934    5570 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 20:04:25.903148 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890937    5570 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 20:04:25.903148 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890940    5570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 20:04:25.903148 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890943    5570 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 20:04:25.903148 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890945    5570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 20:04:25.903148 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890948    5570 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 20:04:25.903148 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890950    5570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 20:04:25.903595 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890953    5570 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 20:04:25.903595 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890956    5570 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 20:04:25.903595 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890958    5570 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 20:04:25.903595 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890961    5570 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 20:04:25.903595 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890963    5570 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 20:04:25.903595 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890967    5570 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 20:04:25.903595 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890969    5570 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 20:04:25.903595 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890972    5570 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 20:04:25.903595 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890974    5570 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 20:04:25.903595 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890977    5570 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 20:04:25.903595 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890980    5570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 20:04:25.903595 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890982    5570 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 20:04:25.903595 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890985    5570 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 20:04:25.903595 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890987    5570 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 20:04:25.903595 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890990    5570 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 20:04:25.903595 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890992    5570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 20:04:25.903595 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890995    5570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 20:04:25.903595 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.890998    5570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 20:04:25.903595 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.891000    5570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 20:04:25.903595 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.891003    5570 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 20:04:25.904202 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.891005    5570 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 20:04:25.904202 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.891008    5570 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 20:04:25.904202 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.891011    5570 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 20:04:25.904202 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.891014    5570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 20:04:25.904202 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.891016    5570 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 20:04:25.904202 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.891019    5570 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 20:04:25.904202 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.891021    5570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 20:04:25.904202 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.891024    5570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 20:04:25.904202 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.891026    5570 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 20:04:25.904202 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.891028    5570 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 20:04:25.904202 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.891031    5570 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 20:04:25.904202 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.891034    5570 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 20:04:25.904202 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.891036    5570 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 20:04:25.904202 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.891040    5570 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 20:04:25.904202 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.891042    5570 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 20:04:25.904202 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.891045    5570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 20:04:25.904202 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.891048    5570 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 20:04:25.904202 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.891050    5570 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 20:04:25.904202 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.891053    5570 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 20:04:25.904700 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.891057    5570 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 20:04:25.904700 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.891060    5570 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 20:04:25.904700 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.891063    5570 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 20:04:25.904700 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.891066    5570 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 20:04:25.904700 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.891069    5570 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 20:04:25.904700 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.891072    5570 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 20:04:25.904700 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.891075    5570 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 20:04:25.904700 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.891078    5570 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 20:04:25.904700 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.891081    5570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 20:04:25.904700 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.891084    5570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 20:04:25.904700 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.891086    5570 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 20:04:25.904700 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.891089    5570 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 20:04:25.904700 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.891092    5570 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 20:04:25.904700 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.891116    5570 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 20:04:25.904700 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.891149    5570 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 20:04:25.904700 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.891154    5570 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 20:04:25.905106 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.891157    5570 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 20:04:25.905106 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.891160    5570 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 20:04:25.905106 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.891163    5570 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 20:04:25.905106 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.891166    5570 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 20:04:25.905106 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.891169    5570 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 20:04:25.905106 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.891172    5570 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 20:04:25.905106 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.891174    5570 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 20:04:25.905106 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.891177    5570 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 20:04:25.905106 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.891179    5570 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 20:04:25.905106 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.891183    5570 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 20:04:25.905106 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.891187    5570 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 20:04:25.905106 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.891190    5570 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 20:04:25.905106 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.891193    5570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 20:04:25.905106 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.891196    5570 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 20:04:25.905106 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.891199    5570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 20:04:25.905106 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.891202    5570 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 20:04:25.905106 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.891204    5570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 20:04:25.905106 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.891207    5570 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 20:04:25.905806 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.891210    5570 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 20:04:25.905806 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.891212    5570 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 20:04:25.905806 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.891215    5570 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 20:04:25.905806 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.891218    5570 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 20:04:25.905806 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.891220    5570 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 20:04:25.905806 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.891223    5570 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 20:04:25.905806 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.891225    5570 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 20:04:25.905806 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.891228    5570 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 20:04:25.905806 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.891231    5570 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 20:04:25.905806 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.891233    5570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 20:04:25.905806 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.891236    5570 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 20:04:25.905806 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.891239    5570 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 20:04:25.905806 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.891242    5570 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 20:04:25.905806 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.891245    5570 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 20:04:25.905806 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.891247    5570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 20:04:25.905806 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.891250    5570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 20:04:25.905806 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.891252    5570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 20:04:25.905806 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.891255    5570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 20:04:25.905806 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.891257    5570 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 20:04:25.905806 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.891260    5570 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 20:04:25.906565 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.891262    5570 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 20:04:25.906565 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.891265    5570 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 20:04:25.906565 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.891267    5570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 20:04:25.906565 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.891270    5570 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 20:04:25.906565 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.891272    5570 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 20:04:25.906565 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.891275    5570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 20:04:25.906565 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.891277    5570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 20:04:25.906565 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.891280    5570 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 20:04:25.906565 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.891282    5570 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 20:04:25.906565 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.891285    5570 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 20:04:25.906565 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.891288    5570 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 20:04:25.906565 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.891290    5570 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 20:04:25.906565 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.891293    5570 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 20:04:25.906565 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.891295    5570 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 20:04:25.906565 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.891298    5570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 20:04:25.906565 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.891301    5570 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 20:04:25.906565 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.891303    5570 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 20:04:25.906565 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.891306    5570 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 20:04:25.906565 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.891310    5570 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 20:04:25.907271 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.891314    5570 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 20:04:25.907271 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.891317    5570 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 20:04:25.907271 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.891320    5570 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 20:04:25.907271 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.891323    5570 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 20:04:25.907271 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.891326    5570 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 20:04:25.907271 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.891331    5570 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 20:04:25.907271 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.891334    5570 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 20:04:25.907271 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.891337    5570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 20:04:25.907271 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.891339    5570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 20:04:25.907271 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.891342    5570 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 20:04:25.907271 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.891344    5570 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 20:04:25.907271 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.891347    5570 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 20:04:25.907271 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.891349    5570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 20:04:25.907271 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.891352    5570 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 20:04:25.907271 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.891355    5570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 20:04:25.907271 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.891357    5570 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 20:04:25.907271 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.891360    5570 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 20:04:25.907271 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.891362    5570 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 20:04:25.907271 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.891365    5570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 20:04:25.907271 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.891367    5570 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 20:04:25.907746 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.891370    5570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 20:04:25.907746 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.891372    5570 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 20:04:25.907746 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.891377    5570 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 20:04:25.907746 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.891380    5570 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 20:04:25.907746 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.891383    5570 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 20:04:25.907746 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.891385    5570 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 20:04:25.907746 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.891388    5570 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 20:04:25.907746 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.891391    5570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 20:04:25.907746 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.891393    5570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 20:04:25.907746 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.891396    5570 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 20:04:25.907746 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.891398    5570 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 20:04:25.907746 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.891401    5570 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 20:04:25.907746 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.891404    5570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 20:04:25.907746 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.891406    5570 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 20:04:25.907746 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.891409    5570 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 20:04:25.907746 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.891411    5570 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 20:04:25.907746 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.891414    5570 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 20:04:25.907746 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.891418    5570 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 20:04:25.907746 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.891422    5570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 20:04:25.907746 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.891424    5570 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 20:04:25.908315 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.891427    5570 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 20:04:25.908315 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.891430    5570 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 20:04:25.908315 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.891433    5570 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 20:04:25.908315 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.891435    5570 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 20:04:25.908315 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.891438    5570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 20:04:25.908315 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.891441    5570 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 20:04:25.908315 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:25.891443    5570 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 20:04:25.908315 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.891502    5570 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 03 20:04:25.908315 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.892489    5570 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Nov 03 20:04:25.908315 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.892538    5570 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Nov 03 20:04:25.908315 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.892723    5570 server.go:1014] "Starting client certificate rotation"
Nov 03 20:04:25.908315 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.892873    5570 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 03 20:04:25.908315 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.892924    5570 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 19:58:14 +0000 UTC" deadline="2028-04-11 16:14:22.130768232 +0000 UTC"
Nov 03 20:04:25.908315 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.892972    5570 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12596h9m56.237800307s"
Nov 03 20:04:25.908315 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.893271    5570 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 20:04:25.908315 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.894477    5570 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 20:04:25.908718 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.895882    5570 log.go:25] "Validated CRI v1 runtime API"
Nov 03 20:04:25.908718 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.897720    5570 log.go:25] "Validated CRI v1 image API"
Nov 03 20:04:25.908718 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.898499    5570 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 03 20:04:25.908718 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.900749    5570 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 94a93814-ebbe-4de0-a302-9c3ea42c5c57:/dev/nvme0n1p4 a9c160b5-f97e-4f1d-8e22-2e39f347ad8a:/dev/nvme0n1p3]
Nov 03 20:04:25.908875 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.900785    5570 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0e73128cc29efba8f4b0ef35efead17c585a7fb5ac4db5f96fbfcde7e777d2d5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0e73128cc29efba8f4b0ef35efead17c585a7fb5ac4db5f96fbfcde7e777d2d5/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1a93c727fa866700775189988b37b9b744284f0a816bb9a05732853f97812258/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1a93c727fa866700775189988b37b9b744284f0a816bb9a05732853f97812258/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/212fa88bc72490cb2e7aee7e40dbef30ea7a30c81b02dce4efa80ce99a6a0225/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/212fa88bc72490cb2e7aee7e40dbef30ea7a30c81b02dce4efa80ce99a6a0225/userdata/shm major:0 minor:240 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2368c89e09f8a61dcbe91cdf62b26e6fc92980ef44c8d851cd77c563cae625c6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2368c89e09f8a61dcbe91cdf62b26e6fc92980ef44c8d851cd77c563cae625c6/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/39b3b6fef3084099c1fad9f5b7009f36f870612c1f58ece11a394def4ed6264f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/39b3b6fef3084099c1fad9f5b7009f36f870612c1f58ece11a394def4ed6264f/userdata/shm major:0 minor:419 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/47a7d0ed1c1bb8bcaef8b72693485b6259b9f0511abffeae75ba9fb88dae4ece/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/47a7d0ed1c1bb8bcaef8b72693485b6259b9f0511abffeae75ba9fb88dae4ece/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5f4449d3d1de720c3b851fbffba06f75746a71960acf6a437b0ae3f926e304ca/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5f4449d3d1de720c3b851fbffba06f75746a71960acf6a437b0ae3f926e304ca/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5f9800b7bb6c494e33253ca718f90909053640f14dee0e4169958b54fb015d21/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5f9800b7bb6c494e33253ca718f90909053640f14dee0e4169958b54fb015d21/userdata/shm major:0 minor:251 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/61df4200b766b2af80af533f595a1e8304287c36ff3f42949b1bce70bb3f12e5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/61df4200b766b2af80af533f595a1e8304287c36ff3f42949b1bce70bb3f12e5/userdata/shm major:0 minor:417 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6fba5ae523e88c172564490952767809562c59f6a0322d27757e10868226a8c0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6fba5ae523e88c172564490952767809562c59f6a0322d27757e10868226a8c0/userdata/shm major:0 minor:342 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/74d96638209d41f05199b24b4b3143d5e94c1b40a389dde0dcfbd1ceef3266b7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/74d96638209d41f05199b24b4b3143d5e94c1b40a389dde0dcfbd1ceef3266b7/userdata/shm major:0 minor:239 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7e5ed4938ac5a455602f5935787f510257d22cfd87e815829fbc27ba58c1a45f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7e5ed4938ac5a455602f5935787f510257d22cfd87e815829fbc27ba58c1a45f/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8dd1760f632c5c28135a89fef44e119cf12f4fd22c927b2490d8a4995b99427a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8dd1760f632c5c28135a89fef44e119cf12f4fd22c927b2490d8a4995b99427a/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8ef0ef34d0b75303f53cea650dd4280a76b4f1e5813cd730e128a5fd3b3b726e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8ef0ef34d0b75303f53cea650dd4280a76b4f1e5813cd730e128a5fd3b3b726e/userdata/shm major:0 minor:288 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a7dae436e7bb325f83307dc7270e2a3b50e7675bfe3214d187fdaa4b63e06161/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a7dae436e7bb325f83307dc7270e2a3b50e7675bfe3214d187fdaa4b63e06161/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b7d5a4479f6b4cfc2d84e97abf7442f621815298db4681aa2e43df8ccb9ec0da/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b7d5a4479f6b4cfc2d84e97abf7442f621815298db4681aa2e43df8ccb9ec0da/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/baed098a69878751166eba110018725dc271d8ffc557e40cfc38922072195360/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/baed098a69878751166eba110018725dc271d8ffc557e40cfc38922072195360/userdata/shm major:0 minor:92 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c7cefb72ae55f2db75ebd2c7f2795449854b5f8f786ac9b0aa02eb6fb0ab5b45/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c7cefb72ae55f2db75ebd2c7f2795449854b5f8f786ac9b0aa02eb6fb0ab5b45/userdata/shm major:0 minor:334 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c93ea4ea8fbd9cc0e9b74d6c46c7ae00477df057982cf51ccc955b5da5867803/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c93ea4ea8fbd9cc0e9b74d6c46c7ae00477df057982cf51ccc955b5da5867803/userdata/shm major:0 minor:421 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cbc010db7797f543fd2ceb98399bf1cae3f0916fdf67b47f9a13d1916ae59363/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cbc010db7797f543fd2ceb98399bf1cae3f0916fdf67b47f9a13d1916ae59363/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ee3ebf1d7f01d0a2963b1d18ebd247697e393917f93326d8b167e966ef5c8601/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ee3ebf1d7f01d0a2963b1d18ebd247697e393917f93326d8b167e966ef5c8601/userdata/shm major:0 minor:206 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fdfee0f501f2c3bd7ac1c00b1dfd5135480bcf94c3cc699f4bc1c05502ba7e57/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fdfee0f501f2c3bd7ac1c00b1dfd5135480bcf94c3cc699f4bc1c05502ba7e57/userdata/shm major:0 minor:87 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/15b5070f-d0d3-4ad8-aa1d-93cc7b96003a/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/15b5070f-d0d3-4ad8-aa1d-93cc7b96003a/volumes/kubernetes.io~secret/agent-certs major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1d4ce60c-e058-4c29-99e8-5f4f6eead744/volumes/kubernetes.io~projected/kube-api-access-s92q2:{mountpoint:/var/lib/kubelet/pods/1d4ce60c-e058-4c29-99e8-5f4f6eead744/volumes/kubernetes.io~projected/kube-api-access-s92q2 major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2073a8c8-6f07-4817-b7b3-468b327e2712/volumes/kubernetes.io~projected/kube-api-access-jrgcs:{mountpoint:/var/lib/kubelet/pods/2073a8c8-6f07-4817-b7b3-468b327e2712/volumes/kubernetes.io~projected/kube-api-access-jrgcs major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2fd3c650-2f4c-404e-ab06-c50612af72ce/volumes/kubernetes.io~projected/kube-api-access-56z7v:{mountpoint:/var/lib/kubelet/pods/2fd3c650-2f4c-404e-ab06-c50612af72ce/volumes/kubernetes.io~projected/kube-api-access-56z7v major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2fd3c650-2f4c-404e-ab06-c50612af72ce/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/2fd3c650-2f4c-404e-ab06-c50612af72ce/volumes/kubernetes.io~secret/metrics-tls major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3f681106-32d4-4550-a00d-1f9c52060b57/volumes/kubernetes.io~projected/kube-api-access-df59d:{mountpoint:/var/lib/kubelet/pods/3f681106-32d4-4550-a00d-1f9c52060b57/volumes/kubernetes.io~projected/kube-api-access-df59d major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3f681106-32d4-4550-a00d-1f9c52060b57/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/3f681106-32d4-4550-a00d-1f9c52060b57/volumes/kubernetes.io~secret/cert major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4c92a5d0-b34f-46a1-8f8f-a9ae3cb8d5a5/volumes/kubernetes.io~projected/kube-api-access-6nd2w:{mountpoint:/var/lib/kubelet/pods/4c92a5d0-b34f-46a1-8f8f-a9ae3cb8d5a5/volumes/kubernetes.io~projected/kube-api-access-6nd2w major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4c92a5d0-b34f-46a1-8f8f-a9ae3cb8d5a5/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/4c92a5d0-b34f-46a1-8f8f-a9ae3cb8d5a5/volumes/kubernetes.io~secret/metrics-certs major:0 minor:410 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/51dffd76-5c30-440c-85ed-d81221663833/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/51dffd76-5c30-440c-85ed-d81221663833/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:415 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/59f6a4d6-afe0-4c96-a54a-d8e6b5c66a38/volumes/kubernetes.io~projected/kube-api-access-w7hn2:{mountpoint:/var/lib/kubelet/pods/59f6a4d6-afe0-4c96-a54a-d8e6b5c66a38/volumes/kubernetes.io~projected/kube-api-access-w7hn2 major:0 minor:333 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/59f6a4d6-afe0-4c96-a54a-d8e6b5c66a38/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/59f6a4d6-afe0-4c96-a54a-d8e6b5c66a38/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:331 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/59f6a4d6-afe0-4c96-a54a-d8e6b5c66a38/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/59f6a4d6-afe0-4c96-a54a-d8e6b5c66a38/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy major:0 minor:329 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/59f6a4d6-afe0-4c96-a54a-d8e6b5c66a38/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics:{mountpoint:/var/lib/kubelet/pods/59f6a4d6-afe0-4c96-a54a-d8e6b5c66a38/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics major:0 minor:328 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/59f6a4d6-afe0-4c96-a54a-d8e6b5c66a38/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules:{mountpoint:/var/lib/kubelet/pods/59f6a4d6-afe0-4c96-a54a-d8e6b5c66a38/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules major:0 minor:324 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/59f6a4d6-afe0-4c96-a54a-d8e6b5c66a38/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/59f6a4d6-afe0-4c96-a54a-d8e6b5c66a38/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web major:0 minor:330 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/59f6a4d6-afe0-4c96-a54a-d8e6b5c66a38/volumes/kubernetes.io~secret/secret-thanos-querier-tls:{mountpoint:/var/lib/kubelet/pods/59f6a4d6-afe0-4c96-a54a-d8e6b5c66a38/volumes/kubernetes.io~secret/secret-thanos-querier-tls major:0 minor:332 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/65008653-586d-4a76-a53f-7efe1f6d839b/volumes/kubernetes.io~projected/kube-api-access-8psbp:{mountpoint:/var/lib/kubelet/pods/65008653-586d-4a76-a53f-7efe1f6d839b/volumes/kubernetes.io~projected/kube-api-access-8psbp major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/682463ee-d9c9-4e9b-82b7-f1116898debf/volumes/kubernetes.io~projected/kube-api-access-nkjpg:{mountpoint:/var/lib/kubelet/pods/682463ee-d9c9-4e9b-82b7-f1116898debf/volumes/kubernetes.io~projected/kube-api-access-nkjpg major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/716cc9cf-d26a-481b-8114-169c6104ce56/volumes/kubernetes.io~projected/kube-api-access-vrdtp:{mountpoint:/var/lib/kubelet/pods/716cc9cf-d26a-481b-8114-169c6104ce56/volumes/kubernetes.io~projected/kube-api-access-vrdtp major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/74d4808e-24ce-4c47-b4ce-e1188eeea6f1/volumes/kubernetes.io~projected/kube-api-access-f547t:{mountpoint:/var/lib/kubelet/pods/74d4808e-24ce-4c47-b4ce-e1188eeea6f1/volumes/kubernetes.io~projected/kube-api-access-f547t major:0 minor:238 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/74d4808e-24ce-4c47-b4ce-e1188eeea6f1/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/74d4808e-24ce-4c47-b4ce-e1188eeea6f1/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:232 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/74d4808e-24ce-4c47-b4ce-e1188eeea6f1/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/74d4808e-24ce-4c47-b4ce-e1188eeea6f1/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:235 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/884cbb80-b3f8-4f90-bd55-328d17943cf3/volumes/kubernetes.io~projected/kube-api-access-szdn8:{mountpoint:/var/lib/kubelet/pods/884cbb80-b3f8-4f90-bd55-328d17943cf3/volumes/kubernetes.io~projected/kube-api-access-szdn8 major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/884cbb80-b3f8-4f90-bd55-328d17943cf3/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/884cbb80-b3f8-4f90-bd55-328d17943cf3/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9126a982-59c9-4cec-a665-5f1bc0ae3389/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/9126a982-59c9-4cec-a665-5f1bc0ae3389/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9126a982-59c9-4cec-a665-5f1bc0ae3389/volumes/kubernetes.io~projected/kube-api-access-89vxf:{mountpoint:/var/lib/kubelet/pods/9126a982-59c9-4cec-a665-5f1bc0ae3389/volumes/kubernetes.io~projected/kube-api-access-89vxf major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9126a982-59c9-4cec-a665-5f1bc0ae3389/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/9126a982-59c9-4cec-a665-5f1bc0ae3389/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cb9bef9b-c7cd-410e-8c1e-1b27474daf81/volumes/kubernetes.io~projected/kube-api-access-4ndt8:{mountpoint:/var/lib/kubelet/pods/cb9bef9b-c7cd-410e-8c1e-1b27474daf81/volumes/kubernetes.io~projected/kube-api-access-4ndt8 major:0 minor:237 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cb9bef9b-c7cd-410e-8c1e-1b27474daf81/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/cb9bef9b-c7cd-410e-8c1e-1b27474daf81/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config major:0 minor:231 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cb9bef9b-c7cd-410e-8c1e-1b27474daf81/volumes/kubernetes.io~secret/openshift-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/cb9bef9b-c7cd-410e-8c1e-1b27474daf81/volumes/kubernetes.io~secret/openshift-state-metrics-tls major:0 minor:233 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d5d14caa-6661-4146-b888-59c3518216d4/volumes/kubernetes.io~projected/kube-api-access-8wq4p:{mountpoint:/var/lib/kubelet/pods/d5d14caa-6661-4146-b888-59c3518216d4/volumes/kubernetes.io~projected/kube-api-access-8wq4p major:0 minor:341 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d5d14caa-6661-4146-b888-59c3518216d4/volumes/kubernetes.io~secret/client-ca-bundle:{mountpoint:/var/lib/kubelet/pods/d5d14caa-6661-4146-b888-59c3518216d4/volumes/kubernetes.io~secret/client-ca-bundle major:0 minor:340 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d5d14caa-6661-4146-b888-59c3518216d4/volumes/kubernetes.io~secret/secret-metrics-server-client-certs:{mountpoint:/var/lib/kubelet/pods/d5d14caa-6661-4146-b888-59c3518216d4/volumes/kubernetes.io~secret/secret-metrics-server-client-certs major:0 minor:339 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d5d14caa-6661-4146-b888-59c3518216d4/volumes/kubernetes.io~secret/secret-metrics-server-tls:{mountpoint:/var/lib/kubelet/pods/d5d14caa-6661-4146-b888-59c3518216d4/volumes/kubernetes.io~secret/secret-metrics-server-tls major:0 minor:338 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d71eaa2a-9659-4827-9dfc-c1fce02565cb/volumes/kubernetes.io~projected/kube-api-access-zcxdr:{mountpoint:/var/lib/kubelet/pods/d71eaa2a-9659-4827-9dfc-c1fce02565cb/volumes/kubernetes.io~projected/kube-api-access-zcxdr major:0 minor:416 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/db895326-506d-479d-a526-6bf71be7f967/volumes/kubernetes.io~projected/kube-api-access-vlzhf:{mountpoint:/var/lib/kubelet/pods/db895326-506d-479d-a526-6bf71be7f967/volumes/kubernetes.io~projected/kube-api-access-vlzhf major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f21cf70e-682f-490c-bda5-3954df150487/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/f21cf70e-682f-490c-bda5-3954df150487/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f21cf70e-682f-490c-bda5-3954df150487/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/f21cf70e-682f-490c-bda5-3954df150487/volumes/kubernetes.io~empty-dir/tmp major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f21cf70e-682f-490c-bda5-3954df150487/volumes/kubernetes.io~projected/kube-api-access-59sqq:{mountpoint:/var/lib/kubelet/pods/f21cf70e-682f-490c-bda5-3954df150487/volumes/kubernetes.io~projected/kube-api-access-59sqq major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f7d30630-43af-4157-b90f-d7eddef25085/volume-subpaths/cluster-tls-config/alertmanager/10:{mountpoint:/var/lib/kubelet/pods/f7d30630-43af-4157-b90f-d7eddef25085/volume-subpaths/cluster-tls-config/alertmanager/10 major:0 minor:282 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f7d30630-43af-4157-b90f-d7eddef25085/volume-subpaths/web-config/alertmanager/9:{mountpoint:/var/lib/kubelet/pods/f7d30630-43af-4157-b90f-d7eddef25085/volume-subpaths/web-config/alertmanager/9 major:0 minor:273 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f7d30630-43af-4157-b90f-d7eddef25085/volume-subpaths/web-config/config-reloader/6:{mountpoint:/var/lib/kubelet/pods/f7d30630-43af-4157-b90f-d7eddef25085/volume-subpaths/web-config/config-reloader/6 major:0 minor:273 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f7d30630-43af-4157-b90f-d7eddef25085/volume-subpaths/web-config/init-config-reloader/6:{mountpoint:/var/lib/kubelet/pods/f7d30630-43af-4157-b90f-d7eddef25085/volume-subpaths/web-config/init-config-reloader/6 major:0 minor:273 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f7d30630-43af-4157-b90f-d7eddef25085/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/f7d30630-43af-4157-b90f-d7eddef25085/volumes/kubernetes.io~empty-dir/config-out major:0 minor:279 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f7d30630-43af-4157-b90f-d7eddef25085/volumes/kubernetes.io~projected/kube-api-access-z67jw:{mountpoint:/var/lib/kubelet/pods/f7d30630-43af-4157-b90f-d7eddef25085/volumes/kubernetes.io~projected/kube-api-access-z67jw major:0 minor:284 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f7d30630-43af-4157-b90f-d7eddef25085/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/f7d30630-43af-4157-b90f-d7eddef25085/volumes/kubernetes.io~projected/tls-assets major:0 minor:281 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f7d30630-43af-4157-b90f-d7eddef25085/volumes/kubernetes.io~secret/cluster-tls-config:{mountpoint:/var/lib/kubelet/pods/f7d30630-43af-4157-b90f-d7eddef25085/volumes/kubernetes.io~secret/cluster-tls-config major:0 minor:282 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f7d30630-43af-4157-b90f-d7eddef25085/volumes/kubernetes.io~secret/config-volume:{mountpoint:/var/lib/kubelet/pods/f7d30630-43af-4157-b90f-d7eddef25085/volumes/kubernetes.io~secret/config-volume major:0 minor:283 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f7d30630-43af-4157-b90f-d7eddef25085/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/f7d30630-43af-4157-b90f-d7eddef25085/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy major:0 minor:277 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f7d30630-43af-4157-b90f-d7eddef25085/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric:{mountpoint:/var/lib/kubelet/pods/f7d30630-43af-4157-b90f-d7eddef25085/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric major:0 minor:278 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f7d30630-43af-4157-b90f-d7eddef25085/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/f7d30630-43af-4157-b90f-d7eddef25085/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web major:0 minor:280 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f7d30630-43af-4157-b90f-d7eddef25085/volumes/kubernetes.io~secret/secret-alertmanager-main-tls:{mountpoint:/var/lib/kubelet/pods/f7d30630-43af-4157-b90f-d7eddef25085/volumes/kubernetes.io~secret/secret-alertmanager-main-tls major:0 minor:285 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f7d30630-43af-4157-b90f-d7eddef25085/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/f7d30630-43af-4157-b90f-d7eddef25085/volumes/kubernetes.io~secret/web-config major:0 minor:273 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ffe074b1-e2a0-403b-9f81-581fdb4d2775/volumes/kubernetes.io~projected/kube-api-access-4g29t:{mountpoint:/var/lib/kubelet/pods/ffe074b1-e2a0-403b-9f81-581fdb4d2775/volumes/kubernetes.io~projected/kube-api-access-4g29t major:0 minor:234 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ffe074b1-e2a0-403b-9f81-581fdb4d2775/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/ffe074b1-e2a0-403b-9f81-581fdb4d2775/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config major:0 minor:230 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ffe074b1-e2a0-403b-9f81-581fdb4d2775/volumes/kubernetes.io~secret/kube-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/ffe074b1-e2a0-403b-9f81-581fdb4d2775/volumes/kubernetes.io~secret/kube-state-metrics-tls major:0 minor:236 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/7a8a3a41bc463bb8d5ccaea668766d192360721ab3dca7b31207d4f0843a9ea2/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/a8378a3853a89f154127a2aa848efa35189426537e255f5fa781c6422caf0af8/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/a579adcac3297742c3df314c8fbd6776f1fcbae3f4195566be6b15dda38401bb/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/8e56efde78d7aad892e39147d305505984d7d7f3a5e3950746ab05244518f52d/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/49cea85e2eabe938428c149c9726ccbc448b1597141dc492d6468eb1d6b47e3d/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/539cf526d538a82f3aa09f8a6405bb3fd64fe39cd65a82bbbb1cf2e3b99e318b/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/bfbb23bf93989475bbf89974ed43b354203926298e3e32fb72bed4d0c7efee25/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/3f07356c9cf25f1d4bee086a2ca9ae355c5d8438c5a9c0e0bbbffd12f90f1173/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-128:{mountpoint:/var/lib/containers/storage/overlay/29ddc3d220eac9c765edbdc906f1e31277c101302fcce277acd25b729e4158e7/merged major:0 minor:128 fsType:overlay blockSize:0} overlay_0-130:{mountpoint:/var/lib/containers/storage/overlay/04491cfe024cb24393f221e9475ed5744b66f2e1a15a5edf25a94467a50104e7/merged major:0 minor:130 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/c8a3a1314efb24441da5edc74c24e59260f73c207aaa8c280ba63d3fb325bf1c/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-145:{mountpoint:/var/lib/containers/storage/overlay/0b967726cae0030b0d00a97dca05131684f28312f5018dbfc85067c9c759caba/merged major:0 minor:145 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/88559d3302bce9e82f455697b0f58e3f35947e29bd9c9ebe40fc42bbeff5560f/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-154:{mountpoint:/var/lib/containers/storage/overlay/b405ca0d9d6f79a84becf33d501ee244a23198cf1b69cacfa5ab3e751fab4bec/merged major:0 minor:154 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/91452f569be9e340c92ce568caba573212ee56f3a16bf08144ec69df459ee1cb/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/cd342116ed56cf0c4608e0aa5b098294b18e6605822236622cfc0f36b4a2250a/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/8b43535c24cf2a16e50100ad05896eb2e97181703eb8161e001e8b1b757726af/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/63617dbe562e396e83bcd79f3ce94185da3342ca428754a0e4298271a99cd325/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/a751e5c26a597e9bf144d4411f3ef4e16bf1e289ae1eadde0bc03f6535b6cf2a/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/a08c584ffe905295ba34a385c5eac0c59e6ceb6cc6efb3ac051df3e6afcb80b0/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/4f11d48789d62d874cee8f21dc54ec4b4f360a05096e474e927b43d4b98e7367/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/bb464cd966b094af6447fc98ea0becefa51015c6ded75de78b6545a8cef0313d/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/fe3a922a259d488e99dd42a7c325649c64105e202764161a602f8df67bf771c0/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/16b94af7b682f709882c304073735c29bc84e587e4e1665626dbb6d276b2f4a9/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/ea6514a364bc0d42ded4cb91cb483b99c6a4a8c06ab9840349de1cbe579d2f94/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-219:{mountpoint:/var/lib/containers/storage/overlay/2e0e225f32a69b66bd5056eb5723babbaf63749b36cb8c30e2e837f6ed80401e/merged major:0 minor:219 fsType:overlay blockSize:0} overlay_0-225:{mountpoint:/var/lib/containers/storage/overlay/18207e5d0d5cc753e06919eb25d0631a9106d10e65e1c3fe40031a94f3c3484f/merged major:0 minor:225 fsType:overlay blockSize:0} overlay_0-247:{mountpoint:/var/lib/containers/storage/overlay/007698cf8c7c8b7ebfc74238d2f749e09d5e6bbb82f1e2675b38ac608f824d66/merged major:0 minor:247 fsType:overlay blockSize:0} overlay_0-249:{mountpoint:/var/lib/containers/storage/overlay/59fd3df0a3811328d2433403d75f3565f372dd5fce1317842228edd65dc4ff89/merged major:0 minor:249 fsType:overlay blockSize:0} overlay_0-253:{mountpoint:/var/lib/containers/storage/overlay/3d883ae2529a272b086bdeb5216b2f8e1f527d7a1383a1580f541c6824f0ed12/merged major:0 minor:253 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/ebdbf8fa9780c7c2a0e8c6a22cfd5e906db06e2e3aa3dd77b136f3e438ca38e0/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/419cbf78c5f2ddb17899828f13e7c323b0cdb4d9f2358124663b3b50f08d140d/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/9e0b080f7b8819c7377acd98e51813205330a1ea9d38a558d49a8ed5696c1770/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-268:{mountpoint:/var/lib/containers/storage/overlay/b735abb186420aaf588ef12292a3c76b911a43cb3f3178810e956be3a95c6c60/merged major:0 minor:268 fsType:overlay blockSize:0} overlay_0-286:{mountpoint:/var/lib/containers/storage/overlay/7555e2bc40a517d6e7fba1f7f21a875a75c7463bd9767d614810918bae27987d/merged major:0 minor:286 fsType:overlay blockSize:0} overlay_0-289:{mountpoint:/var/lib/containers/storage/overlay/d1b7e3ea0ac0779aaa25b55aca949389d35015cd925e295e9ca2bb66106ee978/merged major:0 minor:289 fsType:overlay blockSize:0} overlay_0-295:{mountpoint:/var/lib/containers/storage/overlay/6ea083d81684e8b0fc8fbd57f9b914ee8ec390b933e219a24aef9a40fabf7c11/merged major:0 minor:295 fsType:overlay blockSize:0} overlay_0-301:{mountpoint:/var/lib/containers/storage/overlay/bcef02ddeb00931e73539f8ba4be1312adfc9ae499805be6176875ff4a892ccd/merged major:0 minor:301 fsType:overlay blockSize:0} overlay_0-306:{mountpoint:/var/lib/containers/storage/overlay/2470257cbaafcb4d2a400ba80a7c9c58ee4d15384948894e203808df70aa6b28/merged major:0 minor:306 fsType:overlay blockSize:0} overlay_0-311:{mountpoint:/var/lib/containers/storage/overlay/1cd0cdc9b12faee3bfaee2ed09644fe906658ab084201dc3974adb97f6eca0ff/merged major:0 minor:311 fsType:overlay blockSize:0} overlay_0-313:{mountpoint:/var/lib/containers/storage/overlay/396f5facde4769115fbef5fcd1db7aadaf93310427dacb5e4ccb4402c7f25f8a/merged major:0 minor:313 fsType:overlay blockSize:0} overlay_0-319:{mountpoint:/var/lib/containers/storage/overlay/eb593f6ec3a670746ae57c65f17bf206b93c3fc03a0699d81efed43740dd3a74/merged major:0 minor:319 fsType:overlay blockSize:0} overlay_0-336:{mountpoint:/var/lib/containers/storage/overlay/bb0cb538e0f26966278b72464007c87ca948fd1a2b99c78090364c73427d2373/merged major:0 minor:336 fsType:overlay blockSize:0} overlay_0-344:{mountpoint:/var/lib/containers/storage/overlay/48245b06e4266b2a4985aac89874fc30b3df6e310288f47d06155988a76d22b8/merged major:0 minor:344 fsType:overlay blockSize:0} overlay_0-346:{mountpoint:/var/lib/containers/storage/overlay/47a5b57fce64d656d87951928e22dc3eae69f439b0bfb8b79c108df2312fd85b/merged major:0 minor:346 fsType:overlay blockSize:0} overlay_0-352:{mountpoint:/var/lib/containers/storage/overlay/3840fe9d16b216817f4ea2cff215229be94b104c456200e1b8d6dc98936d2b7b/merged major:0 minor:352 fsType:overlay blockSize:0} overlay_0-354:{mountpoint:/var/lib/containers/storage/overlay/1442ddd965f252ed5d5e9cbcafa8cc9f91283bffc5c21d6f97eefb6ccce37548/merged major:0 minor:354 fsType:overlay blockSize:0} overlay_0-359:{mountpoint:/var/lib/containers/storage/overlay/71af4e37d15c2ff7be537639fc3b162b265ee77a1da264d53fb758007540e3c4/merged major:0 minor:359 fsType:overlay blockSize:0} overlay_0-364:{mountpoint:/var/lib/containers/storage/overlay/6b1817354962664e0709e097f60195849aab23a5128df6445da3d3c452615761/merged major:0 minor:364 fsType:overlay blockSize:0} overlay_0-366:{mountpoint:/var/lib/containers/storage/overlay/d4afbcc2d00e3e42d5d1e5a3c80dcca4b6d6b8188d8a0cda83d0eb06d2821bac/merged major:0 minor:366 fsType:overlay blockSize:0} overlay_0-375:{mountpoint:/var/lib/containers/storage/overlay/d0bc64cd7a7e6c7e3a4ac1c575f539b392f01ab050c7b110b43c2d2220fdc36c/merged major:0 minor:375 fsType:overlay blockSize:0} overlay_0-377:{mountpoint:/var/lib/containers/storage/overlay/48cd641502d21788cd95679512845f98f2a0ce9a30bba218abb17fc4589cdad7/merged major:0 minor:377 fsType:overlay blockSize:0} overlay_0-385:{mountpoint:/var/lib/containers/storage/overlay/5b1fc0cfbfddd2c81561d60ad65d6c74151f7f6a2e1571812c25a6b28fa3cb10/merged major:0 minor:385 fsType:overlay blockSize:0} overlay_0-390:{mountpoint:/var/lib/containers/storage/overlay/988290ce31ea1a78a129ba8ca30baec06fea8da09c2a951e03c9aba4769713d6/merged major:0 minor:390 fsType:overlay blockSize:0} overlay_0-395:{mountpoint:/var/lib/containers/storage/overlay/8b5e984ae936ec7201df453ea4be2f2a6d413663651a3667dee918e874aff1ea/merged major:0 minor:395 fsType:overlay blockSize:0} overlay_0-400:{mountpoint:/var/lib/containers/storage/overlay/bd03b1abee75fe05c9479e44936b2289eea2202a119e7648b3e19a21d38f4251/merged major:0 minor:400 fsType:overlay blockSize:0} overlay_0-405:{mountpoint:/var/lib/containers/storage/overlay/a4bf396ba5dd7677bc621525b0142a6c29c7a1c65e9f4f390ffc4ee299be4206/merged major:0 minor:405 fsType:overlay blockSize:0} overlay_0-423:{mountpoint:/var/lib/containers/storage/overlay/3cc23a893ff6832f7a5da9e1fc6a5e9f15660cd067bc7ac76b971bd8a742c773/merged major:0 minor:423 fsType:overlay blockSize:0} overlay_0-425:{mountpoint:/var/lib/containers/storage/overlay/76787dc0544b61a2ff7829b8676db4e8b0e4f727337c74d928e9128e8ed48e1b/merged major:0 minor:425 fsType:overlay blockSize:0} overlay_0-427:{mountpoint:/var/lib/containers/storage/overlay/652f62636abac0641f6a1bd8af412b4388f5802b4319aa949426ff393a9ce671/merged major:0 minor:427 fsType:overlay blockSize:0} overlay_0-429:{mountpoint:/var/lib/containers/storage/overlay/fe1c9dbb19d2eaa8d1afc7431c1d8487a40c6be4dd6e515613b589a5e15428f8/merged major:0 minor:429 fsType:overlay blockSize:0} overlay_0-431:{mountpoint:/var/lib/containers/storage/overlay/8961b8c256c56c477ed4699114bab984efa6591de2d54e13f91290fb04c6241d/merged major:0 minor:431 fsType:overlay blockSize:0} overlay_0-437:{mountpoint:/var/lib/containers/storage/overlay/cf2cfdf86161c01052531d0a5407112440a38e748f5dbc0f48c3daa711f0c758/merged major:0 minor:437 fsType:overlay blockSize:0} overlay_0-442:{mountpoint:/var/lib/containers/storage/overlay/b21b09441dbfbe8232482e64bd7081dc9562778d26e8182af72b0b2143f77c8e/merged major:0 minor:442 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/777fe6c372a72e902472d8bd6afd1cf5eefc111acee94a94846f2d5a8a51fa62/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/a6c6d5d26ae3ef7fb29af4cb5038f1bd06d3545a8eb2fa3d18f2365ddd896a27/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/6ad29bdc9c369d305aed752e2e79dd7d9fae322614dbba48ab84d2ecc014e12e/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/6c1fb15269b91b87d1495941152473e47ba57a1c043205d564b2965986fc53ac/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/cbb940201791385110db5cc600f8cff80734b0a32e9d31f8180d6ffe193c6bc6/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/c50df3d709cac00f2fa5c6677bded0e7ae4c5d005b81aac1e97018bcbfd0572f/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/1ad2486c3d733ad39ee372b1ce991243bd53e55c44076a35892f6d8e41da3dc1/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/79fffd78a7f4670394b583c6750a7535d543b611d35bad4e24a723dc16cb8153/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-94:{mountpoint:/var/lib/containers/storage/overlay/6e39943b27f155f8f7b417b2d077cda6630ebd2f1f2ee0ae8f925c83efa453bb/merged major:0 minor:94 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/34d761095d7df4e09560824b1347490c1016c66d0cff6378ce607b99c8f5eb2a/merged major:0 minor:96 fsType:overlay blockSize:0}]
Nov 03 20:04:25.909718 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.909316    5570 manager.go:217] Machine: {Timestamp:2026-11-03 20:04:25.908811878 +0000 UTC m=+0.061473756 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3118456 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec28a54264ec73b825d5e6ca81be3d07 SystemUUID:ec28a542-64ec-73b8-25d5-e6ca81be3d07 BootID:f47228ac-c33d-4e1c-a9ea-7ab885c36768 Filesystems:[{Device:/var/lib/kubelet/pods/716cc9cf-d26a-481b-8114-169c6104ce56/volumes/kubernetes.io~projected/kube-api-access-vrdtp DeviceMajor:0 DeviceMinor:61 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/2073a8c8-6f07-4817-b7b3-468b327e2712/volumes/kubernetes.io~projected/kube-api-access-jrgcs DeviceMajor:0 DeviceMinor:64 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-225 DeviceMajor:0 DeviceMinor:225 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c93ea4ea8fbd9cc0e9b74d6c46c7ae00477df057982cf51ccc955b5da5867803/userdata/shm DeviceMajor:0 DeviceMinor:421 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-425 DeviceMajor:0 DeviceMinor:425 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ffe074b1-e2a0-403b-9f81-581fdb4d2775/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:230 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ffe074b1-e2a0-403b-9f81-581fdb4d2775/volumes/kubernetes.io~projected/kube-api-access-4g29t DeviceMajor:0 DeviceMinor:234 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ffe074b1-e2a0-403b-9f81-581fdb4d2775/volumes/kubernetes.io~secret/kube-state-metrics-tls DeviceMajor:0 DeviceMinor:236 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5f9800b7bb6c494e33253ca718f90909053640f14dee0e4169958b54fb015d21/userdata/shm DeviceMajor:0 DeviceMinor:251 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8ef0ef34d0b75303f53cea650dd4280a76b4f1e5813cd730e128a5fd3b3b726e/userdata/shm DeviceMajor:0 DeviceMinor:288 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-336 DeviceMajor:0 DeviceMinor:336 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f7d30630-43af-4157-b90f-d7eddef25085/volume-subpaths/web-config/alertmanager/9 DeviceMajor:0 DeviceMinor:273 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cbc010db7797f543fd2ceb98399bf1cae3f0916fdf67b47f9a13d1916ae59363/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/baed098a69878751166eba110018725dc271d8ffc557e40cfc38922072195360/userdata/shm DeviceMajor:0 DeviceMinor:92 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-319 DeviceMajor:0 DeviceMinor:319 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c7cefb72ae55f2db75ebd2c7f2795449854b5f8f786ac9b0aa02eb6fb0ab5b45/userdata/shm DeviceMajor:0 DeviceMinor:334 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-359 DeviceMajor:0 DeviceMinor:359 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1d4ce60c-e058-4c29-99e8-5f4f6eead744/volumes/kubernetes.io~projected/kube-api-access-s92q2 DeviceMajor:0 DeviceMinor:63 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d5d14caa-6661-4146-b888-59c3518216d4/volumes/kubernetes.io~projected/kube-api-access-8wq4p DeviceMajor:0 DeviceMinor:341 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4c92a5d0-b34f-46a1-8f8f-a9ae3cb8d5a5/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:410 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-301 DeviceMajor:0 DeviceMinor:301 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/59f6a4d6-afe0-4c96-a54a-d8e6b5c66a38/volumes/kubernetes.io~secret/secret-thanos-querier-tls DeviceMajor:0 DeviceMinor:332 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/59f6a4d6-afe0-4c96-a54a-d8e6b5c66a38/volumes/kubernetes.io~projected/kube-api-access-w7hn2 DeviceMajor:0 DeviceMinor:333 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f7d30630-43af-4157-b90f-d7eddef25085/volume-subpaths/web-config/init-config-reloader/6 DeviceMajor:0 DeviceMinor:273 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-364 DeviceMajor:0 DeviceMinor:364 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-268 DeviceMajor:0 DeviceMinor:268 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-405 DeviceMajor:0 DeviceMinor:405 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d71eaa2a-9659-4827-9dfc-c1fce02565cb/volumes/kubernetes.io~projected/kube-api-access-zcxdr DeviceMajor:0 DeviceMinor:416 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-429 DeviceMajor:0 DeviceMinor:429 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/74d4808e-24ce-4c47-b4ce-e1188eeea6f1/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:232 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-375 DeviceMajor:0 DeviceMinor:375 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-423 DeviceMajor:0 DeviceMinor:423 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/d5d14caa-6661-4146-b888-59c3518216d4/volumes/kubernetes.io~secret/client-ca-bundle DeviceMajor:0 DeviceMinor:340 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b7d5a4479f6b4cfc2d84e97abf7442f621815298db4681aa2e43df8ccb9ec0da/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f21cf70e-682f-490c-bda5-3954df150487/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:55 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/74d4808e-24ce-4c47-b4ce-e1188eeea6f1/volumes/kubernetes.io~projected/kube-api-access-f547t DeviceMajor:0 DeviceMinor:238 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/59f6a4d6-afe0-4c96-a54a-d8e6b5c66a38/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy DeviceMajor:0 DeviceMinor:329 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-130 DeviceMajor:0 DeviceMinor:130 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cb9bef9b-c7cd-410e-8c1e-1b27474daf81/volumes/kubernetes.io~projected/kube-api-access-4ndt8 DeviceMajor:0 DeviceMinor:237 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f7d30630-43af-4157-b90f-d7eddef25085/volume-subpaths/web-config/config-reloader/6 DeviceMajor:0 DeviceMinor:273 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f7d30630-43af-4157-b90f-d7eddef25085/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:273 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/59f6a4d6-afe0-4c96-a54a-d8e6b5c66a38/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules DeviceMajor:0 DeviceMinor:324 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/59f6a4d6-afe0-4c96-a54a-d8e6b5c66a38/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:331 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/db895326-506d-479d-a526-6bf71be7f967/volumes/kubernetes.io~projected/kube-api-access-vlzhf DeviceMajor:0 DeviceMinor:62 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-94 DeviceMajor:0 DeviceMinor:94 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f7d30630-43af-4157-b90f-d7eddef25085/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:281 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2fd3c650-2f4c-404e-ab06-c50612af72ce/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:201 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3f681106-32d4-4550-a00d-1f9c52060b57/volumes/kubernetes.io~projected/kube-api-access-df59d DeviceMajor:0 DeviceMinor:203 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a7dae436e7bb325f83307dc7270e2a3b50e7675bfe3214d187fdaa4b63e06161/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-128 DeviceMajor:0 DeviceMinor:128 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d5d14caa-6661-4146-b888-59c3518216d4/volumes/kubernetes.io~secret/secret-metrics-server-tls DeviceMajor:0 DeviceMinor:338 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6fba5ae523e88c172564490952767809562c59f6a0322d27757e10868226a8c0/userdata/shm DeviceMajor:0 DeviceMinor:342 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/51dffd76-5c30-440c-85ed-d81221663833/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:415 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/682463ee-d9c9-4e9b-82b7-f1116898debf/volumes/kubernetes.io~projected/kube-api-access-nkjpg DeviceMajor:0 DeviceMinor:66 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f7d30630-43af-4157-b90f-d7eddef25085/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric DeviceMajor:0 DeviceMinor:278 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f7d30630-43af-4157-b90f-d7eddef25085/volumes/kubernetes.io~secret/secret-alertmanager-main-tls DeviceMajor:0 DeviceMinor:285 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-352 DeviceMajor:0 DeviceMinor:352 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-427 DeviceMajor:0 DeviceMinor:427 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/884cbb80-b3f8-4f90-bd55-328d17943cf3/volumes/kubernetes.io~projected/kube-api-access-szdn8 DeviceMajor:0 DeviceMinor:202 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-306 DeviceMajor:0 DeviceMinor:306 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d5d14caa-6661-4146-b888-59c3518216d4/volumes/kubernetes.io~secret/secret-metrics-server-client-certs DeviceMajor:0 DeviceMinor:339 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/47a7d0ed1c1bb8bcaef8b72693485b6259b9f0511abffeae75ba9fb88dae4ece/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-400 DeviceMajor:0 DeviceMinor:400 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-437 DeviceMajor:0 DeviceMinor:437 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-295 DeviceMajor:0 DeviceMinor:295 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cb9bef9b-c7cd-410e-8c1e-1b27474daf81/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:231 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-247 DeviceMajor:0 DeviceMinor:247 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-313 DeviceMajor:0 DeviceMinor:313 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-390 DeviceMajor:0 DeviceMinor:390 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cb9bef9b-c7cd-410e-8c1e-1b27474daf81/volumes/kubernetes.io~secret/openshift-state-metrics-tls DeviceMajor:0 DeviceMinor:233 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-311 DeviceMajor:0 DeviceMinor:311 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/39b3b6fef3084099c1fad9f5b7009f36f870612c1f58ece11a394def4ed6264f/userdata/shm DeviceMajor:0 DeviceMinor:419 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-145 DeviceMajor:0 DeviceMinor:145 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9126a982-59c9-4cec-a665-5f1bc0ae3389/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/74d4808e-24ce-4c47-b4ce-e1188eeea6f1/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:235 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9126a982-59c9-4cec-a665-5f1bc0ae3389/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:56 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4c92a5d0-b34f-46a1-8f8f-a9ae3cb8d5a5/volumes/kubernetes.io~projected/kube-api-access-6nd2w DeviceMajor:0 DeviceMinor:59 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/65008653-586d-4a76-a53f-7efe1f6d839b/volumes/kubernetes.io~projected/kube-api-access-8psbp DeviceMajor:0 DeviceMinor:60 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1a93c727fa866700775189988b37b9b744284f0a816bb9a05732853f97812258/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3f681106-32d4-4550-a00d-1f9c52060b57/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:197 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/884cbb80-b3f8-4f90-bd55-328d17943cf3/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:45 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-219 DeviceMajor:0 DeviceMinor:219 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-344 DeviceMajor:0 DeviceMinor:344 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-395 DeviceMajor:0 DeviceMinor:395 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f21cf70e-682f-490c-bda5-3954df150487/volumes/kubernetes.io~projected/kube-api-access-59sqq DeviceMajor:0 DeviceMinor:67 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2368c89e09f8a61dcbe91cdf62b26e6fc92980ef44c8d851cd77c563cae625c6/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/74d96638209d41f05199b24b4b3143d5e94c1b40a389dde0dcfbd1ceef3266b7/userdata/shm DeviceMajor:0 DeviceMinor:239 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f7d30630-43af-4157-b90f-d7eddef25085/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:279 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-377 DeviceMajor:0 DeviceMinor:377 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-346 DeviceMajor:0 DeviceMinor:346 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-442 DeviceMajor:0 DeviceMinor:442 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-289 DeviceMajor:0 DeviceMinor:289 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7e5ed4938ac5a455602f5935787f510257d22cfd87e815829fbc27ba58c1a45f/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f21cf70e-682f-490c-bda5-3954df150487/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:57 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9126a982-59c9-4cec-a665-5f1bc0ae3389/volumes/kubernetes.io~projected/kube-api-access-89vxf DeviceMajor:0 DeviceMinor:65 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0e73128cc29efba8f4b0ef35efead17c585a7fb5ac4db5f96fbfcde7e777d2d5/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5f4449d3d1de720c3b851fbffba06f75746a71960acf6a437b0ae3f926e304ca/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ee3ebf1d7f01d0a2963b1d18ebd247697e393917f93326d8b167e966ef5c8601/userdata/shm DeviceMajor:0 DeviceMinor:206 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2fd3c650-2f4c-404e-ab06-c50612af72ce/volumes/kubernetes.io~projected/kube-api-access-56z7v DeviceMajor:0 DeviceMinor:204 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f7d30630-43af-4157-b90f-d7eddef25085/volumes/kubernetes.io~secret/cluster-tls-config DeviceMajor:0 DeviceMinor:282 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f7d30630-43af-4157-b90f-d7eddef25085/volumes/kubernetes.io~projected/kube-api-access-z67jw DeviceMajor:0 DeviceMinor:284 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/61df4200b766b2af80af533f595a1e8304287c36ff3f42949b1bce70bb3f12e5/userdata/shm DeviceMajor:0 DeviceMinor:417 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-431 DeviceMajor:0 DeviceMinor:431 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/15b5070f-d0d3-4ad8-aa1d-93cc7b96003a/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:58 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f7d30630-43af-4157-b90f-d7eddef25085/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy DeviceMajor:0 DeviceMinor:277 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f7d30630-43af-4157-b90f-d7eddef25085/volumes/kubernetes.io~secret/config-volume DeviceMajor:0 DeviceMinor:283 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/59f6a4d6-afe0-4c96-a54a-d8e6b5c66a38/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics DeviceMajor:0 DeviceMinor:328 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f7d30630-43af-4157-b90f-d7eddef25085/volume-subpaths/cluster-tls-config/alertmanager/10 DeviceMajor:0 DeviceMinor:282 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fdfee0f501f2c3bd7ac1c00b1dfd5135480bcf94c3cc699f4bc1c05502ba7e57/userdata/shm DeviceMajor:0 DeviceMinor:87 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/212fa88bc72490cb2e7aee7e40dbef30ea7a30c81b02dce4efa80ce99a6a0225/userdata/shm DeviceMajor:0 DeviceMinor:240 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-249 DeviceMajor:0 DeviceMinor:249 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-253 DeviceMajor:0 DeviceMinor:253 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/59f6a4d6-afe0-4c96-a54a-d8e6b5c66a38/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:330 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-354 DeviceMajor:0 DeviceMinor:354 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-366 DeviceMajor:0 DeviceMinor:366 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8dd1760f632c5c28135a89fef44e119cf12f4fd22c927b2490d8a4995b99427a/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-154 DeviceMajor:0 DeviceMinor:154 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f7d30630-43af-4157-b90f-d7eddef25085/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:280 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-286 DeviceMajor:0 DeviceMinor:286 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-385 DeviceMajor:0 DeviceMinor:385 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:212fa88bc72490c MacAddress:4a:98:af:c4:57:6a Speed:10000 Mtu:8901} {Name:39b3b6fef308409 MacAddress:be:14:82:1e:c1:00 Speed:10000 Mtu:8901} {Name:47a7d0ed1c1bb8b MacAddress:7a:ae:79:c8:35:5c Speed:10000 Mtu:8901} {Name:5f9800b7bb6c494 MacAddress:7a:c9:6c:cc:a8:20 Speed:10000 Mtu:8901} {Name:61df4200b766b2a MacAddress:36:06:25:95:08:67 Speed:10000 Mtu:8901} {Name:6fba5ae523e88c1 MacAddress:b6:0e:bb:3d:7f:0e Speed:10000 Mtu:8901} {Name:8ef0ef34d0b7530 MacAddress:f6:98:2e:2c:31:57 Speed:10000 Mtu:8901} {Name:a7dae436e7bb325 MacAddress:82:4f:7d:4e:03:1c Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:fb:2c:5c:27 Speed:0 Mtu:9001} {Name:br-int MacAddress:8a:10:aa:5b:b7:d6 Speed:0 Mtu:8901} {Name:c7cefb72ae55f2d MacAddress:4a:53:ff:9f:db:44 Speed:10000 Mtu:8901} {Name:c93ea4ea8fbd9cc MacAddress:b6:31:3f:3a:1d:9b Speed:10000 Mtu:8901} {Name:ee3ebf1d7f01d0a MacAddress:ba:2c:47:90:98:fa Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:fb:2c:5c:27 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:da:bc:11:5b:f5:9c Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:16:3b:33:15:54:08 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 03 20:04:25.909999 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.909722    5570 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 03 20:04:25.909999 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.909790    5570 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 03 20:04:25.910174 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.910146    5570 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 03 20:04:25.910329 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.910175    5570 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-132-4.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 03 20:04:25.910403 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.910333    5570 topology_manager.go:143] "Creating topology manager with none policy"
Nov 03 20:04:25.910403 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.910346    5570 container_manager_linux.go:311] "Creating device plugin manager"
Nov 03 20:04:25.910403 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.910362    5570 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 20:04:25.910403 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.910380    5570 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 20:04:25.910605 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.910517    5570 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 03 20:04:25.910755 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.910741    5570 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 03 20:04:25.910900 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.910888    5570 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 03 20:04:25.910962 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.910939    5570 kubelet.go:493] "Attempting to sync node with API server"
Nov 03 20:04:25.910962 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.910953    5570 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 03 20:04:25.911049 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.910969    5570 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 03 20:04:25.911049 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.910982    5570 kubelet.go:405] "Adding apiserver pod source"
Nov 03 20:04:25.911049 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.910995    5570 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 03 20:04:25.911703 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.911687    5570 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 03 20:04:25.911703 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.911702    5570 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 20:04:25.912250 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.912232    5570 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 03 20:04:25.914718 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.914704    5570 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 03 20:04:25.914761 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.914719    5570 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 20:04:25.914889 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.914845    5570 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Nov 03 20:04:25.915043 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.915032    5570 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 03 20:04:25.915072 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.915062    5570 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 03 20:04:25.915156 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.915147    5570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 03 20:04:25.915189 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.915161    5570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 03 20:04:25.915189 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.915167    5570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 03 20:04:25.915189 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.915174    5570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 03 20:04:25.915189 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.915179    5570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 03 20:04:25.915189 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.915184    5570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 03 20:04:25.915189 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.915190    5570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 03 20:04:25.915356 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.915196    5570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 03 20:04:25.915356 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.915203    5570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 03 20:04:25.915356 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.915209    5570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 03 20:04:25.915356 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.915217    5570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 03 20:04:25.915356 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.915227    5570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 03 20:04:25.915356 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.915250    5570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 03 20:04:25.915356 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.915256    5570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 03 20:04:25.918958 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.918945    5570 server.go:1263] "Started kubelet"
Nov 03 20:04:25.919093 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.919043    5570 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 03 20:04:25.919127 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.919118    5570 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 03 20:04:25.920501 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.919336    5570 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 03 20:04:25.920501 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.920245    5570 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 03 20:04:25.919471 ip-10-0-132-4 systemd[1]: Started Kubernetes Kubelet.
Nov 03 20:04:25.923094 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.923005    5570 server.go:317] "Adding debug handlers to kubelet server"
Nov 03 20:04:25.927083 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.927063    5570 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 03 20:04:25.927169 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.927079    5570 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 03 20:04:25.927169 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.927140    5570 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 19:58:16 +0000 UTC" deadline="2028-06-19 19:14:18.107560926 +0000 UTC"
Nov 03 20:04:25.927169 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.927159    5570 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14255h9m52.180405024s"
Nov 03 20:04:25.927303 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.927185    5570 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 03 20:04:25.927303 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.927202    5570 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 03 20:04:25.927303 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.927287    5570 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 03 20:04:25.927303 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:04:25.927190    5570 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-4.ec2.internal\" not found"
Nov 03 20:04:25.929533 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.929504    5570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="15b5070f-d0d3-4ad8-aa1d-93cc7b96003a" volumeName="kubernetes.io/secret/15b5070f-d0d3-4ad8-aa1d-93cc7b96003a-agent-certs" seLinuxMountContext=""
Nov 03 20:04:25.929533 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.929532    5570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2073a8c8-6f07-4817-b7b3-468b327e2712" volumeName="kubernetes.io/configmap/2073a8c8-6f07-4817-b7b3-468b327e2712-iptables-alerter-script" seLinuxMountContext=""
Nov 03 20:04:25.929673 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.929542    5570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4c92a5d0-b34f-46a1-8f8f-a9ae3cb8d5a5" volumeName="kubernetes.io/secret/4c92a5d0-b34f-46a1-8f8f-a9ae3cb8d5a5-metrics-certs" seLinuxMountContext=""
Nov 03 20:04:25.929673 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.929551    5570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="51dffd76-5c30-440c-85ed-d81221663833" volumeName="kubernetes.io/secret/51dffd76-5c30-440c-85ed-d81221663833-original-pull-secret" seLinuxMountContext=""
Nov 03 20:04:25.929673 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.929560    5570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="59f6a4d6-afe0-4c96-a54a-d8e6b5c66a38" volumeName="kubernetes.io/empty-dir/59f6a4d6-afe0-4c96-a54a-d8e6b5c66a38-tmp" seLinuxMountContext=""
Nov 03 20:04:25.929673 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.929568    5570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="74d4808e-24ce-4c47-b4ce-e1188eeea6f1" volumeName="kubernetes.io/secret/74d4808e-24ce-4c47-b4ce-e1188eeea6f1-node-exporter-kube-rbac-proxy-config" seLinuxMountContext=""
Nov 03 20:04:25.929673 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.929575    5570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="884cbb80-b3f8-4f90-bd55-328d17943cf3" volumeName="kubernetes.io/empty-dir/884cbb80-b3f8-4f90-bd55-328d17943cf3-data-volume" seLinuxMountContext=""
Nov 03 20:04:25.929673 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.929582    5570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d5d14caa-6661-4146-b888-59c3518216d4" volumeName="kubernetes.io/secret/d5d14caa-6661-4146-b888-59c3518216d4-client-ca-bundle" seLinuxMountContext=""
Nov 03 20:04:25.929673 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.929590    5570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f7d30630-43af-4157-b90f-d7eddef25085" volumeName="kubernetes.io/configmap/f7d30630-43af-4157-b90f-d7eddef25085-metrics-client-ca" seLinuxMountContext=""
Nov 03 20:04:25.929673 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.929598    5570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f7d30630-43af-4157-b90f-d7eddef25085" volumeName="kubernetes.io/projected/f7d30630-43af-4157-b90f-d7eddef25085-kube-api-access-z67jw" seLinuxMountContext=""
Nov 03 20:04:25.929673 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.929607    5570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="59f6a4d6-afe0-4c96-a54a-d8e6b5c66a38" volumeName="kubernetes.io/configmap/59f6a4d6-afe0-4c96-a54a-d8e6b5c66a38-metrics-client-ca" seLinuxMountContext=""
Nov 03 20:04:25.929673 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.929614    5570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="74d4808e-24ce-4c47-b4ce-e1188eeea6f1" volumeName="kubernetes.io/empty-dir/74d4808e-24ce-4c47-b4ce-e1188eeea6f1-node-exporter-textfile" seLinuxMountContext=""
Nov 03 20:04:25.929673 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.929621    5570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="db895326-506d-479d-a526-6bf71be7f967" volumeName="kubernetes.io/projected/db895326-506d-479d-a526-6bf71be7f967-kube-api-access-vlzhf" seLinuxMountContext=""
Nov 03 20:04:25.929673 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.929629    5570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f7d30630-43af-4157-b90f-d7eddef25085" volumeName="kubernetes.io/secret/f7d30630-43af-4157-b90f-d7eddef25085-web-config" seLinuxMountContext=""
Nov 03 20:04:25.929673 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.929638    5570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2073a8c8-6f07-4817-b7b3-468b327e2712" volumeName="kubernetes.io/projected/2073a8c8-6f07-4817-b7b3-468b327e2712-kube-api-access-jrgcs" seLinuxMountContext=""
Nov 03 20:04:25.929673 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.929646    5570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4c92a5d0-b34f-46a1-8f8f-a9ae3cb8d5a5" volumeName="kubernetes.io/projected/4c92a5d0-b34f-46a1-8f8f-a9ae3cb8d5a5-kube-api-access-6nd2w" seLinuxMountContext=""
Nov 03 20:04:25.929673 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.929661    5570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="59f6a4d6-afe0-4c96-a54a-d8e6b5c66a38" volumeName="kubernetes.io/secret/59f6a4d6-afe0-4c96-a54a-d8e6b5c66a38-secret-thanos-querier-tls" seLinuxMountContext=""
Nov 03 20:04:25.929673 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.929669    5570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9126a982-59c9-4cec-a665-5f1bc0ae3389" volumeName="kubernetes.io/configmap/9126a982-59c9-4cec-a665-5f1bc0ae3389-ovnkube-script-lib" seLinuxMountContext=""
Nov 03 20:04:25.929673 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.929676    5570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cb9bef9b-c7cd-410e-8c1e-1b27474daf81" volumeName="kubernetes.io/secret/cb9bef9b-c7cd-410e-8c1e-1b27474daf81-openshift-state-metrics-tls" seLinuxMountContext=""
Nov 03 20:04:25.929673 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.929683    5570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f7d30630-43af-4157-b90f-d7eddef25085" volumeName="kubernetes.io/configmap/f7d30630-43af-4157-b90f-d7eddef25085-alertmanager-trusted-ca-bundle" seLinuxMountContext=""
Nov 03 20:04:25.930473 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.929690    5570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="59f6a4d6-afe0-4c96-a54a-d8e6b5c66a38" volumeName="kubernetes.io/secret/59f6a4d6-afe0-4c96-a54a-d8e6b5c66a38-secret-thanos-querier-kube-rbac-proxy" seLinuxMountContext=""
Nov 03 20:04:25.930473 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.929698    5570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="74d4808e-24ce-4c47-b4ce-e1188eeea6f1" volumeName="kubernetes.io/secret/74d4808e-24ce-4c47-b4ce-e1188eeea6f1-node-exporter-tls" seLinuxMountContext=""
Nov 03 20:04:25.930473 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.929706    5570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f7d30630-43af-4157-b90f-d7eddef25085" volumeName="kubernetes.io/secret/f7d30630-43af-4157-b90f-d7eddef25085-config-volume" seLinuxMountContext=""
Nov 03 20:04:25.930473 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.929713    5570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ffe074b1-e2a0-403b-9f81-581fdb4d2775" volumeName="kubernetes.io/configmap/ffe074b1-e2a0-403b-9f81-581fdb4d2775-kube-state-metrics-custom-resource-state-configmap" seLinuxMountContext=""
Nov 03 20:04:25.930473 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.929720    5570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2fd3c650-2f4c-404e-ab06-c50612af72ce" volumeName="kubernetes.io/secret/2fd3c650-2f4c-404e-ab06-c50612af72ce-metrics-tls" seLinuxMountContext=""
Nov 03 20:04:25.930473 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.929728    5570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="716cc9cf-d26a-481b-8114-169c6104ce56" volumeName="kubernetes.io/projected/716cc9cf-d26a-481b-8114-169c6104ce56-kube-api-access-vrdtp" seLinuxMountContext=""
Nov 03 20:04:25.930473 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.929735    5570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cb9bef9b-c7cd-410e-8c1e-1b27474daf81" volumeName="kubernetes.io/projected/cb9bef9b-c7cd-410e-8c1e-1b27474daf81-kube-api-access-4ndt8" seLinuxMountContext=""
Nov 03 20:04:25.930473 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.929742    5570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d5d14caa-6661-4146-b888-59c3518216d4" volumeName="kubernetes.io/projected/d5d14caa-6661-4146-b888-59c3518216d4-kube-api-access-8wq4p" seLinuxMountContext=""
Nov 03 20:04:25.930473 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.929753    5570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d71eaa2a-9659-4827-9dfc-c1fce02565cb" volumeName="kubernetes.io/projected/d71eaa2a-9659-4827-9dfc-c1fce02565cb-kube-api-access-zcxdr" seLinuxMountContext=""
Nov 03 20:04:25.930473 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.929786    5570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="db895326-506d-479d-a526-6bf71be7f967" volumeName="kubernetes.io/configmap/db895326-506d-479d-a526-6bf71be7f967-cni-sysctl-allowlist" seLinuxMountContext=""
Nov 03 20:04:25.930473 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.929799    5570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f7d30630-43af-4157-b90f-d7eddef25085" volumeName="kubernetes.io/secret/f7d30630-43af-4157-b90f-d7eddef25085-secret-alertmanager-kube-rbac-proxy-metric" seLinuxMountContext=""
Nov 03 20:04:25.930473 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.929810    5570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ffe074b1-e2a0-403b-9f81-581fdb4d2775" volumeName="kubernetes.io/projected/ffe074b1-e2a0-403b-9f81-581fdb4d2775-kube-api-access-4g29t" seLinuxMountContext=""
Nov 03 20:04:25.930473 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.929827    5570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="65008653-586d-4a76-a53f-7efe1f6d839b" volumeName="kubernetes.io/projected/65008653-586d-4a76-a53f-7efe1f6d839b-kube-api-access-8psbp" seLinuxMountContext=""
Nov 03 20:04:25.930473 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.929842    5570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9126a982-59c9-4cec-a665-5f1bc0ae3389" volumeName="kubernetes.io/secret/9126a982-59c9-4cec-a665-5f1bc0ae3389-ovn-node-metrics-cert" seLinuxMountContext=""
Nov 03 20:04:25.930473 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.929856    5570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d5d14caa-6661-4146-b888-59c3518216d4" volumeName="kubernetes.io/configmap/d5d14caa-6661-4146-b888-59c3518216d4-metrics-server-audit-profiles" seLinuxMountContext=""
Nov 03 20:04:25.930473 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.929868    5570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="db895326-506d-479d-a526-6bf71be7f967" volumeName="kubernetes.io/configmap/db895326-506d-479d-a526-6bf71be7f967-cni-binary-copy" seLinuxMountContext=""
Nov 03 20:04:25.930473 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.929879    5570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f21cf70e-682f-490c-bda5-3954df150487" volumeName="kubernetes.io/empty-dir/f21cf70e-682f-490c-bda5-3954df150487-tmp" seLinuxMountContext=""
Nov 03 20:04:25.930473 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.929891    5570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ffe074b1-e2a0-403b-9f81-581fdb4d2775" volumeName="kubernetes.io/configmap/ffe074b1-e2a0-403b-9f81-581fdb4d2775-metrics-client-ca" seLinuxMountContext=""
Nov 03 20:04:25.930473 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.929907    5570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="682463ee-d9c9-4e9b-82b7-f1116898debf" volumeName="kubernetes.io/configmap/682463ee-d9c9-4e9b-82b7-f1116898debf-cni-binary-copy" seLinuxMountContext=""
Nov 03 20:04:25.930473 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.929926    5570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1d4ce60c-e058-4c29-99e8-5f4f6eead744" volumeName="kubernetes.io/projected/1d4ce60c-e058-4c29-99e8-5f4f6eead744-kube-api-access-s92q2" seLinuxMountContext=""
Nov 03 20:04:25.930473 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.929943    5570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3f681106-32d4-4550-a00d-1f9c52060b57" volumeName="kubernetes.io/projected/3f681106-32d4-4550-a00d-1f9c52060b57-kube-api-access-df59d" seLinuxMountContext=""
Nov 03 20:04:25.930473 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.929958    5570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="59f6a4d6-afe0-4c96-a54a-d8e6b5c66a38" volumeName="kubernetes.io/secret/59f6a4d6-afe0-4c96-a54a-d8e6b5c66a38-secret-thanos-querier-kube-rbac-proxy-web" seLinuxMountContext=""
Nov 03 20:04:25.930473 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.929971    5570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9126a982-59c9-4cec-a665-5f1bc0ae3389" volumeName="kubernetes.io/projected/9126a982-59c9-4cec-a665-5f1bc0ae3389-kube-api-access-89vxf" seLinuxMountContext=""
Nov 03 20:04:25.930473 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.929987    5570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d5d14caa-6661-4146-b888-59c3518216d4" volumeName="kubernetes.io/configmap/d5d14caa-6661-4146-b888-59c3518216d4-configmap-kubelet-serving-ca-bundle" seLinuxMountContext=""
Nov 03 20:04:25.930473 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.930007    5570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="db895326-506d-479d-a526-6bf71be7f967" volumeName="kubernetes.io/configmap/db895326-506d-479d-a526-6bf71be7f967-whereabouts-flatfile-configmap" seLinuxMountContext=""
Nov 03 20:04:25.930473 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.930025    5570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="884cbb80-b3f8-4f90-bd55-328d17943cf3" volumeName="kubernetes.io/projected/884cbb80-b3f8-4f90-bd55-328d17943cf3-kube-api-access-szdn8" seLinuxMountContext=""
Nov 03 20:04:25.930473 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.930041    5570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f7d30630-43af-4157-b90f-d7eddef25085" volumeName="kubernetes.io/secret/f7d30630-43af-4157-b90f-d7eddef25085-secret-alertmanager-main-tls" seLinuxMountContext=""
Nov 03 20:04:25.930473 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.930057    5570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="59f6a4d6-afe0-4c96-a54a-d8e6b5c66a38" volumeName="kubernetes.io/secret/59f6a4d6-afe0-4c96-a54a-d8e6b5c66a38-secret-thanos-querier-kube-rbac-proxy-metrics" seLinuxMountContext=""
Nov 03 20:04:25.930473 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.930073    5570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cb9bef9b-c7cd-410e-8c1e-1b27474daf81" volumeName="kubernetes.io/configmap/cb9bef9b-c7cd-410e-8c1e-1b27474daf81-metrics-client-ca" seLinuxMountContext=""
Nov 03 20:04:25.930473 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.930093    5570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d5d14caa-6661-4146-b888-59c3518216d4" volumeName="kubernetes.io/secret/d5d14caa-6661-4146-b888-59c3518216d4-secret-metrics-server-client-certs" seLinuxMountContext=""
Nov 03 20:04:25.930473 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.930114    5570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f21cf70e-682f-490c-bda5-3954df150487" volumeName="kubernetes.io/empty-dir/f21cf70e-682f-490c-bda5-3954df150487-etc-tuned" seLinuxMountContext=""
Nov 03 20:04:25.930473 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.930129    5570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1d4ce60c-e058-4c29-99e8-5f4f6eead744" volumeName="kubernetes.io/configmap/1d4ce60c-e058-4c29-99e8-5f4f6eead744-serviceca" seLinuxMountContext=""
Nov 03 20:04:25.930473 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.930143    5570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f21cf70e-682f-490c-bda5-3954df150487" volumeName="kubernetes.io/projected/f21cf70e-682f-490c-bda5-3954df150487-kube-api-access-59sqq" seLinuxMountContext=""
Nov 03 20:04:25.930473 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.930158    5570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3f681106-32d4-4550-a00d-1f9c52060b57" volumeName="kubernetes.io/secret/3f681106-32d4-4550-a00d-1f9c52060b57-cert" seLinuxMountContext=""
Nov 03 20:04:25.930473 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.930169    5570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="716cc9cf-d26a-481b-8114-169c6104ce56" volumeName="kubernetes.io/empty-dir/716cc9cf-d26a-481b-8114-169c6104ce56-tmp-dir" seLinuxMountContext=""
Nov 03 20:04:25.930473 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.930185    5570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ffe074b1-e2a0-403b-9f81-581fdb4d2775" volumeName="kubernetes.io/secret/ffe074b1-e2a0-403b-9f81-581fdb4d2775-kube-state-metrics-kube-rbac-proxy-config" seLinuxMountContext=""
Nov 03 20:04:25.930473 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.930215    5570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ffe074b1-e2a0-403b-9f81-581fdb4d2775" volumeName="kubernetes.io/secret/ffe074b1-e2a0-403b-9f81-581fdb4d2775-kube-state-metrics-tls" seLinuxMountContext=""
Nov 03 20:04:25.930473 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.930233    5570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="15b5070f-d0d3-4ad8-aa1d-93cc7b96003a" volumeName="kubernetes.io/configmap/15b5070f-d0d3-4ad8-aa1d-93cc7b96003a-konnectivity-ca" seLinuxMountContext=""
Nov 03 20:04:25.930473 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.930252    5570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2fd3c650-2f4c-404e-ab06-c50612af72ce" volumeName="kubernetes.io/empty-dir/2fd3c650-2f4c-404e-ab06-c50612af72ce-tmp-dir" seLinuxMountContext=""
Nov 03 20:04:25.930473 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.930268    5570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="59f6a4d6-afe0-4c96-a54a-d8e6b5c66a38" volumeName="kubernetes.io/secret/59f6a4d6-afe0-4c96-a54a-d8e6b5c66a38-secret-thanos-querier-kube-rbac-proxy-rules" seLinuxMountContext=""
Nov 03 20:04:25.930473 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.930282    5570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="682463ee-d9c9-4e9b-82b7-f1116898debf" volumeName="kubernetes.io/configmap/682463ee-d9c9-4e9b-82b7-f1116898debf-multus-daemon-config" seLinuxMountContext=""
Nov 03 20:04:25.930473 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.930299    5570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="884cbb80-b3f8-4f90-bd55-328d17943cf3" volumeName="kubernetes.io/configmap/884cbb80-b3f8-4f90-bd55-328d17943cf3-kube-rbac-proxy-cm" seLinuxMountContext=""
Nov 03 20:04:25.930473 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.930313    5570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cb9bef9b-c7cd-410e-8c1e-1b27474daf81" volumeName="kubernetes.io/secret/cb9bef9b-c7cd-410e-8c1e-1b27474daf81-openshift-state-metrics-kube-rbac-proxy-config" seLinuxMountContext=""
Nov 03 20:04:25.930473 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.930327    5570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f7d30630-43af-4157-b90f-d7eddef25085" volumeName="kubernetes.io/empty-dir/f7d30630-43af-4157-b90f-d7eddef25085-config-out" seLinuxMountContext=""
Nov 03 20:04:25.930473 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.930345    5570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f7d30630-43af-4157-b90f-d7eddef25085" volumeName="kubernetes.io/projected/f7d30630-43af-4157-b90f-d7eddef25085-tls-assets" seLinuxMountContext=""
Nov 03 20:04:25.930473 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.930361    5570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2fd3c650-2f4c-404e-ab06-c50612af72ce" volumeName="kubernetes.io/projected/2fd3c650-2f4c-404e-ab06-c50612af72ce-kube-api-access-56z7v" seLinuxMountContext=""
Nov 03 20:04:25.930473 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.930376    5570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="59f6a4d6-afe0-4c96-a54a-d8e6b5c66a38" volumeName="kubernetes.io/projected/59f6a4d6-afe0-4c96-a54a-d8e6b5c66a38-kube-api-access-w7hn2" seLinuxMountContext=""
Nov 03 20:04:25.930473 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.930393    5570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="59f6a4d6-afe0-4c96-a54a-d8e6b5c66a38" volumeName="kubernetes.io/secret/59f6a4d6-afe0-4c96-a54a-d8e6b5c66a38-secret-grpc-tls" seLinuxMountContext=""
Nov 03 20:04:25.930473 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.930406    5570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="74d4808e-24ce-4c47-b4ce-e1188eeea6f1" volumeName="kubernetes.io/configmap/74d4808e-24ce-4c47-b4ce-e1188eeea6f1-node-exporter-accelerators-collector-config" seLinuxMountContext=""
Nov 03 20:04:25.930473 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.930421    5570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="884cbb80-b3f8-4f90-bd55-328d17943cf3" volumeName="kubernetes.io/secret/884cbb80-b3f8-4f90-bd55-328d17943cf3-insights-runtime-extractor-tls" seLinuxMountContext=""
Nov 03 20:04:25.930473 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.930439    5570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f7d30630-43af-4157-b90f-d7eddef25085" volumeName="kubernetes.io/secret/f7d30630-43af-4157-b90f-d7eddef25085-secret-alertmanager-kube-rbac-proxy" seLinuxMountContext=""
Nov 03 20:04:25.930473 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.930376    5570 factory.go:55] Registering systemd factory
Nov 03 20:04:25.930473 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.930455    5570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f7d30630-43af-4157-b90f-d7eddef25085" volumeName="kubernetes.io/secret/f7d30630-43af-4157-b90f-d7eddef25085-secret-alertmanager-kube-rbac-proxy-web" seLinuxMountContext=""
Nov 03 20:04:25.930473 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.930472    5570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="682463ee-d9c9-4e9b-82b7-f1116898debf" volumeName="kubernetes.io/projected/682463ee-d9c9-4e9b-82b7-f1116898debf-kube-api-access-nkjpg" seLinuxMountContext=""
Nov 03 20:04:25.930473 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.930476    5570 factory.go:223] Registration of the systemd container factory successfully
Nov 03 20:04:25.930473 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.930488    5570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="74d4808e-24ce-4c47-b4ce-e1188eeea6f1" volumeName="kubernetes.io/configmap/74d4808e-24ce-4c47-b4ce-e1188eeea6f1-metrics-client-ca" seLinuxMountContext=""
Nov 03 20:04:25.932973 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.930502    5570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="74d4808e-24ce-4c47-b4ce-e1188eeea6f1" volumeName="kubernetes.io/projected/74d4808e-24ce-4c47-b4ce-e1188eeea6f1-kube-api-access-f547t" seLinuxMountContext=""
Nov 03 20:04:25.932973 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.930518    5570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9126a982-59c9-4cec-a665-5f1bc0ae3389" volumeName="kubernetes.io/configmap/9126a982-59c9-4cec-a665-5f1bc0ae3389-ovnkube-config" seLinuxMountContext=""
Nov 03 20:04:25.932973 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.930536    5570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d5d14caa-6661-4146-b888-59c3518216d4" volumeName="kubernetes.io/secret/d5d14caa-6661-4146-b888-59c3518216d4-secret-metrics-server-tls" seLinuxMountContext=""
Nov 03 20:04:25.932973 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.930552    5570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2fd3c650-2f4c-404e-ab06-c50612af72ce" volumeName="kubernetes.io/configmap/2fd3c650-2f4c-404e-ab06-c50612af72ce-config-volume" seLinuxMountContext=""
Nov 03 20:04:25.932973 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.930565    5570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9126a982-59c9-4cec-a665-5f1bc0ae3389" volumeName="kubernetes.io/configmap/9126a982-59c9-4cec-a665-5f1bc0ae3389-env-overrides" seLinuxMountContext=""
Nov 03 20:04:25.932973 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.930582    5570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d5d14caa-6661-4146-b888-59c3518216d4" volumeName="kubernetes.io/empty-dir/d5d14caa-6661-4146-b888-59c3518216d4-audit-log" seLinuxMountContext=""
Nov 03 20:04:25.932973 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.930597    5570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f7d30630-43af-4157-b90f-d7eddef25085" volumeName="kubernetes.io/empty-dir/f7d30630-43af-4157-b90f-d7eddef25085-alertmanager-main-db" seLinuxMountContext=""
Nov 03 20:04:25.932973 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.930612    5570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f7d30630-43af-4157-b90f-d7eddef25085" volumeName="kubernetes.io/secret/f7d30630-43af-4157-b90f-d7eddef25085-cluster-tls-config" seLinuxMountContext=""
Nov 03 20:04:25.932973 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.930629    5570 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ffe074b1-e2a0-403b-9f81-581fdb4d2775" volumeName="kubernetes.io/empty-dir/ffe074b1-e2a0-403b-9f81-581fdb4d2775-volume-directive-shadow" seLinuxMountContext=""
Nov 03 20:04:25.932973 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.930647    5570 reconstruct.go:97] "Volume reconstruction finished"
Nov 03 20:04:25.932973 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.930660    5570 reconciler.go:29] "Reconciler: start to sync state"
Nov 03 20:04:25.932973 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.931390    5570 factory.go:153] Registering CRI-O factory
Nov 03 20:04:25.932973 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.931475    5570 factory.go:223] Registration of the crio container factory successfully
Nov 03 20:04:25.932973 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.931566    5570 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 03 20:04:25.932973 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.931589    5570 factory.go:103] Registering Raw factory
Nov 03 20:04:25.932973 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.931606    5570 manager.go:1199] Started watching for new ooms in manager
Nov 03 20:04:25.932973 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.931952    5570 manager.go:319] Starting recovery of all containers
Nov 03 20:04:25.954761 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.954712    5570 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 03 20:04:25.956644 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.956618    5570 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 03 20:04:25.956644 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.956643    5570 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 03 20:04:25.956836 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.956667    5570 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 03 20:04:25.956836 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:04:25.956709    5570 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Nov 03 20:04:25.968909 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.968878    5570 generic.go:356] "Generic (PLEG): container finished" podID="f7d30630-43af-4157-b90f-d7eddef25085" containerID="785eedffac63800711a89514aec53147c740e3e22adcbae196ce94713ef27782" exitCode=0
Nov 03 20:04:25.970417 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.970367    5570 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Nov 03 20:04:25.973595 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.973566    5570 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 20:04:25.973700 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.973642    5570 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 20:04:25.980030 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.980006    5570 generic.go:356] "Generic (PLEG): container finished" podID="74d4808e-24ce-4c47-b4ce-e1188eeea6f1" containerID="4e6d31cdfaa90ea53ffda45d72fa7ea49e5905a25dc84b2607056358cf3863ed" exitCode=0
Nov 03 20:04:25.990011 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.989982    5570 generic.go:356] "Generic (PLEG): container finished" podID="5dff8c1e3d20edc13c5947130e8a5c14" containerID="a36b3d172800908c76d5e9fe34ca4648289f454367148e1e5ea3959a3de88550" exitCode=0
Nov 03 20:04:25.994724 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.994700    5570 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 20:04:25.995243 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:25.995224    5570 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 20:04:26.009149 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:26.009123    5570 generic.go:356] "Generic (PLEG): container finished" podID="db895326-506d-479d-a526-6bf71be7f967" containerID="f67c692909431ab5ef63756f63292003698c2a636cd7f4307477e9439029858a" exitCode=0
Nov 03 20:04:26.009149 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:26.009145    5570 generic.go:356] "Generic (PLEG): container finished" podID="db895326-506d-479d-a526-6bf71be7f967" containerID="c3dc107a0c3b728fd541122d7dd467b9ed50d1642039d41bfe2634b0427a7687" exitCode=0
Nov 03 20:04:26.009149 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:26.009153    5570 generic.go:356] "Generic (PLEG): container finished" podID="db895326-506d-479d-a526-6bf71be7f967" containerID="214cfdccb0b4051ea1e1eed79c0796d2215b432938f46ad5c6bc09d398b15f6c" exitCode=0
Nov 03 20:04:26.009378 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:26.009160    5570 generic.go:356] "Generic (PLEG): container finished" podID="db895326-506d-479d-a526-6bf71be7f967" containerID="18e7de51ee6637ad7c2c65b56530df6fba343d1141603e76d8231e46da4d68ed" exitCode=0
Nov 03 20:04:26.009378 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:26.009168    5570 generic.go:356] "Generic (PLEG): container finished" podID="db895326-506d-479d-a526-6bf71be7f967" containerID="f3db6f558a5b35961c39f860bf189899d8c34ed67dbb5a8263c3eb7c4119e2f7" exitCode=0
Nov 03 20:04:26.009378 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:26.009176    5570 generic.go:356] "Generic (PLEG): container finished" podID="db895326-506d-479d-a526-6bf71be7f967" containerID="a701c1ac8d52e702cf58a7a8a4c2499cd44c210eb94b4dd1f0b6a71b821668a4" exitCode=0
Nov 03 20:04:26.041264 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:26.041201    5570 manager.go:324] Recovery completed
Nov 03 20:04:26.057742 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:04:26.057717    5570 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet"
Nov 03 20:04:26.063083 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:26.063062    5570 cpu_manager.go:226] "Starting" policy="none"
Nov 03 20:04:26.063083 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:26.063083    5570 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 03 20:04:26.063242 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:26.063104    5570 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 03 20:04:26.063283 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:26.063266    5570 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Nov 03 20:04:26.063318 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:26.063285    5570 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Nov 03 20:04:26.063318 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:26.063310    5570 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Nov 03 20:04:26.063381 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:26.063319    5570 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Nov 03 20:04:26.063381 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:26.063326    5570 policy_none.go:50] "Start"
Nov 03 20:04:26.063381 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:26.063338    5570 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 03 20:04:26.063381 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:26.063350    5570 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 03 20:04:26.063533 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:26.063522    5570 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Nov 03 20:04:26.063565 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:26.063534    5570 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Nov 03 20:04:26.063565 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:26.063540    5570 policy_none.go:44] "Start"
Nov 03 20:04:26.068096 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:26.068051    5570 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 03 20:04:26.070807 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:26.070788    5570 manager.go:341] "Starting Device Plugin manager"
Nov 03 20:04:26.071573 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:04:26.070826    5570 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 03 20:04:26.071620 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:26.071580    5570 server.go:87] "Starting device plugin registration server"
Nov 03 20:04:26.071973 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:26.071959    5570 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 03 20:04:26.072029 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:26.071977    5570 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 03 20:04:26.072075 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:26.072062    5570 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 03 20:04:26.072188 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:26.072155    5570 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 03 20:04:26.072257 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:26.072198    5570 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 03 20:04:26.072257 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:26.072205    5570 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 03 20:04:26.072344 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:26.072253    5570 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T20:04:26.072184508Z","UUID":"3e7e60d2-8302-417a-8fc8-0ae24e612601","Handler":null,"Name":"","Endpoint":""}
Nov 03 20:04:26.074163 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:04:26.074136    5570 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 03 20:04:26.074548 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:26.074501    5570 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 03 20:04:26.074548 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:26.074540    5570 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 03 20:04:26.172611 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:26.172577    5570 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 20:04:26.174676 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:26.174655    5570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-4.ec2.internal" event="NodeHasSufficientMemory"
Nov 03 20:04:26.174811 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:26.174687    5570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-4.ec2.internal" event="NodeHasNoDiskPressure"
Nov 03 20:04:26.174811 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:26.174697    5570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-4.ec2.internal" event="NodeHasSufficientPID"
Nov 03 20:04:26.174811 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:26.174738    5570 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-132-4.ec2.internal"
Nov 03 20:04:26.208199 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:26.208164    5570 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-132-4.ec2.internal"
Nov 03 20:04:26.208358 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:26.208251    5570 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-132-4.ec2.internal"
Nov 03 20:04:26.258049 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:26.257995    5570 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-132-4.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-4.ec2.internal"]
Nov 03 20:04:26.258594 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:26.258561    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-4.ec2.internal" event={"ID":"a66c9af704a53c4c9b04653e0d8d8d8f","Type":"ContainerStarted","Data":"02df7ae991422a8e3fe518fd89cfc64e29f3dedc413076f1956aeec1ec7c5185"}
Nov 03 20:04:26.258652 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:26.258601    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-4.ec2.internal" event={"ID":"a66c9af704a53c4c9b04653e0d8d8d8f","Type":"ContainerStarted","Data":"b7d5a4479f6b4cfc2d84e97abf7442f621815298db4681aa2e43df8ccb9ec0da"}
Nov 03 20:04:26.258689 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:26.258650    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-4.ec2.internal" event={"ID":"5dff8c1e3d20edc13c5947130e8a5c14","Type":"ContainerStarted","Data":"f9e1beb267e03b7b0f87971a0c028bb780f4b92d5f92314d8290765e99268d5a"}
Nov 03 20:04:26.258689 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:26.258659    5570 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-4.ec2.internal"
Nov 03 20:04:26.258758 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:26.258666    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-4.ec2.internal" event={"ID":"5dff8c1e3d20edc13c5947130e8a5c14","Type":"ContainerDied","Data":"a36b3d172800908c76d5e9fe34ca4648289f454367148e1e5ea3959a3de88550"}
Nov 03 20:04:26.258834 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:26.258756    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-4.ec2.internal" event={"ID":"5dff8c1e3d20edc13c5947130e8a5c14","Type":"ContainerStarted","Data":"7e5ed4938ac5a455602f5935787f510257d22cfd87e815829fbc27ba58c1a45f"}
Nov 03 20:04:26.258868 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:26.258827    5570 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-4.ec2.internal"
Nov 03 20:04:26.279925 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:26.279902    5570 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 20:04:26.280034 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:04:26.279952    5570 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-132-4.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-4.ec2.internal"
Nov 03 20:04:26.280265 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:26.280244    5570 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 20:04:26.280388 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:04:26.280285    5570 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-132-4.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-4.ec2.internal"
Nov 03 20:04:26.372669 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:26.372569    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a66c9af704a53c4c9b04653e0d8d8d8f-config\") pod \"kube-apiserver-proxy-ip-10-0-132-4.ec2.internal\" (UID: \"a66c9af704a53c4c9b04653e0d8d8d8f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-4.ec2.internal"
Nov 03 20:04:26.372669 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:26.372644    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5dff8c1e3d20edc13c5947130e8a5c14-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-4.ec2.internal\" (UID: \"5dff8c1e3d20edc13c5947130e8a5c14\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-4.ec2.internal"
Nov 03 20:04:26.372883 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:26.372670    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5dff8c1e3d20edc13c5947130e8a5c14-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-4.ec2.internal\" (UID: \"5dff8c1e3d20edc13c5947130e8a5c14\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-4.ec2.internal"
Nov 03 20:04:26.473185 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:26.473151    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a66c9af704a53c4c9b04653e0d8d8d8f-config\") pod \"kube-apiserver-proxy-ip-10-0-132-4.ec2.internal\" (UID: \"a66c9af704a53c4c9b04653e0d8d8d8f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-4.ec2.internal"
Nov 03 20:04:26.473185 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:26.473192    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5dff8c1e3d20edc13c5947130e8a5c14-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-4.ec2.internal\" (UID: \"5dff8c1e3d20edc13c5947130e8a5c14\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-4.ec2.internal"
Nov 03 20:04:26.473414 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:26.473210    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5dff8c1e3d20edc13c5947130e8a5c14-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-4.ec2.internal\" (UID: \"5dff8c1e3d20edc13c5947130e8a5c14\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-4.ec2.internal"
Nov 03 20:04:26.473414 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:26.473282    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a66c9af704a53c4c9b04653e0d8d8d8f-config\") pod \"kube-apiserver-proxy-ip-10-0-132-4.ec2.internal\" (UID: \"a66c9af704a53c4c9b04653e0d8d8d8f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-4.ec2.internal"
Nov 03 20:04:26.473414 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:26.473335    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5dff8c1e3d20edc13c5947130e8a5c14-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-4.ec2.internal\" (UID: \"5dff8c1e3d20edc13c5947130e8a5c14\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-4.ec2.internal"
Nov 03 20:04:26.473414 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:26.473378    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5dff8c1e3d20edc13c5947130e8a5c14-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-4.ec2.internal\" (UID: \"5dff8c1e3d20edc13c5947130e8a5c14\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-4.ec2.internal"
Nov 03 20:04:26.911963 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:26.911923    5570 apiserver.go:52] "Watching apiserver"
Nov 03 20:04:26.963043 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:26.963004    5570 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 03 20:04:26.963599 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:26.963577    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-5jnts","openshift-monitoring/alertmanager-main-0","openshift-monitoring/kube-state-metrics-5dc8d7d89b-q8b6s","openshift-multus/network-metrics-daemon-c947w","openshift-ovn-kubernetes/ovnkube-node-6sv7j","kube-system/kube-apiserver-proxy-ip-10-0-132-4.ec2.internal","openshift-dns/node-resolver-t6dqw","openshift-monitoring/openshift-state-metrics-7bd7489957-rwhnf","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9f5s5","openshift-dns/dns-default-7zl2f","openshift-monitoring/metrics-server-654b7fcdc8-l6l9f","openshift-network-diagnostics/network-check-target-6dfr5","openshift-insights/insights-runtime-extractor-8bwhz","openshift-monitoring/node-exporter-j46kr","openshift-multus/multus-additional-cni-plugins-mhtmw","openshift-monitoring/thanos-querier-5bd4bfb68d-9449x","kube-system/global-pull-secret-syncer-j4jzv","openshift-cluster-node-tuning-operator/tuned-4dmkq","openshift-multus/multus-h6p95","kube-system/konnectivity-agent-5p48g","openshift-image-registry/node-ca-zzt8t","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-4.ec2.internal","openshift-network-operator/iptables-alerter-9s4wk"]
Nov 03 20:04:26.980425 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:26.980400    5570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 03 20:04:26.980567 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:26.980451    5570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 03 20:04:26.980686 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:26.980666    5570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 03 20:04:26.980759 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:26.980685    5570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 03 20:04:26.980759 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:26.980688    5570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 03 20:04:26.980759 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:26.980698    5570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 03 20:04:26.981068 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:26.981046    5570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 03 20:04:26.981213 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:26.981191    5570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 03 20:04:26.981341 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:26.981255    5570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 03 20:04:26.981341 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:26.981261    5570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 03 20:04:26.981341 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:26.981297    5570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 03 20:04:26.981562 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:26.981546    5570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 03 20:04:26.981618 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:26.981546    5570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 03 20:04:26.981664 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:26.981639    5570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 03 20:04:26.981757 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:26.981733    5570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 03 20:04:26.981889 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:26.981815    5570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 03 20:04:27.007652 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.007631    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 03 20:04:27.007971 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.007951    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 03 20:04:27.008032 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.007989    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-phn2h\""
Nov 03 20:04:27.008032 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.008007    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Nov 03 20:04:27.008133 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.008042    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Nov 03 20:04:27.008133 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.008082    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Nov 03 20:04:27.008305 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.008286    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-7kzdq\""
Nov 03 20:04:27.008357 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.008290    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-n7xhm\""
Nov 03 20:04:27.008394 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.008361    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 03 20:04:27.008515 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.008501    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-hwv2m\""
Nov 03 20:04:27.008663 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.008650    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Nov 03 20:04:27.008744 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.008730    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Nov 03 20:04:27.008846 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.008832    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Nov 03 20:04:27.008919 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.008842    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Nov 03 20:04:27.008919 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.008849    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Nov 03 20:04:27.008919 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.008881    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Nov 03 20:04:27.009062 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.008958    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Nov 03 20:04:27.009062 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.008960    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-ao94rttoovh0l\""
Nov 03 20:04:27.009062 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.008959    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Nov 03 20:04:27.009062 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.008988    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Nov 03 20:04:27.009235 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.009111    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-2qw54\""
Nov 03 20:04:27.009235 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.009175    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-kqzwq\""
Nov 03 20:04:27.009235 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.009203    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-p4f2k\""
Nov 03 20:04:27.009235 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.009206    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-m22xr\""
Nov 03 20:04:27.009235 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.009234    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-5kpvl\""
Nov 03 20:04:27.009456 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.009240    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-86559\""
Nov 03 20:04:27.009456 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.009429    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Nov 03 20:04:27.009548 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.009524    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Nov 03 20:04:27.009548 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.009532    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Nov 03 20:04:27.009548 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.009523    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Nov 03 20:04:27.009697 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.009571    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Nov 03 20:04:27.009697 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.009578    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-6ldnaap773utg\""
Nov 03 20:04:27.009697 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.009577    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-kzs97\""
Nov 03 20:04:27.009697 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.009594    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 03 20:04:27.009697 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.009579    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 03 20:04:27.009697 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.009604    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-kk6pp\""
Nov 03 20:04:27.009697 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.009592    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-v94zj\""
Nov 03 20:04:27.009697 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.009627    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-89ghp\""
Nov 03 20:04:27.009697 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.009615    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-mbtbz\""
Nov 03 20:04:27.009697 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.009615    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-89f7f\""
Nov 03 20:04:27.009697 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.009656    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 03 20:04:27.009697 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.009661    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 03 20:04:27.009697 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.009693    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-q2t72\""
Nov 03 20:04:27.009697 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.009695    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 03 20:04:27.009697 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.009701    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Nov 03 20:04:27.010142 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.009705    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-b4qhs\""
Nov 03 20:04:27.010142 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.009695    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Nov 03 20:04:27.011140 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.011128    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-tmbxc\""
Nov 03 20:04:27.011189 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.011156    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Nov 03 20:04:27.011189 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.011164    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-rnjcd\""
Nov 03 20:04:27.011189 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.011170    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-dfgm6\""
Nov 03 20:04:27.011286 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.011170    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Nov 03 20:04:27.019052 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.019035    5570 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-4.ec2.internal"
Nov 03 20:04:27.019165 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.019141    5570 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-4.ec2.internal"
Nov 03 20:04:27.029796 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.029763    5570 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 03 20:04:27.039017 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.038997    5570 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 20:04:27.039166 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:04:27.039046    5570 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-132-4.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-4.ec2.internal"
Nov 03 20:04:27.039358 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.039339    5570 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 20:04:27.039406 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:04:27.039380    5570 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-132-4.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-4.ec2.internal"
Nov 03 20:04:27.076098 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.076071    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/65008653-586d-4a76-a53f-7efe1f6d839b-sys-fs\") pod \"aws-ebs-csi-driver-node-9f5s5\" (UID: \"65008653-586d-4a76-a53f-7efe1f6d839b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9f5s5"
Nov 03 20:04:27.076098 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.076099    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/51dffd76-5c30-440c-85ed-d81221663833-kubelet-config\") pod \"global-pull-secret-syncer-j4jzv\" (UID: \"51dffd76-5c30-440c-85ed-d81221663833\") " pod="kube-system/global-pull-secret-syncer-j4jzv"
Nov 03 20:04:27.076252 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.076118    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f7d30630-43af-4157-b90f-d7eddef25085-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f7d30630-43af-4157-b90f-d7eddef25085\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:04:27.076252 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.076137    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/d5d14caa-6661-4146-b888-59c3518216d4-metrics-server-audit-profiles\") pod \"metrics-server-654b7fcdc8-l6l9f\" (UID: \"d5d14caa-6661-4146-b888-59c3518216d4\") " pod="openshift-monitoring/metrics-server-654b7fcdc8-l6l9f"
Nov 03 20:04:27.076252 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.076217    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/682463ee-d9c9-4e9b-82b7-f1116898debf-system-cni-dir\") pod \"multus-h6p95\" (UID: \"682463ee-d9c9-4e9b-82b7-f1116898debf\") " pod="openshift-multus/multus-h6p95"
Nov 03 20:04:27.076463 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.076285    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f7d30630-43af-4157-b90f-d7eddef25085-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f7d30630-43af-4157-b90f-d7eddef25085\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:04:27.076463 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.076307    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcxdr\" (UniqueName: \"kubernetes.io/projected/d71eaa2a-9659-4827-9dfc-c1fce02565cb-kube-api-access-zcxdr\") pod \"network-check-target-6dfr5\" (UID: \"d71eaa2a-9659-4827-9dfc-c1fce02565cb\") " pod="openshift-network-diagnostics/network-check-target-6dfr5"
Nov 03 20:04:27.076463 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.076326    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9126a982-59c9-4cec-a665-5f1bc0ae3389-systemd-units\") pod \"ovnkube-node-6sv7j\" (UID: \"9126a982-59c9-4cec-a665-5f1bc0ae3389\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sv7j"
Nov 03 20:04:27.076463 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.076353    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/51dffd76-5c30-440c-85ed-d81221663833-original-pull-secret\") pod \"global-pull-secret-syncer-j4jzv\" (UID: \"51dffd76-5c30-440c-85ed-d81221663833\") " pod="kube-system/global-pull-secret-syncer-j4jzv"
Nov 03 20:04:27.076463 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.076405    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3f681106-32d4-4550-a00d-1f9c52060b57-cert\") pod \"ingress-canary-5jnts\" (UID: \"3f681106-32d4-4550-a00d-1f9c52060b57\") " pod="openshift-ingress-canary/ingress-canary-5jnts"
Nov 03 20:04:27.076463 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.076444    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2fd3c650-2f4c-404e-ab06-c50612af72ce-config-volume\") pod \"dns-default-7zl2f\" (UID: \"2fd3c650-2f4c-404e-ab06-c50612af72ce\") " pod="openshift-dns/dns-default-7zl2f"
Nov 03 20:04:27.076706 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.076475    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9126a982-59c9-4cec-a665-5f1bc0ae3389-host-run-netns\") pod \"ovnkube-node-6sv7j\" (UID: \"9126a982-59c9-4cec-a665-5f1bc0ae3389\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sv7j"
Nov 03 20:04:27.076706 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.076493    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f7d30630-43af-4157-b90f-d7eddef25085-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f7d30630-43af-4157-b90f-d7eddef25085\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:04:27.076706 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.076503    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9126a982-59c9-4cec-a665-5f1bc0ae3389-run-systemd\") pod \"ovnkube-node-6sv7j\" (UID: \"9126a982-59c9-4cec-a665-5f1bc0ae3389\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sv7j"
Nov 03 20:04:27.076706 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.076529    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f21cf70e-682f-490c-bda5-3954df150487-etc-modprobe-d\") pod \"tuned-4dmkq\" (UID: \"f21cf70e-682f-490c-bda5-3954df150487\") " pod="openshift-cluster-node-tuning-operator/tuned-4dmkq"
Nov 03 20:04:27.076706 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.076589    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f21cf70e-682f-490c-bda5-3954df150487-etc-systemd\") pod \"tuned-4dmkq\" (UID: \"f21cf70e-682f-490c-bda5-3954df150487\") " pod="openshift-cluster-node-tuning-operator/tuned-4dmkq"
Nov 03 20:04:27.076706 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.076640    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f21cf70e-682f-490c-bda5-3954df150487-etc-tuned\") pod \"tuned-4dmkq\" (UID: \"f21cf70e-682f-490c-bda5-3954df150487\") " pod="openshift-cluster-node-tuning-operator/tuned-4dmkq"
Nov 03 20:04:27.076706 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.076705    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3f681106-32d4-4550-a00d-1f9c52060b57-cert\") pod \"ingress-canary-5jnts\" (UID: \"3f681106-32d4-4550-a00d-1f9c52060b57\") " pod="openshift-ingress-canary/ingress-canary-5jnts"
Nov 03 20:04:27.077087 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.076734    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f21cf70e-682f-490c-bda5-3954df150487-etc-tuned\") pod \"tuned-4dmkq\" (UID: \"f21cf70e-682f-490c-bda5-3954df150487\") " pod="openshift-cluster-node-tuning-operator/tuned-4dmkq"
Nov 03 20:04:27.077087 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.076738    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/51dffd76-5c30-440c-85ed-d81221663833-original-pull-secret\") pod \"global-pull-secret-syncer-j4jzv\" (UID: \"51dffd76-5c30-440c-85ed-d81221663833\") " pod="kube-system/global-pull-secret-syncer-j4jzv"
Nov 03 20:04:27.077087 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.076738    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f21cf70e-682f-490c-bda5-3954df150487-host\") pod \"tuned-4dmkq\" (UID: \"f21cf70e-682f-490c-bda5-3954df150487\") " pod="openshift-cluster-node-tuning-operator/tuned-4dmkq"
Nov 03 20:04:27.077087 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.076826    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7hn2\" (UniqueName: \"kubernetes.io/projected/59f6a4d6-afe0-4c96-a54a-d8e6b5c66a38-kube-api-access-w7hn2\") pod \"thanos-querier-5bd4bfb68d-9449x\" (UID: \"59f6a4d6-afe0-4c96-a54a-d8e6b5c66a38\") " pod="openshift-monitoring/thanos-querier-5bd4bfb68d-9449x"
Nov 03 20:04:27.077087 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.076857    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/d5d14caa-6661-4146-b888-59c3518216d4-audit-log\") pod \"metrics-server-654b7fcdc8-l6l9f\" (UID: \"d5d14caa-6661-4146-b888-59c3518216d4\") " pod="openshift-monitoring/metrics-server-654b7fcdc8-l6l9f"
Nov 03 20:04:27.077087 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.076893    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/716cc9cf-d26a-481b-8114-169c6104ce56-hosts-file\") pod \"node-resolver-t6dqw\" (UID: \"716cc9cf-d26a-481b-8114-169c6104ce56\") " pod="openshift-dns/node-resolver-t6dqw"
Nov 03 20:04:27.077087 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.076923    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/74d4808e-24ce-4c47-b4ce-e1188eeea6f1-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-j46kr\" (UID: \"74d4808e-24ce-4c47-b4ce-e1188eeea6f1\") " pod="openshift-monitoring/node-exporter-j46kr"
Nov 03 20:04:27.077087 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.076949    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9126a982-59c9-4cec-a665-5f1bc0ae3389-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6sv7j\" (UID: \"9126a982-59c9-4cec-a665-5f1bc0ae3389\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sv7j"
Nov 03 20:04:27.077087 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.076978    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/65008653-586d-4a76-a53f-7efe1f6d839b-device-dir\") pod \"aws-ebs-csi-driver-node-9f5s5\" (UID: \"65008653-586d-4a76-a53f-7efe1f6d839b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9f5s5"
Nov 03 20:04:27.077087 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.076989    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/d5d14caa-6661-4146-b888-59c3518216d4-audit-log\") pod \"metrics-server-654b7fcdc8-l6l9f\" (UID: \"d5d14caa-6661-4146-b888-59c3518216d4\") " pod="openshift-monitoring/metrics-server-654b7fcdc8-l6l9f"
Nov 03 20:04:27.077087 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.077003    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f547t\" (UniqueName: \"kubernetes.io/projected/74d4808e-24ce-4c47-b4ce-e1188eeea6f1-kube-api-access-f547t\") pod \"node-exporter-j46kr\" (UID: \"74d4808e-24ce-4c47-b4ce-e1188eeea6f1\") " pod="openshift-monitoring/node-exporter-j46kr"
Nov 03 20:04:27.077087 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.077052    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cb9bef9b-c7cd-410e-8c1e-1b27474daf81-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-rwhnf\" (UID: \"cb9bef9b-c7cd-410e-8c1e-1b27474daf81\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-rwhnf"
Nov 03 20:04:27.077087 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.077079    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4g29t\" (UniqueName: \"kubernetes.io/projected/ffe074b1-e2a0-403b-9f81-581fdb4d2775-kube-api-access-4g29t\") pod \"kube-state-metrics-5dc8d7d89b-q8b6s\" (UID: \"ffe074b1-e2a0-403b-9f81-581fdb4d2775\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-q8b6s"
Nov 03 20:04:27.077659 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.077104    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/682463ee-d9c9-4e9b-82b7-f1116898debf-host-var-lib-kubelet\") pod \"multus-h6p95\" (UID: \"682463ee-d9c9-4e9b-82b7-f1116898debf\") " pod="openshift-multus/multus-h6p95"
Nov 03 20:04:27.077659 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.077118    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/74d4808e-24ce-4c47-b4ce-e1188eeea6f1-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-j46kr\" (UID: \"74d4808e-24ce-4c47-b4ce-e1188eeea6f1\") " pod="openshift-monitoring/node-exporter-j46kr"
Nov 03 20:04:27.077659 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.077129    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-df59d\" (UniqueName: \"kubernetes.io/projected/3f681106-32d4-4550-a00d-1f9c52060b57-kube-api-access-df59d\") pod \"ingress-canary-5jnts\" (UID: \"3f681106-32d4-4550-a00d-1f9c52060b57\") " pod="openshift-ingress-canary/ingress-canary-5jnts"
Nov 03 20:04:27.077659 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.077186    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f7d30630-43af-4157-b90f-d7eddef25085-config-volume\") pod \"alertmanager-main-0\" (UID: \"f7d30630-43af-4157-b90f-d7eddef25085\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:04:27.077659 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.077214    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/682463ee-d9c9-4e9b-82b7-f1116898debf-hostroot\") pod \"multus-h6p95\" (UID: \"682463ee-d9c9-4e9b-82b7-f1116898debf\") " pod="openshift-multus/multus-h6p95"
Nov 03 20:04:27.077659 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.077238    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9126a982-59c9-4cec-a665-5f1bc0ae3389-log-socket\") pod \"ovnkube-node-6sv7j\" (UID: \"9126a982-59c9-4cec-a665-5f1bc0ae3389\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sv7j"
Nov 03 20:04:27.077659 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.077276    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9126a982-59c9-4cec-a665-5f1bc0ae3389-host-run-ovn-kubernetes\") pod \"ovnkube-node-6sv7j\" (UID: \"9126a982-59c9-4cec-a665-5f1bc0ae3389\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sv7j"
Nov 03 20:04:27.077659 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.077300    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9126a982-59c9-4cec-a665-5f1bc0ae3389-host-cni-netd\") pod \"ovnkube-node-6sv7j\" (UID: \"9126a982-59c9-4cec-a665-5f1bc0ae3389\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sv7j"
Nov 03 20:04:27.077659 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.077367    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/ffe074b1-e2a0-403b-9f81-581fdb4d2775-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-q8b6s\" (UID: \"ffe074b1-e2a0-403b-9f81-581fdb4d2775\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-q8b6s"
Nov 03 20:04:27.077659 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.077394    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/884cbb80-b3f8-4f90-bd55-328d17943cf3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8bwhz\" (UID: \"884cbb80-b3f8-4f90-bd55-328d17943cf3\") " pod="openshift-insights/insights-runtime-extractor-8bwhz"
Nov 03 20:04:27.077659 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.077421    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/65008653-586d-4a76-a53f-7efe1f6d839b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9f5s5\" (UID: \"65008653-586d-4a76-a53f-7efe1f6d839b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9f5s5"
Nov 03 20:04:27.077659 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.077440    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/74d4808e-24ce-4c47-b4ce-e1188eeea6f1-node-exporter-tls\") pod \"node-exporter-j46kr\" (UID: \"74d4808e-24ce-4c47-b4ce-e1188eeea6f1\") " pod="openshift-monitoring/node-exporter-j46kr"
Nov 03 20:04:27.077659 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.077451    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f7d30630-43af-4157-b90f-d7eddef25085-config-volume\") pod \"alertmanager-main-0\" (UID: \"f7d30630-43af-4157-b90f-d7eddef25085\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:04:27.077659 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.077459    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ffe074b1-e2a0-403b-9f81-581fdb4d2775-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-q8b6s\" (UID: \"ffe074b1-e2a0-403b-9f81-581fdb4d2775\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-q8b6s"
Nov 03 20:04:27.077659 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.077494    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/db895326-506d-479d-a526-6bf71be7f967-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mhtmw\" (UID: \"db895326-506d-479d-a526-6bf71be7f967\") " pod="openshift-multus/multus-additional-cni-plugins-mhtmw"
Nov 03 20:04:27.077659 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.077526    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/db895326-506d-479d-a526-6bf71be7f967-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mhtmw\" (UID: \"db895326-506d-479d-a526-6bf71be7f967\") " pod="openshift-multus/multus-additional-cni-plugins-mhtmw"
Nov 03 20:04:27.077659 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.077561    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/59f6a4d6-afe0-4c96-a54a-d8e6b5c66a38-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5bd4bfb68d-9449x\" (UID: \"59f6a4d6-afe0-4c96-a54a-d8e6b5c66a38\") " pod="openshift-monitoring/thanos-querier-5bd4bfb68d-9449x"
Nov 03 20:04:27.077659 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.077593    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/884cbb80-b3f8-4f90-bd55-328d17943cf3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8bwhz\" (UID: \"884cbb80-b3f8-4f90-bd55-328d17943cf3\") " pod="openshift-insights/insights-runtime-extractor-8bwhz"
Nov 03 20:04:27.077659 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.077604    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56z7v\" (UniqueName: \"kubernetes.io/projected/2fd3c650-2f4c-404e-ab06-c50612af72ce-kube-api-access-56z7v\") pod \"dns-default-7zl2f\" (UID: \"2fd3c650-2f4c-404e-ab06-c50612af72ce\") " pod="openshift-dns/dns-default-7zl2f"
Nov 03 20:04:27.077659 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.077632    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2073a8c8-6f07-4817-b7b3-468b327e2712-host-slash\") pod \"iptables-alerter-9s4wk\" (UID: \"2073a8c8-6f07-4817-b7b3-468b327e2712\") " pod="openshift-network-operator/iptables-alerter-9s4wk"
Nov 03 20:04:27.077659 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.077652    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ndt8\" (UniqueName: \"kubernetes.io/projected/cb9bef9b-c7cd-410e-8c1e-1b27474daf81-kube-api-access-4ndt8\") pod \"openshift-state-metrics-7bd7489957-rwhnf\" (UID: \"cb9bef9b-c7cd-410e-8c1e-1b27474daf81\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-rwhnf"
Nov 03 20:04:27.077659 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.077668    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/74d4808e-24ce-4c47-b4ce-e1188eeea6f1-node-exporter-tls\") pod \"node-exporter-j46kr\" (UID: \"74d4808e-24ce-4c47-b4ce-e1188eeea6f1\") " pod="openshift-monitoring/node-exporter-j46kr"
Nov 03 20:04:27.078712 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.077680    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1d4ce60c-e058-4c29-99e8-5f4f6eead744-serviceca\") pod \"node-ca-zzt8t\" (UID: \"1d4ce60c-e058-4c29-99e8-5f4f6eead744\") " pod="openshift-image-registry/node-ca-zzt8t"
Nov 03 20:04:27.078712 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.077701    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9126a982-59c9-4cec-a665-5f1bc0ae3389-ovnkube-config\") pod \"ovnkube-node-6sv7j\" (UID: \"9126a982-59c9-4cec-a665-5f1bc0ae3389\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sv7j"
Nov 03 20:04:27.078712 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.077715    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f21cf70e-682f-490c-bda5-3954df150487-tmp\") pod \"tuned-4dmkq\" (UID: \"f21cf70e-682f-490c-bda5-3954df150487\") " pod="openshift-cluster-node-tuning-operator/tuned-4dmkq"
Nov 03 20:04:27.078712 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.077730    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c92a5d0-b34f-46a1-8f8f-a9ae3cb8d5a5-metrics-certs\") pod \"network-metrics-daemon-c947w\" (UID: \"4c92a5d0-b34f-46a1-8f8f-a9ae3cb8d5a5\") " pod="openshift-multus/network-metrics-daemon-c947w"
Nov 03 20:04:27.078712 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.077756    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s92q2\" (UniqueName: \"kubernetes.io/projected/1d4ce60c-e058-4c29-99e8-5f4f6eead744-kube-api-access-s92q2\") pod \"node-ca-zzt8t\" (UID: \"1d4ce60c-e058-4c29-99e8-5f4f6eead744\") " pod="openshift-image-registry/node-ca-zzt8t"
Nov 03 20:04:27.078712 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.077798    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/59f6a4d6-afe0-4c96-a54a-d8e6b5c66a38-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5bd4bfb68d-9449x\" (UID: \"59f6a4d6-afe0-4c96-a54a-d8e6b5c66a38\") " pod="openshift-monitoring/thanos-querier-5bd4bfb68d-9449x"
Nov 03 20:04:27.078712 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.077799    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/682463ee-d9c9-4e9b-82b7-f1116898debf-host-var-lib-cni-bin\") pod \"multus-h6p95\" (UID: \"682463ee-d9c9-4e9b-82b7-f1116898debf\") " pod="openshift-multus/multus-h6p95"
Nov 03 20:04:27.078712 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.077851    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f21cf70e-682f-490c-bda5-3954df150487-tmp\") pod \"tuned-4dmkq\" (UID: \"f21cf70e-682f-490c-bda5-3954df150487\") " pod="openshift-cluster-node-tuning-operator/tuned-4dmkq"
Nov 03 20:04:27.078712 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.077937    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrgcs\" (UniqueName: \"kubernetes.io/projected/2073a8c8-6f07-4817-b7b3-468b327e2712-kube-api-access-jrgcs\") pod \"iptables-alerter-9s4wk\" (UID: \"2073a8c8-6f07-4817-b7b3-468b327e2712\") " pod="openshift-network-operator/iptables-alerter-9s4wk"
Nov 03 20:04:27.078712 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.077968    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9126a982-59c9-4cec-a665-5f1bc0ae3389-ovnkube-script-lib\") pod \"ovnkube-node-6sv7j\" (UID: \"9126a982-59c9-4cec-a665-5f1bc0ae3389\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sv7j"
Nov 03 20:04:27.078712 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.077937    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9126a982-59c9-4cec-a665-5f1bc0ae3389-ovnkube-config\") pod \"ovnkube-node-6sv7j\" (UID: \"9126a982-59c9-4cec-a665-5f1bc0ae3389\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sv7j"
Nov 03 20:04:27.078712 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.077986    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/59f6a4d6-afe0-4c96-a54a-d8e6b5c66a38-secret-thanos-querier-tls\") pod \"thanos-querier-5bd4bfb68d-9449x\" (UID: \"59f6a4d6-afe0-4c96-a54a-d8e6b5c66a38\") " pod="openshift-monitoring/thanos-querier-5bd4bfb68d-9449x"
Nov 03 20:04:27.078712 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.078006    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fd3c650-2f4c-404e-ab06-c50612af72ce-tmp-dir\") pod \"dns-default-7zl2f\" (UID: \"2fd3c650-2f4c-404e-ab06-c50612af72ce\") " pod="openshift-dns/dns-default-7zl2f"
Nov 03 20:04:27.078712 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.077985    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c92a5d0-b34f-46a1-8f8f-a9ae3cb8d5a5-metrics-certs\") pod \"network-metrics-daemon-c947w\" (UID: \"4c92a5d0-b34f-46a1-8f8f-a9ae3cb8d5a5\") " pod="openshift-multus/network-metrics-daemon-c947w"
Nov 03 20:04:27.078712 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.078051    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89vxf\" (UniqueName: \"kubernetes.io/projected/9126a982-59c9-4cec-a665-5f1bc0ae3389-kube-api-access-89vxf\") pod \"ovnkube-node-6sv7j\" (UID: \"9126a982-59c9-4cec-a665-5f1bc0ae3389\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sv7j"
Nov 03 20:04:27.078712 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.078113    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/74d4808e-24ce-4c47-b4ce-e1188eeea6f1-root\") pod \"node-exporter-j46kr\" (UID: \"74d4808e-24ce-4c47-b4ce-e1188eeea6f1\") " pod="openshift-monitoring/node-exporter-j46kr"
Nov 03 20:04:27.078712 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.078132    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fd3c650-2f4c-404e-ab06-c50612af72ce-tmp-dir\") pod \"dns-default-7zl2f\" (UID: \"2fd3c650-2f4c-404e-ab06-c50612af72ce\") " pod="openshift-dns/dns-default-7zl2f"
Nov 03 20:04:27.078712 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.078145    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f21cf70e-682f-490c-bda5-3954df150487-run\") pod \"tuned-4dmkq\" (UID: \"f21cf70e-682f-490c-bda5-3954df150487\") " pod="openshift-cluster-node-tuning-operator/tuned-4dmkq"
Nov 03 20:04:27.078712 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.078201    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlzhf\" (UniqueName: \"kubernetes.io/projected/db895326-506d-479d-a526-6bf71be7f967-kube-api-access-vlzhf\") pod \"multus-additional-cni-plugins-mhtmw\" (UID: \"db895326-506d-479d-a526-6bf71be7f967\") " pod="openshift-multus/multus-additional-cni-plugins-mhtmw"
Nov 03 20:04:27.078712 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.078221    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/59f6a4d6-afe0-4c96-a54a-d8e6b5c66a38-secret-thanos-querier-tls\") pod \"thanos-querier-5bd4bfb68d-9449x\" (UID: \"59f6a4d6-afe0-4c96-a54a-d8e6b5c66a38\") " pod="openshift-monitoring/thanos-querier-5bd4bfb68d-9449x"
Nov 03 20:04:27.078712 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.078207    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9126a982-59c9-4cec-a665-5f1bc0ae3389-ovnkube-script-lib\") pod \"ovnkube-node-6sv7j\" (UID: \"9126a982-59c9-4cec-a665-5f1bc0ae3389\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sv7j"
Nov 03 20:04:27.078712 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.078297    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2fd3c650-2f4c-404e-ab06-c50612af72ce-metrics-tls\") pod \"dns-default-7zl2f\" (UID: \"2fd3c650-2f4c-404e-ab06-c50612af72ce\") " pod="openshift-dns/dns-default-7zl2f"
Nov 03 20:04:27.078712 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.078332    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/74d4808e-24ce-4c47-b4ce-e1188eeea6f1-metrics-client-ca\") pod \"node-exporter-j46kr\" (UID: \"74d4808e-24ce-4c47-b4ce-e1188eeea6f1\") " pod="openshift-monitoring/node-exporter-j46kr"
Nov 03 20:04:27.078712 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.078384    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f7d30630-43af-4157-b90f-d7eddef25085-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f7d30630-43af-4157-b90f-d7eddef25085\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:04:27.078712 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.078390    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1d4ce60c-e058-4c29-99e8-5f4f6eead744-serviceca\") pod \"node-ca-zzt8t\" (UID: \"1d4ce60c-e058-4c29-99e8-5f4f6eead744\") " pod="openshift-image-registry/node-ca-zzt8t"
Nov 03 20:04:27.078712 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.078401    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ffe074b1-e2a0-403b-9f81-581fdb4d2775-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-q8b6s\" (UID: \"ffe074b1-e2a0-403b-9f81-581fdb4d2775\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-q8b6s"
Nov 03 20:04:27.078712 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.078419    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/682463ee-d9c9-4e9b-82b7-f1116898debf-cnibin\") pod \"multus-h6p95\" (UID: \"682463ee-d9c9-4e9b-82b7-f1116898debf\") " pod="openshift-multus/multus-h6p95"
Nov 03 20:04:27.078712 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.078465    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f7d30630-43af-4157-b90f-d7eddef25085-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f7d30630-43af-4157-b90f-d7eddef25085\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:04:27.078712 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.078475    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2fd3c650-2f4c-404e-ab06-c50612af72ce-metrics-tls\") pod \"dns-default-7zl2f\" (UID: \"2fd3c650-2f4c-404e-ab06-c50612af72ce\") " pod="openshift-dns/dns-default-7zl2f"
Nov 03 20:04:27.078712 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.078489    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f7d30630-43af-4157-b90f-d7eddef25085-web-config\") pod \"alertmanager-main-0\" (UID: \"f7d30630-43af-4157-b90f-d7eddef25085\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:04:27.078712 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.078515    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/65008653-586d-4a76-a53f-7efe1f6d839b-etc-selinux\") pod \"aws-ebs-csi-driver-node-9f5s5\" (UID: \"65008653-586d-4a76-a53f-7efe1f6d839b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9f5s5"
Nov 03 20:04:27.078712 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.078534    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8psbp\" (UniqueName: \"kubernetes.io/projected/65008653-586d-4a76-a53f-7efe1f6d839b-kube-api-access-8psbp\") pod \"aws-ebs-csi-driver-node-9f5s5\" (UID: \"65008653-586d-4a76-a53f-7efe1f6d839b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9f5s5"
Nov 03 20:04:27.078712 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.078549    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9126a982-59c9-4cec-a665-5f1bc0ae3389-node-log\") pod \"ovnkube-node-6sv7j\" (UID: \"9126a982-59c9-4cec-a665-5f1bc0ae3389\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sv7j"
Nov 03 20:04:27.078712 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.078565    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/d5d14caa-6661-4146-b888-59c3518216d4-secret-metrics-server-tls\") pod \"metrics-server-654b7fcdc8-l6l9f\" (UID: \"d5d14caa-6661-4146-b888-59c3518216d4\") " pod="openshift-monitoring/metrics-server-654b7fcdc8-l6l9f"
Nov 03 20:04:27.078712 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.078613    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f7d30630-43af-4157-b90f-d7eddef25085-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f7d30630-43af-4157-b90f-d7eddef25085\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:04:27.078712 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.078623    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ffe074b1-e2a0-403b-9f81-581fdb4d2775-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-q8b6s\" (UID: \"ffe074b1-e2a0-403b-9f81-581fdb4d2775\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-q8b6s"
Nov 03 20:04:27.078712 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.078642    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9126a982-59c9-4cec-a665-5f1bc0ae3389-host-cni-bin\") pod \"ovnkube-node-6sv7j\" (UID: \"9126a982-59c9-4cec-a665-5f1bc0ae3389\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sv7j"
Nov 03 20:04:27.078712 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.078663    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f7d30630-43af-4157-b90f-d7eddef25085-web-config\") pod \"alertmanager-main-0\" (UID: \"f7d30630-43af-4157-b90f-d7eddef25085\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:04:27.078712 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.078695    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f21cf70e-682f-490c-bda5-3954df150487-etc-sysconfig\") pod \"tuned-4dmkq\" (UID: \"f21cf70e-682f-490c-bda5-3954df150487\") " pod="openshift-cluster-node-tuning-operator/tuned-4dmkq"
Nov 03 20:04:27.080616 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.078807    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/59f6a4d6-afe0-4c96-a54a-d8e6b5c66a38-tmp\") pod \"thanos-querier-5bd4bfb68d-9449x\" (UID: \"59f6a4d6-afe0-4c96-a54a-d8e6b5c66a38\") " pod="openshift-monitoring/thanos-querier-5bd4bfb68d-9449x"
Nov 03 20:04:27.080616 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.078830    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/d5d14caa-6661-4146-b888-59c3518216d4-secret-metrics-server-tls\") pod \"metrics-server-654b7fcdc8-l6l9f\" (UID: \"d5d14caa-6661-4146-b888-59c3518216d4\") " pod="openshift-monitoring/metrics-server-654b7fcdc8-l6l9f"
Nov 03 20:04:27.080616 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.078834    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f7d30630-43af-4157-b90f-d7eddef25085-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f7d30630-43af-4157-b90f-d7eddef25085\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:04:27.080616 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.078857    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/59f6a4d6-afe0-4c96-a54a-d8e6b5c66a38-secret-grpc-tls\") pod \"thanos-querier-5bd4bfb68d-9449x\" (UID: \"59f6a4d6-afe0-4c96-a54a-d8e6b5c66a38\") " pod="openshift-monitoring/thanos-querier-5bd4bfb68d-9449x"
Nov 03 20:04:27.080616 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.078885    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/59f6a4d6-afe0-4c96-a54a-d8e6b5c66a38-tmp\") pod \"thanos-querier-5bd4bfb68d-9449x\" (UID: \"59f6a4d6-afe0-4c96-a54a-d8e6b5c66a38\") " pod="openshift-monitoring/thanos-querier-5bd4bfb68d-9449x"
Nov 03 20:04:27.080616 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.078896    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/d5d14caa-6661-4146-b888-59c3518216d4-client-ca-bundle\") pod \"metrics-server-654b7fcdc8-l6l9f\" (UID: \"d5d14caa-6661-4146-b888-59c3518216d4\") " pod="openshift-monitoring/metrics-server-654b7fcdc8-l6l9f"
Nov 03 20:04:27.080616 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.078935    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkjpg\" (UniqueName: \"kubernetes.io/projected/682463ee-d9c9-4e9b-82b7-f1116898debf-kube-api-access-nkjpg\") pod \"multus-h6p95\" (UID: \"682463ee-d9c9-4e9b-82b7-f1116898debf\") " pod="openshift-multus/multus-h6p95"
Nov 03 20:04:27.080616 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.078962    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f7d30630-43af-4157-b90f-d7eddef25085-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f7d30630-43af-4157-b90f-d7eddef25085\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:04:27.080616 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.079022    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f21cf70e-682f-490c-bda5-3954df150487-etc-kubernetes\") pod \"tuned-4dmkq\" (UID: \"f21cf70e-682f-490c-bda5-3954df150487\") " pod="openshift-cluster-node-tuning-operator/tuned-4dmkq"
Nov 03 20:04:27.080616 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.079051    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/d5d14caa-6661-4146-b888-59c3518216d4-secret-metrics-server-client-certs\") pod \"metrics-server-654b7fcdc8-l6l9f\" (UID: \"d5d14caa-6661-4146-b888-59c3518216d4\") " pod="openshift-monitoring/metrics-server-654b7fcdc8-l6l9f"
Nov 03 20:04:27.080616 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.079092    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f7d30630-43af-4157-b90f-d7eddef25085-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f7d30630-43af-4157-b90f-d7eddef25085\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:04:27.080616 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.079125    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f7d30630-43af-4157-b90f-d7eddef25085-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f7d30630-43af-4157-b90f-d7eddef25085\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:04:27.080616 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.079135    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/59f6a4d6-afe0-4c96-a54a-d8e6b5c66a38-secret-grpc-tls\") pod \"thanos-querier-5bd4bfb68d-9449x\" (UID: \"59f6a4d6-afe0-4c96-a54a-d8e6b5c66a38\") " pod="openshift-monitoring/thanos-querier-5bd4bfb68d-9449x"
Nov 03 20:04:27.080616 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.079151    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9126a982-59c9-4cec-a665-5f1bc0ae3389-etc-openvswitch\") pod \"ovnkube-node-6sv7j\" (UID: \"9126a982-59c9-4cec-a665-5f1bc0ae3389\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sv7j"
Nov 03 20:04:27.080616 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.079138    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/d5d14caa-6661-4146-b888-59c3518216d4-client-ca-bundle\") pod \"metrics-server-654b7fcdc8-l6l9f\" (UID: \"d5d14caa-6661-4146-b888-59c3518216d4\") " pod="openshift-monitoring/metrics-server-654b7fcdc8-l6l9f"
Nov 03 20:04:27.080616 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.079183    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5d14caa-6661-4146-b888-59c3518216d4-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-654b7fcdc8-l6l9f\" (UID: \"d5d14caa-6661-4146-b888-59c3518216d4\") " pod="openshift-monitoring/metrics-server-654b7fcdc8-l6l9f"
Nov 03 20:04:27.080616 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.079209    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2073a8c8-6f07-4817-b7b3-468b327e2712-iptables-alerter-script\") pod \"iptables-alerter-9s4wk\" (UID: \"2073a8c8-6f07-4817-b7b3-468b327e2712\") " pod="openshift-network-operator/iptables-alerter-9s4wk"
Nov 03 20:04:27.080616 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.079216    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f7d30630-43af-4157-b90f-d7eddef25085-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f7d30630-43af-4157-b90f-d7eddef25085\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:04:27.080616 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.079231    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9126a982-59c9-4cec-a665-5f1bc0ae3389-run-ovn\") pod \"ovnkube-node-6sv7j\" (UID: \"9126a982-59c9-4cec-a665-5f1bc0ae3389\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sv7j"
Nov 03 20:04:27.080616 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.079282    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f21cf70e-682f-490c-bda5-3954df150487-sys\") pod \"tuned-4dmkq\" (UID: \"f21cf70e-682f-490c-bda5-3954df150487\") " pod="openshift-cluster-node-tuning-operator/tuned-4dmkq"
Nov 03 20:04:27.080616 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.079300    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f21cf70e-682f-490c-bda5-3954df150487-lib-modules\") pod \"tuned-4dmkq\" (UID: \"f21cf70e-682f-490c-bda5-3954df150487\") " pod="openshift-cluster-node-tuning-operator/tuned-4dmkq"
Nov 03 20:04:27.080616 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.079316    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/d5d14caa-6661-4146-b888-59c3518216d4-secret-metrics-server-client-certs\") pod \"metrics-server-654b7fcdc8-l6l9f\" (UID: \"d5d14caa-6661-4146-b888-59c3518216d4\") " pod="openshift-monitoring/metrics-server-654b7fcdc8-l6l9f"
Nov 03 20:04:27.080616 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.079340    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f7d30630-43af-4157-b90f-d7eddef25085-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f7d30630-43af-4157-b90f-d7eddef25085\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:04:27.080616 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.079355    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/74d4808e-24ce-4c47-b4ce-e1188eeea6f1-node-exporter-textfile\") pod \"node-exporter-j46kr\" (UID: \"74d4808e-24ce-4c47-b4ce-e1188eeea6f1\") " pod="openshift-monitoring/node-exporter-j46kr"
Nov 03 20:04:27.080616 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.079377    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2073a8c8-6f07-4817-b7b3-468b327e2712-iptables-alerter-script\") pod \"iptables-alerter-9s4wk\" (UID: \"2073a8c8-6f07-4817-b7b3-468b327e2712\") " pod="openshift-network-operator/iptables-alerter-9s4wk"
Nov 03 20:04:27.080616 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.079389    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f7d30630-43af-4157-b90f-d7eddef25085-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f7d30630-43af-4157-b90f-d7eddef25085\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:04:27.080616 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.079401    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/74d4808e-24ce-4c47-b4ce-e1188eeea6f1-node-exporter-textfile\") pod \"node-exporter-j46kr\" (UID: \"74d4808e-24ce-4c47-b4ce-e1188eeea6f1\") " pod="openshift-monitoring/node-exporter-j46kr"
Nov 03 20:04:27.080616 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.079422    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9126a982-59c9-4cec-a665-5f1bc0ae3389-var-lib-openvswitch\") pod \"ovnkube-node-6sv7j\" (UID: \"9126a982-59c9-4cec-a665-5f1bc0ae3389\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sv7j"
Nov 03 20:04:27.080616 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.079451    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/59f6a4d6-afe0-4c96-a54a-d8e6b5c66a38-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5bd4bfb68d-9449x\" (UID: \"59f6a4d6-afe0-4c96-a54a-d8e6b5c66a38\") " pod="openshift-monitoring/thanos-querier-5bd4bfb68d-9449x"
Nov 03 20:04:27.080616 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.079482    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f7d30630-43af-4157-b90f-d7eddef25085-config-out\") pod \"alertmanager-main-0\" (UID: \"f7d30630-43af-4157-b90f-d7eddef25085\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:04:27.080616 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.079514    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ffe074b1-e2a0-403b-9f81-581fdb4d2775-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-q8b6s\" (UID: \"ffe074b1-e2a0-403b-9f81-581fdb4d2775\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-q8b6s"
Nov 03 20:04:27.080616 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.079539    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/59f6a4d6-afe0-4c96-a54a-d8e6b5c66a38-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5bd4bfb68d-9449x\" (UID: \"59f6a4d6-afe0-4c96-a54a-d8e6b5c66a38\") " pod="openshift-monitoring/thanos-querier-5bd4bfb68d-9449x"
Nov 03 20:04:27.080616 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.079567    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/51dffd76-5c30-440c-85ed-d81221663833-dbus\") pod \"global-pull-secret-syncer-j4jzv\" (UID: \"51dffd76-5c30-440c-85ed-d81221663833\") " pod="kube-system/global-pull-secret-syncer-j4jzv"
Nov 03 20:04:27.080616 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.079569    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f7d30630-43af-4157-b90f-d7eddef25085-config-out\") pod \"alertmanager-main-0\" (UID: \"f7d30630-43af-4157-b90f-d7eddef25085\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:04:27.080616 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.079540    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f7d30630-43af-4157-b90f-d7eddef25085-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f7d30630-43af-4157-b90f-d7eddef25085\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:04:27.080616 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.079599    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f7d30630-43af-4157-b90f-d7eddef25085-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f7d30630-43af-4157-b90f-d7eddef25085\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:04:27.080616 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.079650    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/db895326-506d-479d-a526-6bf71be7f967-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mhtmw\" (UID: \"db895326-506d-479d-a526-6bf71be7f967\") " pod="openshift-multus/multus-additional-cni-plugins-mhtmw"
Nov 03 20:04:27.080616 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.079694    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/884cbb80-b3f8-4f90-bd55-328d17943cf3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8bwhz\" (UID: \"884cbb80-b3f8-4f90-bd55-328d17943cf3\") " pod="openshift-insights/insights-runtime-extractor-8bwhz"
Nov 03 20:04:27.080616 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.079698    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/59f6a4d6-afe0-4c96-a54a-d8e6b5c66a38-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5bd4bfb68d-9449x\" (UID: \"59f6a4d6-afe0-4c96-a54a-d8e6b5c66a38\") " pod="openshift-monitoring/thanos-querier-5bd4bfb68d-9449x"
Nov 03 20:04:27.080616 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.079727    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/59f6a4d6-afe0-4c96-a54a-d8e6b5c66a38-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5bd4bfb68d-9449x\" (UID: \"59f6a4d6-afe0-4c96-a54a-d8e6b5c66a38\") " pod="openshift-monitoring/thanos-querier-5bd4bfb68d-9449x"
Nov 03 20:04:27.080616 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.079734    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szdn8\" (UniqueName: \"kubernetes.io/projected/884cbb80-b3f8-4f90-bd55-328d17943cf3-kube-api-access-szdn8\") pod \"insights-runtime-extractor-8bwhz\" (UID: \"884cbb80-b3f8-4f90-bd55-328d17943cf3\") " pod="openshift-insights/insights-runtime-extractor-8bwhz"
Nov 03 20:04:27.080616 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.079765    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/682463ee-d9c9-4e9b-82b7-f1116898debf-multus-socket-dir-parent\") pod \"multus-h6p95\" (UID: \"682463ee-d9c9-4e9b-82b7-f1116898debf\") " pod="openshift-multus/multus-h6p95"
Nov 03 20:04:27.080616 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.079801    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f7d30630-43af-4157-b90f-d7eddef25085-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f7d30630-43af-4157-b90f-d7eddef25085\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:04:27.080616 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.079824    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/682463ee-d9c9-4e9b-82b7-f1116898debf-host-var-lib-cni-multus\") pod \"multus-h6p95\" (UID: \"682463ee-d9c9-4e9b-82b7-f1116898debf\") " pod="openshift-multus/multus-h6p95"
Nov 03 20:04:27.080616 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.079836    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ffe074b1-e2a0-403b-9f81-581fdb4d2775-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-q8b6s\" (UID: \"ffe074b1-e2a0-403b-9f81-581fdb4d2775\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-q8b6s"
Nov 03 20:04:27.080616 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.079855    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nd2w\" (UniqueName: \"kubernetes.io/projected/4c92a5d0-b34f-46a1-8f8f-a9ae3cb8d5a5-kube-api-access-6nd2w\") pod \"network-metrics-daemon-c947w\" (UID: \"4c92a5d0-b34f-46a1-8f8f-a9ae3cb8d5a5\") " pod="openshift-multus/network-metrics-daemon-c947w"
Nov 03 20:04:27.080616 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.079883    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wq4p\" (UniqueName: \"kubernetes.io/projected/d5d14caa-6661-4146-b888-59c3518216d4-kube-api-access-8wq4p\") pod \"metrics-server-654b7fcdc8-l6l9f\" (UID: \"d5d14caa-6661-4146-b888-59c3518216d4\") " pod="openshift-monitoring/metrics-server-654b7fcdc8-l6l9f"
Nov 03 20:04:27.080616 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.079909    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/682463ee-d9c9-4e9b-82b7-f1116898debf-host-run-k8s-cni-cncf-io\") pod \"multus-h6p95\" (UID: \"682463ee-d9c9-4e9b-82b7-f1116898debf\") " pod="openshift-multus/multus-h6p95"
Nov 03 20:04:27.080616 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.079944    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/682463ee-d9c9-4e9b-82b7-f1116898debf-multus-daemon-config\") pod \"multus-h6p95\" (UID: \"682463ee-d9c9-4e9b-82b7-f1116898debf\") " pod="openshift-multus/multus-h6p95"
Nov 03 20:04:27.080616 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.079972    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9126a982-59c9-4cec-a665-5f1bc0ae3389-run-openvswitch\") pod \"ovnkube-node-6sv7j\" (UID: \"9126a982-59c9-4cec-a665-5f1bc0ae3389\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sv7j"
Nov 03 20:04:27.080616 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.079995    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f21cf70e-682f-490c-bda5-3954df150487-var-lib-kubelet\") pod \"tuned-4dmkq\" (UID: \"f21cf70e-682f-490c-bda5-3954df150487\") " pod="openshift-cluster-node-tuning-operator/tuned-4dmkq"
Nov 03 20:04:27.080616 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.080023    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/ffe074b1-e2a0-403b-9f81-581fdb4d2775-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-q8b6s\" (UID: \"ffe074b1-e2a0-403b-9f81-581fdb4d2775\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-q8b6s"
Nov 03 20:04:27.080616 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.080047    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/884cbb80-b3f8-4f90-bd55-328d17943cf3-crio-socket\") pod \"insights-runtime-extractor-8bwhz\" (UID: \"884cbb80-b3f8-4f90-bd55-328d17943cf3\") " pod="openshift-insights/insights-runtime-extractor-8bwhz"
Nov 03 20:04:27.080616 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.080075    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/682463ee-d9c9-4e9b-82b7-f1116898debf-etc-kubernetes\") pod \"multus-h6p95\" (UID: \"682463ee-d9c9-4e9b-82b7-f1116898debf\") " pod="openshift-multus/multus-h6p95"
Nov 03 20:04:27.080616 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.080100    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9126a982-59c9-4cec-a665-5f1bc0ae3389-env-overrides\") pod \"ovnkube-node-6sv7j\" (UID: \"9126a982-59c9-4cec-a665-5f1bc0ae3389\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sv7j"
Nov 03 20:04:27.080616 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.080112    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/ffe074b1-e2a0-403b-9f81-581fdb4d2775-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-q8b6s\" (UID: \"ffe074b1-e2a0-403b-9f81-581fdb4d2775\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-q8b6s"
Nov 03 20:04:27.080616 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.080127    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f21cf70e-682f-490c-bda5-3954df150487-etc-sysctl-d\") pod \"tuned-4dmkq\" (UID: \"f21cf70e-682f-490c-bda5-3954df150487\") " pod="openshift-cluster-node-tuning-operator/tuned-4dmkq"
Nov 03 20:04:27.080616 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.080153    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/15b5070f-d0d3-4ad8-aa1d-93cc7b96003a-agent-certs\") pod \"konnectivity-agent-5p48g\" (UID: \"15b5070f-d0d3-4ad8-aa1d-93cc7b96003a\") " pod="kube-system/konnectivity-agent-5p48g"
Nov 03 20:04:27.080616 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.080188    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/884cbb80-b3f8-4f90-bd55-328d17943cf3-data-volume\") pod \"insights-runtime-extractor-8bwhz\" (UID: \"884cbb80-b3f8-4f90-bd55-328d17943cf3\") " pod="openshift-insights/insights-runtime-extractor-8bwhz"
Nov 03 20:04:27.080616 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.080221    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/716cc9cf-d26a-481b-8114-169c6104ce56-tmp-dir\") pod \"node-resolver-t6dqw\" (UID: \"716cc9cf-d26a-481b-8114-169c6104ce56\") " pod="openshift-dns/node-resolver-t6dqw"
Nov 03 20:04:27.080616 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.080242    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9126a982-59c9-4cec-a665-5f1bc0ae3389-env-overrides\") pod \"ovnkube-node-6sv7j\" (UID: \"9126a982-59c9-4cec-a665-5f1bc0ae3389\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sv7j"
Nov 03 20:04:27.080616 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.080251    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/884cbb80-b3f8-4f90-bd55-328d17943cf3-data-volume\") pod \"insights-runtime-extractor-8bwhz\" (UID: \"884cbb80-b3f8-4f90-bd55-328d17943cf3\") " pod="openshift-insights/insights-runtime-extractor-8bwhz"
Nov 03 20:04:27.080616 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.080250    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/65008653-586d-4a76-a53f-7efe1f6d839b-socket-dir\") pod \"aws-ebs-csi-driver-node-9f5s5\" (UID: \"65008653-586d-4a76-a53f-7efe1f6d839b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9f5s5"
Nov 03 20:04:27.080616 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.080286    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/682463ee-d9c9-4e9b-82b7-f1116898debf-host-run-multus-certs\") pod \"multus-h6p95\" (UID: \"682463ee-d9c9-4e9b-82b7-f1116898debf\") " pod="openshift-multus/multus-h6p95"
Nov 03 20:04:27.080616 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.080289    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/716cc9cf-d26a-481b-8114-169c6104ce56-tmp-dir\") pod \"node-resolver-t6dqw\" (UID: \"716cc9cf-d26a-481b-8114-169c6104ce56\") " pod="openshift-dns/node-resolver-t6dqw"
Nov 03 20:04:27.080616 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.080313    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/682463ee-d9c9-4e9b-82b7-f1116898debf-multus-conf-dir\") pod \"multus-h6p95\" (UID: \"682463ee-d9c9-4e9b-82b7-f1116898debf\") " pod="openshift-multus/multus-h6p95"
Nov 03 20:04:27.080616 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.080343    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/74d4808e-24ce-4c47-b4ce-e1188eeea6f1-sys\") pod \"node-exporter-j46kr\" (UID: \"74d4808e-24ce-4c47-b4ce-e1188eeea6f1\") " pod="openshift-monitoring/node-exporter-j46kr"
Nov 03 20:04:27.080616 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.080369    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/74d4808e-24ce-4c47-b4ce-e1188eeea6f1-node-exporter-accelerators-collector-config\") pod \"node-exporter-j46kr\" (UID: \"74d4808e-24ce-4c47-b4ce-e1188eeea6f1\") " pod="openshift-monitoring/node-exporter-j46kr"
Nov 03 20:04:27.080616 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.080393    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cb9bef9b-c7cd-410e-8c1e-1b27474daf81-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-rwhnf\" (UID: \"cb9bef9b-c7cd-410e-8c1e-1b27474daf81\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-rwhnf"
Nov 03 20:04:27.080616 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.080403    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/15b5070f-d0d3-4ad8-aa1d-93cc7b96003a-agent-certs\") pod \"konnectivity-agent-5p48g\" (UID: \"15b5070f-d0d3-4ad8-aa1d-93cc7b96003a\") " pod="kube-system/konnectivity-agent-5p48g"
Nov 03 20:04:27.080616 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.080411    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9126a982-59c9-4cec-a665-5f1bc0ae3389-host-kubelet\") pod \"ovnkube-node-6sv7j\" (UID: \"9126a982-59c9-4cec-a665-5f1bc0ae3389\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sv7j"
Nov 03 20:04:27.080616 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.080430    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrdtp\" (UniqueName: \"kubernetes.io/projected/716cc9cf-d26a-481b-8114-169c6104ce56-kube-api-access-vrdtp\") pod \"node-resolver-t6dqw\" (UID: \"716cc9cf-d26a-481b-8114-169c6104ce56\") " pod="openshift-dns/node-resolver-t6dqw"
Nov 03 20:04:27.080616 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.080445    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f21cf70e-682f-490c-bda5-3954df150487-etc-sysctl-conf\") pod \"tuned-4dmkq\" (UID: \"f21cf70e-682f-490c-bda5-3954df150487\") " pod="openshift-cluster-node-tuning-operator/tuned-4dmkq"
Nov 03 20:04:27.080616 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.080475    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/15b5070f-d0d3-4ad8-aa1d-93cc7b96003a-konnectivity-ca\") pod \"konnectivity-agent-5p48g\" (UID: \"15b5070f-d0d3-4ad8-aa1d-93cc7b96003a\") " pod="kube-system/konnectivity-agent-5p48g"
Nov 03 20:04:27.080616 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.080530    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/59f6a4d6-afe0-4c96-a54a-d8e6b5c66a38-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5bd4bfb68d-9449x\" (UID: \"59f6a4d6-afe0-4c96-a54a-d8e6b5c66a38\") " pod="openshift-monitoring/thanos-querier-5bd4bfb68d-9449x"
Nov 03 20:04:27.080616 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.080548    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/682463ee-d9c9-4e9b-82b7-f1116898debf-multus-cni-dir\") pod \"multus-h6p95\" (UID: \"682463ee-d9c9-4e9b-82b7-f1116898debf\") " pod="openshift-multus/multus-h6p95"
Nov 03 20:04:27.080616 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.080584    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cb9bef9b-c7cd-410e-8c1e-1b27474daf81-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-rwhnf\" (UID: \"cb9bef9b-c7cd-410e-8c1e-1b27474daf81\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-rwhnf"
Nov 03 20:04:27.080616 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.080587    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/682463ee-d9c9-4e9b-82b7-f1116898debf-host-run-netns\") pod \"multus-h6p95\" (UID: \"682463ee-d9c9-4e9b-82b7-f1116898debf\") " pod="openshift-multus/multus-h6p95"
Nov 03 20:04:27.080616 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.080629    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/cb9bef9b-c7cd-410e-8c1e-1b27474daf81-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-rwhnf\" (UID: \"cb9bef9b-c7cd-410e-8c1e-1b27474daf81\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-rwhnf"
Nov 03 20:04:27.080616 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.080633    5570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 03 20:04:27.080616 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.080664    5570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 03 20:04:27.080616 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.080676    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z67jw\" (UniqueName: \"kubernetes.io/projected/f7d30630-43af-4157-b90f-d7eddef25085-kube-api-access-z67jw\") pod \"alertmanager-main-0\" (UID: \"f7d30630-43af-4157-b90f-d7eddef25085\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:04:27.080616 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.080686    5570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Nov 03 20:04:27.083713 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.080709    5570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 03 20:04:27.083713 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.080747    5570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 03 20:04:27.083713 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.080749    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59sqq\" (UniqueName: \"kubernetes.io/projected/f21cf70e-682f-490c-bda5-3954df150487-kube-api-access-59sqq\") pod \"tuned-4dmkq\" (UID: \"f21cf70e-682f-490c-bda5-3954df150487\") " pod="openshift-cluster-node-tuning-operator/tuned-4dmkq"
Nov 03 20:04:27.083713 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.080766    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/59f6a4d6-afe0-4c96-a54a-d8e6b5c66a38-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5bd4bfb68d-9449x\" (UID: \"59f6a4d6-afe0-4c96-a54a-d8e6b5c66a38\") " pod="openshift-monitoring/thanos-querier-5bd4bfb68d-9449x"
Nov 03 20:04:27.083713 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.080823    5570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 03 20:04:27.083713 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.080896    5570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 03 20:04:27.083713 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.080901    5570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Nov 03 20:04:27.083713 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.080922    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/cb9bef9b-c7cd-410e-8c1e-1b27474daf81-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-rwhnf\" (UID: \"cb9bef9b-c7cd-410e-8c1e-1b27474daf81\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-rwhnf"
Nov 03 20:04:27.083713 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.080956    5570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 03 20:04:27.083713 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.080973    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/db895326-506d-479d-a526-6bf71be7f967-cni-binary-copy\") pod \"multus-additional-cni-plugins-mhtmw\" (UID: \"db895326-506d-479d-a526-6bf71be7f967\") " pod="openshift-multus/multus-additional-cni-plugins-mhtmw"
Nov 03 20:04:27.083713 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.081036    5570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 03 20:04:27.083713 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.081038    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/59f6a4d6-afe0-4c96-a54a-d8e6b5c66a38-metrics-client-ca\") pod \"thanos-querier-5bd4bfb68d-9449x\" (UID: \"59f6a4d6-afe0-4c96-a54a-d8e6b5c66a38\") " pod="openshift-monitoring/thanos-querier-5bd4bfb68d-9449x"
Nov 03 20:04:27.083713 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.081077    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/682463ee-d9c9-4e9b-82b7-f1116898debf-cni-binary-copy\") pod \"multus-h6p95\" (UID: \"682463ee-d9c9-4e9b-82b7-f1116898debf\") " pod="openshift-multus/multus-h6p95"
Nov 03 20:04:27.083713 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.081081    5570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 03 20:04:27.083713 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.081106    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/74d4808e-24ce-4c47-b4ce-e1188eeea6f1-node-exporter-wtmp\") pod \"node-exporter-j46kr\" (UID: \"74d4808e-24ce-4c47-b4ce-e1188eeea6f1\") " pod="openshift-monitoring/node-exporter-j46kr"
Nov 03 20:04:27.083713 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.081144    5570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 03 20:04:27.083713 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.081135    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1d4ce60c-e058-4c29-99e8-5f4f6eead744-host\") pod \"node-ca-zzt8t\" (UID: \"1d4ce60c-e058-4c29-99e8-5f4f6eead744\") " pod="openshift-image-registry/node-ca-zzt8t"
Nov 03 20:04:27.083713 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.081186    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9126a982-59c9-4cec-a665-5f1bc0ae3389-host-slash\") pod \"ovnkube-node-6sv7j\" (UID: \"9126a982-59c9-4cec-a665-5f1bc0ae3389\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sv7j"
Nov 03 20:04:27.083713 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.081214    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/db895326-506d-479d-a526-6bf71be7f967-cni-binary-copy\") pod \"multus-additional-cni-plugins-mhtmw\" (UID: \"db895326-506d-479d-a526-6bf71be7f967\") " pod="openshift-multus/multus-additional-cni-plugins-mhtmw"
Nov 03 20:04:27.083713 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.081223    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9126a982-59c9-4cec-a665-5f1bc0ae3389-ovn-node-metrics-cert\") pod \"ovnkube-node-6sv7j\" (UID: \"9126a982-59c9-4cec-a665-5f1bc0ae3389\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sv7j"
Nov 03 20:04:27.083713 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.081238    5570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 03 20:04:27.083713 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.081273    5570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Nov 03 20:04:27.083713 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.081273    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/db895326-506d-479d-a526-6bf71be7f967-system-cni-dir\") pod \"multus-additional-cni-plugins-mhtmw\" (UID: \"db895326-506d-479d-a526-6bf71be7f967\") " pod="openshift-multus/multus-additional-cni-plugins-mhtmw"
Nov 03 20:04:27.083713 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.081284    5570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Nov 03 20:04:27.083713 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.081296    5570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 03 20:04:27.083713 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.081300    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/682463ee-d9c9-4e9b-82b7-f1116898debf-cni-binary-copy\") pod \"multus-h6p95\" (UID: \"682463ee-d9c9-4e9b-82b7-f1116898debf\") " pod="openshift-multus/multus-h6p95"
Nov 03 20:04:27.083713 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.081314    5570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 03 20:04:27.083713 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.081286    5570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 03 20:04:27.083713 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.081350    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/db895326-506d-479d-a526-6bf71be7f967-cnibin\") pod \"multus-additional-cni-plugins-mhtmw\" (UID: \"db895326-506d-479d-a526-6bf71be7f967\") " pod="openshift-multus/multus-additional-cni-plugins-mhtmw"
Nov 03 20:04:27.083713 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.081352    5570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 03 20:04:27.083713 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.081385    5570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 03 20:04:27.083713 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.081406    5570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 03 20:04:27.083713 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.081417    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/65008653-586d-4a76-a53f-7efe1f6d839b-registration-dir\") pod \"aws-ebs-csi-driver-node-9f5s5\" (UID: \"65008653-586d-4a76-a53f-7efe1f6d839b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9f5s5"
Nov 03 20:04:27.083713 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.081499    5570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 03 20:04:27.083713 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.081504    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9126a982-59c9-4cec-a665-5f1bc0ae3389-ovn-node-metrics-cert\") pod \"ovnkube-node-6sv7j\" (UID: \"9126a982-59c9-4cec-a665-5f1bc0ae3389\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sv7j"
Nov 03 20:04:27.083713 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.081706    5570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Nov 03 20:04:27.083713 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.082275    5570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Nov 03 20:04:27.087113 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.087092    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2fd3c650-2f4c-404e-ab06-c50612af72ce-config-volume\") pod \"dns-default-7zl2f\" (UID: \"2fd3c650-2f4c-404e-ab06-c50612af72ce\") " pod="openshift-dns/dns-default-7zl2f"
Nov 03 20:04:27.087193 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.087119    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f7d30630-43af-4157-b90f-d7eddef25085-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f7d30630-43af-4157-b90f-d7eddef25085\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:04:27.087193 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.087167    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/d5d14caa-6661-4146-b888-59c3518216d4-metrics-server-audit-profiles\") pod \"metrics-server-654b7fcdc8-l6l9f\" (UID: \"d5d14caa-6661-4146-b888-59c3518216d4\") " pod="openshift-monitoring/metrics-server-654b7fcdc8-l6l9f"
Nov 03 20:04:27.087279 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.087265    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cb9bef9b-c7cd-410e-8c1e-1b27474daf81-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-rwhnf\" (UID: \"cb9bef9b-c7cd-410e-8c1e-1b27474daf81\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-rwhnf"
Nov 03 20:04:27.088289 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.088232    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/ffe074b1-e2a0-403b-9f81-581fdb4d2775-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-q8b6s\" (UID: \"ffe074b1-e2a0-403b-9f81-581fdb4d2775\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-q8b6s"
Nov 03 20:04:27.088289 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.088234    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/db895326-506d-479d-a526-6bf71be7f967-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mhtmw\" (UID: \"db895326-506d-479d-a526-6bf71be7f967\") " pod="openshift-multus/multus-additional-cni-plugins-mhtmw"
Nov 03 20:04:27.088436 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.088353    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ffe074b1-e2a0-403b-9f81-581fdb4d2775-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-q8b6s\" (UID: \"ffe074b1-e2a0-403b-9f81-581fdb4d2775\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-q8b6s"
Nov 03 20:04:27.091840 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.088839    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/74d4808e-24ce-4c47-b4ce-e1188eeea6f1-metrics-client-ca\") pod \"node-exporter-j46kr\" (UID: \"74d4808e-24ce-4c47-b4ce-e1188eeea6f1\") " pod="openshift-monitoring/node-exporter-j46kr"
Nov 03 20:04:27.091840 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.091013    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/884cbb80-b3f8-4f90-bd55-328d17943cf3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8bwhz\" (UID: \"884cbb80-b3f8-4f90-bd55-328d17943cf3\") " pod="openshift-insights/insights-runtime-extractor-8bwhz"
Nov 03 20:04:27.091840 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.091206    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5d14caa-6661-4146-b888-59c3518216d4-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-654b7fcdc8-l6l9f\" (UID: \"d5d14caa-6661-4146-b888-59c3518216d4\") " pod="openshift-monitoring/metrics-server-654b7fcdc8-l6l9f"
Nov 03 20:04:27.091840 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.091479    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f7d30630-43af-4157-b90f-d7eddef25085-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f7d30630-43af-4157-b90f-d7eddef25085\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:04:27.091840 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.091667    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/59f6a4d6-afe0-4c96-a54a-d8e6b5c66a38-metrics-client-ca\") pod \"thanos-querier-5bd4bfb68d-9449x\" (UID: \"59f6a4d6-afe0-4c96-a54a-d8e6b5c66a38\") " pod="openshift-monitoring/thanos-querier-5bd4bfb68d-9449x"
Nov 03 20:04:27.092134 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.091873    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/682463ee-d9c9-4e9b-82b7-f1116898debf-multus-daemon-config\") pod \"multus-h6p95\" (UID: \"682463ee-d9c9-4e9b-82b7-f1116898debf\") " pod="openshift-multus/multus-h6p95"
Nov 03 20:04:27.092134 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.091990    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/db895326-506d-479d-a526-6bf71be7f967-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mhtmw\" (UID: \"db895326-506d-479d-a526-6bf71be7f967\") " pod="openshift-multus/multus-additional-cni-plugins-mhtmw"
Nov 03 20:04:27.092134 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.092061    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/15b5070f-d0d3-4ad8-aa1d-93cc7b96003a-konnectivity-ca\") pod \"konnectivity-agent-5p48g\" (UID: \"15b5070f-d0d3-4ad8-aa1d-93cc7b96003a\") " pod="kube-system/konnectivity-agent-5p48g"
Nov 03 20:04:27.092269 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.092209    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/74d4808e-24ce-4c47-b4ce-e1188eeea6f1-node-exporter-accelerators-collector-config\") pod \"node-exporter-j46kr\" (UID: \"74d4808e-24ce-4c47-b4ce-e1188eeea6f1\") " pod="openshift-monitoring/node-exporter-j46kr"
Nov 03 20:04:27.095426 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.095382    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcxdr\" (UniqueName: \"kubernetes.io/projected/d71eaa2a-9659-4827-9dfc-c1fce02565cb-kube-api-access-zcxdr\") pod \"network-check-target-6dfr5\" (UID: \"d71eaa2a-9659-4827-9dfc-c1fce02565cb\") " pod="openshift-network-diagnostics/network-check-target-6dfr5"
Nov 03 20:04:27.095542 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.095480    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f547t\" (UniqueName: \"kubernetes.io/projected/74d4808e-24ce-4c47-b4ce-e1188eeea6f1-kube-api-access-f547t\") pod \"node-exporter-j46kr\" (UID: \"74d4808e-24ce-4c47-b4ce-e1188eeea6f1\") " pod="openshift-monitoring/node-exporter-j46kr"
Nov 03 20:04:27.095724 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.095703    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7hn2\" (UniqueName: \"kubernetes.io/projected/59f6a4d6-afe0-4c96-a54a-d8e6b5c66a38-kube-api-access-w7hn2\") pod \"thanos-querier-5bd4bfb68d-9449x\" (UID: \"59f6a4d6-afe0-4c96-a54a-d8e6b5c66a38\") " pod="openshift-monitoring/thanos-querier-5bd4bfb68d-9449x"
Nov 03 20:04:27.098965 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.098943    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s92q2\" (UniqueName: \"kubernetes.io/projected/1d4ce60c-e058-4c29-99e8-5f4f6eead744-kube-api-access-s92q2\") pod \"node-ca-zzt8t\" (UID: \"1d4ce60c-e058-4c29-99e8-5f4f6eead744\") " pod="openshift-image-registry/node-ca-zzt8t"
Nov 03 20:04:27.099619 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.099598    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4g29t\" (UniqueName: \"kubernetes.io/projected/ffe074b1-e2a0-403b-9f81-581fdb4d2775-kube-api-access-4g29t\") pod \"kube-state-metrics-5dc8d7d89b-q8b6s\" (UID: \"ffe074b1-e2a0-403b-9f81-581fdb4d2775\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-q8b6s"
Nov 03 20:04:27.104187 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.104165    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrgcs\" (UniqueName: \"kubernetes.io/projected/2073a8c8-6f07-4817-b7b3-468b327e2712-kube-api-access-jrgcs\") pod \"iptables-alerter-9s4wk\" (UID: \"2073a8c8-6f07-4817-b7b3-468b327e2712\") " pod="openshift-network-operator/iptables-alerter-9s4wk"
Nov 03 20:04:27.104311 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.104278    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56z7v\" (UniqueName: \"kubernetes.io/projected/2fd3c650-2f4c-404e-ab06-c50612af72ce-kube-api-access-56z7v\") pod \"dns-default-7zl2f\" (UID: \"2fd3c650-2f4c-404e-ab06-c50612af72ce\") " pod="openshift-dns/dns-default-7zl2f"
Nov 03 20:04:27.104402 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.104277    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlzhf\" (UniqueName: \"kubernetes.io/projected/db895326-506d-479d-a526-6bf71be7f967-kube-api-access-vlzhf\") pod \"multus-additional-cni-plugins-mhtmw\" (UID: \"db895326-506d-479d-a526-6bf71be7f967\") " pod="openshift-multus/multus-additional-cni-plugins-mhtmw"
Nov 03 20:04:27.104494 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.104477    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-df59d\" (UniqueName: \"kubernetes.io/projected/3f681106-32d4-4550-a00d-1f9c52060b57-kube-api-access-df59d\") pod \"ingress-canary-5jnts\" (UID: \"3f681106-32d4-4550-a00d-1f9c52060b57\") " pod="openshift-ingress-canary/ingress-canary-5jnts"
Nov 03 20:04:27.104560 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.104522    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ndt8\" (UniqueName: \"kubernetes.io/projected/cb9bef9b-c7cd-410e-8c1e-1b27474daf81-kube-api-access-4ndt8\") pod \"openshift-state-metrics-7bd7489957-rwhnf\" (UID: \"cb9bef9b-c7cd-410e-8c1e-1b27474daf81\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-rwhnf"
Nov 03 20:04:27.104679 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.104660    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrdtp\" (UniqueName: \"kubernetes.io/projected/716cc9cf-d26a-481b-8114-169c6104ce56-kube-api-access-vrdtp\") pod \"node-resolver-t6dqw\" (UID: \"716cc9cf-d26a-481b-8114-169c6104ce56\") " pod="openshift-dns/node-resolver-t6dqw"
Nov 03 20:04:27.104742 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.104703    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-59sqq\" (UniqueName: \"kubernetes.io/projected/f21cf70e-682f-490c-bda5-3954df150487-kube-api-access-59sqq\") pod \"tuned-4dmkq\" (UID: \"f21cf70e-682f-490c-bda5-3954df150487\") " pod="openshift-cluster-node-tuning-operator/tuned-4dmkq"
Nov 03 20:04:27.105626 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.105263    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nd2w\" (UniqueName: \"kubernetes.io/projected/4c92a5d0-b34f-46a1-8f8f-a9ae3cb8d5a5-kube-api-access-6nd2w\") pod \"network-metrics-daemon-c947w\" (UID: \"4c92a5d0-b34f-46a1-8f8f-a9ae3cb8d5a5\") " pod="openshift-multus/network-metrics-daemon-c947w"
Nov 03 20:04:27.105626 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.105303    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89vxf\" (UniqueName: \"kubernetes.io/projected/9126a982-59c9-4cec-a665-5f1bc0ae3389-kube-api-access-89vxf\") pod \"ovnkube-node-6sv7j\" (UID: \"9126a982-59c9-4cec-a665-5f1bc0ae3389\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sv7j"
Nov 03 20:04:27.105626 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.105307    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-szdn8\" (UniqueName: \"kubernetes.io/projected/884cbb80-b3f8-4f90-bd55-328d17943cf3-kube-api-access-szdn8\") pod \"insights-runtime-extractor-8bwhz\" (UID: \"884cbb80-b3f8-4f90-bd55-328d17943cf3\") " pod="openshift-insights/insights-runtime-extractor-8bwhz"
Nov 03 20:04:27.105626 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.105305    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkjpg\" (UniqueName: \"kubernetes.io/projected/682463ee-d9c9-4e9b-82b7-f1116898debf-kube-api-access-nkjpg\") pod \"multus-h6p95\" (UID: \"682463ee-d9c9-4e9b-82b7-f1116898debf\") " pod="openshift-multus/multus-h6p95"
Nov 03 20:04:27.105626 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.105327    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8psbp\" (UniqueName: \"kubernetes.io/projected/65008653-586d-4a76-a53f-7efe1f6d839b-kube-api-access-8psbp\") pod \"aws-ebs-csi-driver-node-9f5s5\" (UID: \"65008653-586d-4a76-a53f-7efe1f6d839b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9f5s5"
Nov 03 20:04:27.105626 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.105527    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z67jw\" (UniqueName: \"kubernetes.io/projected/f7d30630-43af-4157-b90f-d7eddef25085-kube-api-access-z67jw\") pod \"alertmanager-main-0\" (UID: \"f7d30630-43af-4157-b90f-d7eddef25085\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:04:27.105626 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.105596    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wq4p\" (UniqueName: \"kubernetes.io/projected/d5d14caa-6661-4146-b888-59c3518216d4-kube-api-access-8wq4p\") pod \"metrics-server-654b7fcdc8-l6l9f\" (UID: \"d5d14caa-6661-4146-b888-59c3518216d4\") " pod="openshift-monitoring/metrics-server-654b7fcdc8-l6l9f"
Nov 03 20:04:27.182109 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.182027    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/682463ee-d9c9-4e9b-82b7-f1116898debf-host-var-lib-cni-bin\") pod \"multus-h6p95\" (UID: \"682463ee-d9c9-4e9b-82b7-f1116898debf\") " pod="openshift-multus/multus-h6p95"
Nov 03 20:04:27.182109 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.182064    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/74d4808e-24ce-4c47-b4ce-e1188eeea6f1-root\") pod \"node-exporter-j46kr\" (UID: \"74d4808e-24ce-4c47-b4ce-e1188eeea6f1\") " pod="openshift-monitoring/node-exporter-j46kr"
Nov 03 20:04:27.182109 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.182081    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f21cf70e-682f-490c-bda5-3954df150487-run\") pod \"tuned-4dmkq\" (UID: \"f21cf70e-682f-490c-bda5-3954df150487\") " pod="openshift-cluster-node-tuning-operator/tuned-4dmkq"
Nov 03 20:04:27.182109 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.182097    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/682463ee-d9c9-4e9b-82b7-f1116898debf-cnibin\") pod \"multus-h6p95\" (UID: \"682463ee-d9c9-4e9b-82b7-f1116898debf\") " pod="openshift-multus/multus-h6p95"
Nov 03 20:04:27.182412 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.182141    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/682463ee-d9c9-4e9b-82b7-f1116898debf-host-var-lib-cni-bin\") pod \"multus-h6p95\" (UID: \"682463ee-d9c9-4e9b-82b7-f1116898debf\") " pod="openshift-multus/multus-h6p95"
Nov 03 20:04:27.182412 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.182149    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/74d4808e-24ce-4c47-b4ce-e1188eeea6f1-root\") pod \"node-exporter-j46kr\" (UID: \"74d4808e-24ce-4c47-b4ce-e1188eeea6f1\") " pod="openshift-monitoring/node-exporter-j46kr"
Nov 03 20:04:27.182412 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.182213    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f21cf70e-682f-490c-bda5-3954df150487-run\") pod \"tuned-4dmkq\" (UID: \"f21cf70e-682f-490c-bda5-3954df150487\") " pod="openshift-cluster-node-tuning-operator/tuned-4dmkq"
Nov 03 20:04:27.182412 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.182249    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/65008653-586d-4a76-a53f-7efe1f6d839b-etc-selinux\") pod \"aws-ebs-csi-driver-node-9f5s5\" (UID: \"65008653-586d-4a76-a53f-7efe1f6d839b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9f5s5"
Nov 03 20:04:27.182412 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.182260    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/682463ee-d9c9-4e9b-82b7-f1116898debf-cnibin\") pod \"multus-h6p95\" (UID: \"682463ee-d9c9-4e9b-82b7-f1116898debf\") " pod="openshift-multus/multus-h6p95"
Nov 03 20:04:27.182412 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.182274    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9126a982-59c9-4cec-a665-5f1bc0ae3389-node-log\") pod \"ovnkube-node-6sv7j\" (UID: \"9126a982-59c9-4cec-a665-5f1bc0ae3389\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sv7j"
Nov 03 20:04:27.182412 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.182292    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9126a982-59c9-4cec-a665-5f1bc0ae3389-host-cni-bin\") pod \"ovnkube-node-6sv7j\" (UID: \"9126a982-59c9-4cec-a665-5f1bc0ae3389\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sv7j"
Nov 03 20:04:27.182412 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.182306    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f21cf70e-682f-490c-bda5-3954df150487-etc-sysconfig\") pod \"tuned-4dmkq\" (UID: \"f21cf70e-682f-490c-bda5-3954df150487\") " pod="openshift-cluster-node-tuning-operator/tuned-4dmkq"
Nov 03 20:04:27.182412 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.182315    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/65008653-586d-4a76-a53f-7efe1f6d839b-etc-selinux\") pod \"aws-ebs-csi-driver-node-9f5s5\" (UID: \"65008653-586d-4a76-a53f-7efe1f6d839b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9f5s5"
Nov 03 20:04:27.182412 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.182329    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9126a982-59c9-4cec-a665-5f1bc0ae3389-node-log\") pod \"ovnkube-node-6sv7j\" (UID: \"9126a982-59c9-4cec-a665-5f1bc0ae3389\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sv7j"
Nov 03 20:04:27.182412 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.182322    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f21cf70e-682f-490c-bda5-3954df150487-etc-kubernetes\") pod \"tuned-4dmkq\" (UID: \"f21cf70e-682f-490c-bda5-3954df150487\") " pod="openshift-cluster-node-tuning-operator/tuned-4dmkq"
Nov 03 20:04:27.182412 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.182360    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f21cf70e-682f-490c-bda5-3954df150487-etc-kubernetes\") pod \"tuned-4dmkq\" (UID: \"f21cf70e-682f-490c-bda5-3954df150487\") " pod="openshift-cluster-node-tuning-operator/tuned-4dmkq"
Nov 03 20:04:27.182412 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.182369    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9126a982-59c9-4cec-a665-5f1bc0ae3389-etc-openvswitch\") pod \"ovnkube-node-6sv7j\" (UID: \"9126a982-59c9-4cec-a665-5f1bc0ae3389\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sv7j"
Nov 03 20:04:27.182412 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.182374    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9126a982-59c9-4cec-a665-5f1bc0ae3389-host-cni-bin\") pod \"ovnkube-node-6sv7j\" (UID: \"9126a982-59c9-4cec-a665-5f1bc0ae3389\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sv7j"
Nov 03 20:04:27.182412 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.182381    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f21cf70e-682f-490c-bda5-3954df150487-etc-sysconfig\") pod \"tuned-4dmkq\" (UID: \"f21cf70e-682f-490c-bda5-3954df150487\") " pod="openshift-cluster-node-tuning-operator/tuned-4dmkq"
Nov 03 20:04:27.182412 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.182396    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9126a982-59c9-4cec-a665-5f1bc0ae3389-run-ovn\") pod \"ovnkube-node-6sv7j\" (UID: \"9126a982-59c9-4cec-a665-5f1bc0ae3389\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sv7j"
Nov 03 20:04:27.182412 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.182414    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f21cf70e-682f-490c-bda5-3954df150487-sys\") pod \"tuned-4dmkq\" (UID: \"f21cf70e-682f-490c-bda5-3954df150487\") " pod="openshift-cluster-node-tuning-operator/tuned-4dmkq"
Nov 03 20:04:27.183002 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.182428    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f21cf70e-682f-490c-bda5-3954df150487-lib-modules\") pod \"tuned-4dmkq\" (UID: \"f21cf70e-682f-490c-bda5-3954df150487\") " pod="openshift-cluster-node-tuning-operator/tuned-4dmkq"
Nov 03 20:04:27.183002 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.182445    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9126a982-59c9-4cec-a665-5f1bc0ae3389-var-lib-openvswitch\") pod \"ovnkube-node-6sv7j\" (UID: \"9126a982-59c9-4cec-a665-5f1bc0ae3389\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sv7j"
Nov 03 20:04:27.183002 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.182469    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9126a982-59c9-4cec-a665-5f1bc0ae3389-var-lib-openvswitch\") pod \"ovnkube-node-6sv7j\" (UID: \"9126a982-59c9-4cec-a665-5f1bc0ae3389\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sv7j"
Nov 03 20:04:27.183002 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.182471    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/51dffd76-5c30-440c-85ed-d81221663833-dbus\") pod \"global-pull-secret-syncer-j4jzv\" (UID: \"51dffd76-5c30-440c-85ed-d81221663833\") " pod="kube-system/global-pull-secret-syncer-j4jzv"
Nov 03 20:04:27.183002 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.182498    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f21cf70e-682f-490c-bda5-3954df150487-sys\") pod \"tuned-4dmkq\" (UID: \"f21cf70e-682f-490c-bda5-3954df150487\") " pod="openshift-cluster-node-tuning-operator/tuned-4dmkq"
Nov 03 20:04:27.183002 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.182499    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/682463ee-d9c9-4e9b-82b7-f1116898debf-multus-socket-dir-parent\") pod \"multus-h6p95\" (UID: \"682463ee-d9c9-4e9b-82b7-f1116898debf\") " pod="openshift-multus/multus-h6p95"
Nov 03 20:04:27.183002 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.182520    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9126a982-59c9-4cec-a665-5f1bc0ae3389-run-ovn\") pod \"ovnkube-node-6sv7j\" (UID: \"9126a982-59c9-4cec-a665-5f1bc0ae3389\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sv7j"
Nov 03 20:04:27.183002 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.182526    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/682463ee-d9c9-4e9b-82b7-f1116898debf-host-var-lib-cni-multus\") pod \"multus-h6p95\" (UID: \"682463ee-d9c9-4e9b-82b7-f1116898debf\") " pod="openshift-multus/multus-h6p95"
Nov 03 20:04:27.183002 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.182554    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/682463ee-d9c9-4e9b-82b7-f1116898debf-host-run-k8s-cni-cncf-io\") pod \"multus-h6p95\" (UID: \"682463ee-d9c9-4e9b-82b7-f1116898debf\") " pod="openshift-multus/multus-h6p95"
Nov 03 20:04:27.183002 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.182567    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/51dffd76-5c30-440c-85ed-d81221663833-dbus\") pod \"global-pull-secret-syncer-j4jzv\" (UID: \"51dffd76-5c30-440c-85ed-d81221663833\") " pod="kube-system/global-pull-secret-syncer-j4jzv"
Nov 03 20:04:27.183002 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.182574    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9126a982-59c9-4cec-a665-5f1bc0ae3389-run-openvswitch\") pod \"ovnkube-node-6sv7j\" (UID: \"9126a982-59c9-4cec-a665-5f1bc0ae3389\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sv7j"
Nov 03 20:04:27.183002 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.182593    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9126a982-59c9-4cec-a665-5f1bc0ae3389-etc-openvswitch\") pod \"ovnkube-node-6sv7j\" (UID: \"9126a982-59c9-4cec-a665-5f1bc0ae3389\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sv7j"
Nov 03 20:04:27.183002 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.182596    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f21cf70e-682f-490c-bda5-3954df150487-var-lib-kubelet\") pod \"tuned-4dmkq\" (UID: \"f21cf70e-682f-490c-bda5-3954df150487\") " pod="openshift-cluster-node-tuning-operator/tuned-4dmkq"
Nov 03 20:04:27.183002 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.182624    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/682463ee-d9c9-4e9b-82b7-f1116898debf-host-run-k8s-cni-cncf-io\") pod \"multus-h6p95\" (UID: \"682463ee-d9c9-4e9b-82b7-f1116898debf\") " pod="openshift-multus/multus-h6p95"
Nov 03 20:04:27.183002 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.182627    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f21cf70e-682f-490c-bda5-3954df150487-lib-modules\") pod \"tuned-4dmkq\" (UID: \"f21cf70e-682f-490c-bda5-3954df150487\") " pod="openshift-cluster-node-tuning-operator/tuned-4dmkq"
Nov 03 20:04:27.183002 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.182635    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/884cbb80-b3f8-4f90-bd55-328d17943cf3-crio-socket\") pod \"insights-runtime-extractor-8bwhz\" (UID: \"884cbb80-b3f8-4f90-bd55-328d17943cf3\") " pod="openshift-insights/insights-runtime-extractor-8bwhz"
Nov 03 20:04:27.183002 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.182634    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/682463ee-d9c9-4e9b-82b7-f1116898debf-host-var-lib-cni-multus\") pod \"multus-h6p95\" (UID: \"682463ee-d9c9-4e9b-82b7-f1116898debf\") " pod="openshift-multus/multus-h6p95"
Nov 03 20:04:27.183002 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.182672    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/682463ee-d9c9-4e9b-82b7-f1116898debf-etc-kubernetes\") pod \"multus-h6p95\" (UID: \"682463ee-d9c9-4e9b-82b7-f1116898debf\") " pod="openshift-multus/multus-h6p95"
Nov 03 20:04:27.183002 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.182681    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9126a982-59c9-4cec-a665-5f1bc0ae3389-run-openvswitch\") pod \"ovnkube-node-6sv7j\" (UID: \"9126a982-59c9-4cec-a665-5f1bc0ae3389\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sv7j"
Nov 03 20:04:27.183002 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.182686    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/884cbb80-b3f8-4f90-bd55-328d17943cf3-crio-socket\") pod \"insights-runtime-extractor-8bwhz\" (UID: \"884cbb80-b3f8-4f90-bd55-328d17943cf3\") " pod="openshift-insights/insights-runtime-extractor-8bwhz"
Nov 03 20:04:27.183002 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.182684    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/682463ee-d9c9-4e9b-82b7-f1116898debf-multus-socket-dir-parent\") pod \"multus-h6p95\" (UID: \"682463ee-d9c9-4e9b-82b7-f1116898debf\") " pod="openshift-multus/multus-h6p95"
Nov 03 20:04:27.183002 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.182732    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f21cf70e-682f-490c-bda5-3954df150487-var-lib-kubelet\") pod \"tuned-4dmkq\" (UID: \"f21cf70e-682f-490c-bda5-3954df150487\") " pod="openshift-cluster-node-tuning-operator/tuned-4dmkq"
Nov 03 20:04:27.183002 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.182759    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/682463ee-d9c9-4e9b-82b7-f1116898debf-etc-kubernetes\") pod \"multus-h6p95\" (UID: \"682463ee-d9c9-4e9b-82b7-f1116898debf\") " pod="openshift-multus/multus-h6p95"
Nov 03 20:04:27.183002 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.182762    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f21cf70e-682f-490c-bda5-3954df150487-etc-sysctl-d\") pod \"tuned-4dmkq\" (UID: \"f21cf70e-682f-490c-bda5-3954df150487\") " pod="openshift-cluster-node-tuning-operator/tuned-4dmkq"
Nov 03 20:04:27.183002 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.182821    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/65008653-586d-4a76-a53f-7efe1f6d839b-socket-dir\") pod \"aws-ebs-csi-driver-node-9f5s5\" (UID: \"65008653-586d-4a76-a53f-7efe1f6d839b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9f5s5"
Nov 03 20:04:27.183002 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.182844    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/682463ee-d9c9-4e9b-82b7-f1116898debf-host-run-multus-certs\") pod \"multus-h6p95\" (UID: \"682463ee-d9c9-4e9b-82b7-f1116898debf\") " pod="openshift-multus/multus-h6p95"
Nov 03 20:04:27.183002 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.182858    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/682463ee-d9c9-4e9b-82b7-f1116898debf-multus-conf-dir\") pod \"multus-h6p95\" (UID: \"682463ee-d9c9-4e9b-82b7-f1116898debf\") " pod="openshift-multus/multus-h6p95"
Nov 03 20:04:27.183002 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.182874    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/74d4808e-24ce-4c47-b4ce-e1188eeea6f1-sys\") pod \"node-exporter-j46kr\" (UID: \"74d4808e-24ce-4c47-b4ce-e1188eeea6f1\") " pod="openshift-monitoring/node-exporter-j46kr"
Nov 03 20:04:27.183002 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.182889    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/682463ee-d9c9-4e9b-82b7-f1116898debf-multus-conf-dir\") pod \"multus-h6p95\" (UID: \"682463ee-d9c9-4e9b-82b7-f1116898debf\") " pod="openshift-multus/multus-h6p95"
Nov 03 20:04:27.183002 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.182914    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/682463ee-d9c9-4e9b-82b7-f1116898debf-host-run-multus-certs\") pod \"multus-h6p95\" (UID: \"682463ee-d9c9-4e9b-82b7-f1116898debf\") " pod="openshift-multus/multus-h6p95"
Nov 03 20:04:27.183002 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.182914    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9126a982-59c9-4cec-a665-5f1bc0ae3389-host-kubelet\") pod \"ovnkube-node-6sv7j\" (UID: \"9126a982-59c9-4cec-a665-5f1bc0ae3389\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sv7j"
Nov 03 20:04:27.183002 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.182912    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/74d4808e-24ce-4c47-b4ce-e1188eeea6f1-sys\") pod \"node-exporter-j46kr\" (UID: \"74d4808e-24ce-4c47-b4ce-e1188eeea6f1\") " pod="openshift-monitoring/node-exporter-j46kr"
Nov 03 20:04:27.183002 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.182949    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f21cf70e-682f-490c-bda5-3954df150487-etc-sysctl-conf\") pod \"tuned-4dmkq\" (UID: \"f21cf70e-682f-490c-bda5-3954df150487\") " pod="openshift-cluster-node-tuning-operator/tuned-4dmkq"
Nov 03 20:04:27.183002 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.182928    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/65008653-586d-4a76-a53f-7efe1f6d839b-socket-dir\") pod \"aws-ebs-csi-driver-node-9f5s5\" (UID: \"65008653-586d-4a76-a53f-7efe1f6d839b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9f5s5"
Nov 03 20:04:27.183002 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.182974    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/682463ee-d9c9-4e9b-82b7-f1116898debf-multus-cni-dir\") pod \"multus-h6p95\" (UID: \"682463ee-d9c9-4e9b-82b7-f1116898debf\") " pod="openshift-multus/multus-h6p95"
Nov 03 20:04:27.183002 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.182974    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f21cf70e-682f-490c-bda5-3954df150487-etc-sysctl-d\") pod \"tuned-4dmkq\" (UID: \"f21cf70e-682f-490c-bda5-3954df150487\") " pod="openshift-cluster-node-tuning-operator/tuned-4dmkq"
Nov 03 20:04:27.183002 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.183004    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9126a982-59c9-4cec-a665-5f1bc0ae3389-host-kubelet\") pod \"ovnkube-node-6sv7j\" (UID: \"9126a982-59c9-4cec-a665-5f1bc0ae3389\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sv7j"
Nov 03 20:04:27.183002 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.183014    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/682463ee-d9c9-4e9b-82b7-f1116898debf-host-run-netns\") pod \"multus-h6p95\" (UID: \"682463ee-d9c9-4e9b-82b7-f1116898debf\") " pod="openshift-multus/multus-h6p95"
Nov 03 20:04:27.183002 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.183029    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/682463ee-d9c9-4e9b-82b7-f1116898debf-multus-cni-dir\") pod \"multus-h6p95\" (UID: \"682463ee-d9c9-4e9b-82b7-f1116898debf\") " pod="openshift-multus/multus-h6p95"
Nov 03 20:04:27.184383 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.183039    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/74d4808e-24ce-4c47-b4ce-e1188eeea6f1-node-exporter-wtmp\") pod \"node-exporter-j46kr\" (UID: \"74d4808e-24ce-4c47-b4ce-e1188eeea6f1\") " pod="openshift-monitoring/node-exporter-j46kr"
Nov 03 20:04:27.184383 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.183071    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1d4ce60c-e058-4c29-99e8-5f4f6eead744-host\") pod \"node-ca-zzt8t\" (UID: \"1d4ce60c-e058-4c29-99e8-5f4f6eead744\") " pod="openshift-image-registry/node-ca-zzt8t"
Nov 03 20:04:27.184383 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.183071    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f21cf70e-682f-490c-bda5-3954df150487-etc-sysctl-conf\") pod \"tuned-4dmkq\" (UID: \"f21cf70e-682f-490c-bda5-3954df150487\") " pod="openshift-cluster-node-tuning-operator/tuned-4dmkq"
Nov 03 20:04:27.184383 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.183073    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/682463ee-d9c9-4e9b-82b7-f1116898debf-host-run-netns\") pod \"multus-h6p95\" (UID: \"682463ee-d9c9-4e9b-82b7-f1116898debf\") " pod="openshift-multus/multus-h6p95"
Nov 03 20:04:27.184383 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.183093    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9126a982-59c9-4cec-a665-5f1bc0ae3389-host-slash\") pod \"ovnkube-node-6sv7j\" (UID: \"9126a982-59c9-4cec-a665-5f1bc0ae3389\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sv7j"
Nov 03 20:04:27.184383 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.183119    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/74d4808e-24ce-4c47-b4ce-e1188eeea6f1-node-exporter-wtmp\") pod \"node-exporter-j46kr\" (UID: \"74d4808e-24ce-4c47-b4ce-e1188eeea6f1\") " pod="openshift-monitoring/node-exporter-j46kr"
Nov 03 20:04:27.184383 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.183126    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1d4ce60c-e058-4c29-99e8-5f4f6eead744-host\") pod \"node-ca-zzt8t\" (UID: \"1d4ce60c-e058-4c29-99e8-5f4f6eead744\") " pod="openshift-image-registry/node-ca-zzt8t"
Nov 03 20:04:27.184383 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.183123    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/db895326-506d-479d-a526-6bf71be7f967-system-cni-dir\") pod \"multus-additional-cni-plugins-mhtmw\" (UID: \"db895326-506d-479d-a526-6bf71be7f967\") " pod="openshift-multus/multus-additional-cni-plugins-mhtmw"
Nov 03 20:04:27.184383 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.183152    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/db895326-506d-479d-a526-6bf71be7f967-system-cni-dir\") pod \"multus-additional-cni-plugins-mhtmw\" (UID: \"db895326-506d-479d-a526-6bf71be7f967\") " pod="openshift-multus/multus-additional-cni-plugins-mhtmw"
Nov 03 20:04:27.184383 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.183162    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9126a982-59c9-4cec-a665-5f1bc0ae3389-host-slash\") pod \"ovnkube-node-6sv7j\" (UID: \"9126a982-59c9-4cec-a665-5f1bc0ae3389\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sv7j"
Nov 03 20:04:27.184383 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.183165    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/db895326-506d-479d-a526-6bf71be7f967-cnibin\") pod \"multus-additional-cni-plugins-mhtmw\" (UID: \"db895326-506d-479d-a526-6bf71be7f967\") " pod="openshift-multus/multus-additional-cni-plugins-mhtmw"
Nov 03 20:04:27.184383 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.183195    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/db895326-506d-479d-a526-6bf71be7f967-cnibin\") pod \"multus-additional-cni-plugins-mhtmw\" (UID: \"db895326-506d-479d-a526-6bf71be7f967\") " pod="openshift-multus/multus-additional-cni-plugins-mhtmw"
Nov 03 20:04:27.184383 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.183197    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/65008653-586d-4a76-a53f-7efe1f6d839b-registration-dir\") pod \"aws-ebs-csi-driver-node-9f5s5\" (UID: \"65008653-586d-4a76-a53f-7efe1f6d839b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9f5s5"
Nov 03 20:04:27.184383 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.183225    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/65008653-586d-4a76-a53f-7efe1f6d839b-sys-fs\") pod \"aws-ebs-csi-driver-node-9f5s5\" (UID: \"65008653-586d-4a76-a53f-7efe1f6d839b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9f5s5"
Nov 03 20:04:27.184383 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.183239    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/65008653-586d-4a76-a53f-7efe1f6d839b-registration-dir\") pod \"aws-ebs-csi-driver-node-9f5s5\" (UID: \"65008653-586d-4a76-a53f-7efe1f6d839b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9f5s5"
Nov 03 20:04:27.184383 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.183255    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/51dffd76-5c30-440c-85ed-d81221663833-kubelet-config\") pod \"global-pull-secret-syncer-j4jzv\" (UID: \"51dffd76-5c30-440c-85ed-d81221663833\") " pod="kube-system/global-pull-secret-syncer-j4jzv"
Nov 03 20:04:27.184383 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.183276    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/682463ee-d9c9-4e9b-82b7-f1116898debf-system-cni-dir\") pod \"multus-h6p95\" (UID: \"682463ee-d9c9-4e9b-82b7-f1116898debf\") " pod="openshift-multus/multus-h6p95"
Nov 03 20:04:27.184383 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.183258    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/65008653-586d-4a76-a53f-7efe1f6d839b-sys-fs\") pod \"aws-ebs-csi-driver-node-9f5s5\" (UID: \"65008653-586d-4a76-a53f-7efe1f6d839b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9f5s5"
Nov 03 20:04:27.184383 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.183313    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9126a982-59c9-4cec-a665-5f1bc0ae3389-systemd-units\") pod \"ovnkube-node-6sv7j\" (UID: \"9126a982-59c9-4cec-a665-5f1bc0ae3389\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sv7j"
Nov 03 20:04:27.184383 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.183341    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9126a982-59c9-4cec-a665-5f1bc0ae3389-systemd-units\") pod \"ovnkube-node-6sv7j\" (UID: \"9126a982-59c9-4cec-a665-5f1bc0ae3389\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sv7j"
Nov 03 20:04:27.184383 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.183362    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/51dffd76-5c30-440c-85ed-d81221663833-kubelet-config\") pod \"global-pull-secret-syncer-j4jzv\" (UID: \"51dffd76-5c30-440c-85ed-d81221663833\") " pod="kube-system/global-pull-secret-syncer-j4jzv"
Nov 03 20:04:27.184383 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.183370    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/682463ee-d9c9-4e9b-82b7-f1116898debf-system-cni-dir\") pod \"multus-h6p95\" (UID: \"682463ee-d9c9-4e9b-82b7-f1116898debf\") " pod="openshift-multus/multus-h6p95"
Nov 03 20:04:27.184383 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.183369    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9126a982-59c9-4cec-a665-5f1bc0ae3389-host-run-netns\") pod \"ovnkube-node-6sv7j\" (UID: \"9126a982-59c9-4cec-a665-5f1bc0ae3389\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sv7j"
Nov 03 20:04:27.184383 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.183395    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9126a982-59c9-4cec-a665-5f1bc0ae3389-host-run-netns\") pod \"ovnkube-node-6sv7j\" (UID: \"9126a982-59c9-4cec-a665-5f1bc0ae3389\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sv7j"
Nov 03 20:04:27.184383 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.183401    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9126a982-59c9-4cec-a665-5f1bc0ae3389-run-systemd\") pod \"ovnkube-node-6sv7j\" (UID: \"9126a982-59c9-4cec-a665-5f1bc0ae3389\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sv7j"
Nov 03 20:04:27.184383 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.183418    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9126a982-59c9-4cec-a665-5f1bc0ae3389-run-systemd\") pod \"ovnkube-node-6sv7j\" (UID: \"9126a982-59c9-4cec-a665-5f1bc0ae3389\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sv7j"
Nov 03 20:04:27.184383 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.183440    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f21cf70e-682f-490c-bda5-3954df150487-etc-modprobe-d\") pod \"tuned-4dmkq\" (UID: \"f21cf70e-682f-490c-bda5-3954df150487\") " pod="openshift-cluster-node-tuning-operator/tuned-4dmkq"
Nov 03 20:04:27.184383 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.183464    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f21cf70e-682f-490c-bda5-3954df150487-etc-systemd\") pod \"tuned-4dmkq\" (UID: \"f21cf70e-682f-490c-bda5-3954df150487\") " pod="openshift-cluster-node-tuning-operator/tuned-4dmkq"
Nov 03 20:04:27.184383 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.183487    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f21cf70e-682f-490c-bda5-3954df150487-host\") pod \"tuned-4dmkq\" (UID: \"f21cf70e-682f-490c-bda5-3954df150487\") " pod="openshift-cluster-node-tuning-operator/tuned-4dmkq"
Nov 03 20:04:27.184383 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.183513    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/716cc9cf-d26a-481b-8114-169c6104ce56-hosts-file\") pod \"node-resolver-t6dqw\" (UID: \"716cc9cf-d26a-481b-8114-169c6104ce56\") " pod="openshift-dns/node-resolver-t6dqw"
Nov 03 20:04:27.184383 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.183534    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f21cf70e-682f-490c-bda5-3954df150487-etc-systemd\") pod \"tuned-4dmkq\" (UID: \"f21cf70e-682f-490c-bda5-3954df150487\") " pod="openshift-cluster-node-tuning-operator/tuned-4dmkq"
Nov 03 20:04:27.184383 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.183541    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9126a982-59c9-4cec-a665-5f1bc0ae3389-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6sv7j\" (UID: \"9126a982-59c9-4cec-a665-5f1bc0ae3389\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sv7j"
Nov 03 20:04:27.184383 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.183565    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/716cc9cf-d26a-481b-8114-169c6104ce56-hosts-file\") pod \"node-resolver-t6dqw\" (UID: \"716cc9cf-d26a-481b-8114-169c6104ce56\") " pod="openshift-dns/node-resolver-t6dqw"
Nov 03 20:04:27.184383 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.183568    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/65008653-586d-4a76-a53f-7efe1f6d839b-device-dir\") pod \"aws-ebs-csi-driver-node-9f5s5\" (UID: \"65008653-586d-4a76-a53f-7efe1f6d839b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9f5s5"
Nov 03 20:04:27.184383 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.183576    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9126a982-59c9-4cec-a665-5f1bc0ae3389-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6sv7j\" (UID: \"9126a982-59c9-4cec-a665-5f1bc0ae3389\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sv7j"
Nov 03 20:04:27.184383 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.183547    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f21cf70e-682f-490c-bda5-3954df150487-host\") pod \"tuned-4dmkq\" (UID: \"f21cf70e-682f-490c-bda5-3954df150487\") " pod="openshift-cluster-node-tuning-operator/tuned-4dmkq"
Nov 03 20:04:27.184383 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.183589    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f21cf70e-682f-490c-bda5-3954df150487-etc-modprobe-d\") pod \"tuned-4dmkq\" (UID: \"f21cf70e-682f-490c-bda5-3954df150487\") " pod="openshift-cluster-node-tuning-operator/tuned-4dmkq"
Nov 03 20:04:27.184383 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.183601    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/65008653-586d-4a76-a53f-7efe1f6d839b-device-dir\") pod \"aws-ebs-csi-driver-node-9f5s5\" (UID: \"65008653-586d-4a76-a53f-7efe1f6d839b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9f5s5"
Nov 03 20:04:27.184383 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.183608    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/682463ee-d9c9-4e9b-82b7-f1116898debf-host-var-lib-kubelet\") pod \"multus-h6p95\" (UID: \"682463ee-d9c9-4e9b-82b7-f1116898debf\") " pod="openshift-multus/multus-h6p95"
Nov 03 20:04:27.184383 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.183629    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/682463ee-d9c9-4e9b-82b7-f1116898debf-hostroot\") pod \"multus-h6p95\" (UID: \"682463ee-d9c9-4e9b-82b7-f1116898debf\") " pod="openshift-multus/multus-h6p95"
Nov 03 20:04:27.184383 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.183650    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9126a982-59c9-4cec-a665-5f1bc0ae3389-log-socket\") pod \"ovnkube-node-6sv7j\" (UID: \"9126a982-59c9-4cec-a665-5f1bc0ae3389\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sv7j"
Nov 03 20:04:27.184383 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.183667    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9126a982-59c9-4cec-a665-5f1bc0ae3389-host-run-ovn-kubernetes\") pod \"ovnkube-node-6sv7j\" (UID: \"9126a982-59c9-4cec-a665-5f1bc0ae3389\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sv7j"
Nov 03 20:04:27.184383 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.183671    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/682463ee-d9c9-4e9b-82b7-f1116898debf-host-var-lib-kubelet\") pod \"multus-h6p95\" (UID: \"682463ee-d9c9-4e9b-82b7-f1116898debf\") " pod="openshift-multus/multus-h6p95"
Nov 03 20:04:27.184383 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.183698    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/682463ee-d9c9-4e9b-82b7-f1116898debf-hostroot\") pod \"multus-h6p95\" (UID: \"682463ee-d9c9-4e9b-82b7-f1116898debf\") " pod="openshift-multus/multus-h6p95"
Nov 03 20:04:27.184383 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.183705    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9126a982-59c9-4cec-a665-5f1bc0ae3389-host-cni-netd\") pod \"ovnkube-node-6sv7j\" (UID: \"9126a982-59c9-4cec-a665-5f1bc0ae3389\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sv7j"
Nov 03 20:04:27.184383 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.183703    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9126a982-59c9-4cec-a665-5f1bc0ae3389-log-socket\") pod \"ovnkube-node-6sv7j\" (UID: \"9126a982-59c9-4cec-a665-5f1bc0ae3389\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sv7j"
Nov 03 20:04:27.184383 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.183720    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9126a982-59c9-4cec-a665-5f1bc0ae3389-host-run-ovn-kubernetes\") pod \"ovnkube-node-6sv7j\" (UID: \"9126a982-59c9-4cec-a665-5f1bc0ae3389\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sv7j"
Nov 03 20:04:27.184383 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.183730    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/65008653-586d-4a76-a53f-7efe1f6d839b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9f5s5\" (UID: \"65008653-586d-4a76-a53f-7efe1f6d839b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9f5s5"
Nov 03 20:04:27.184383 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.183754    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9126a982-59c9-4cec-a665-5f1bc0ae3389-host-cni-netd\") pod \"ovnkube-node-6sv7j\" (UID: \"9126a982-59c9-4cec-a665-5f1bc0ae3389\") " pod="openshift-ovn-kubernetes/ovnkube-node-6sv7j"
Nov 03 20:04:27.184383 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.183832    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/65008653-586d-4a76-a53f-7efe1f6d839b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9f5s5\" (UID: \"65008653-586d-4a76-a53f-7efe1f6d839b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9f5s5"
Nov 03 20:04:27.184383 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.183853    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/db895326-506d-479d-a526-6bf71be7f967-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mhtmw\" (UID: \"db895326-506d-479d-a526-6bf71be7f967\") " pod="openshift-multus/multus-additional-cni-plugins-mhtmw"
Nov 03 20:04:27.184383 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.183875    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2073a8c8-6f07-4817-b7b3-468b327e2712-host-slash\") pod \"iptables-alerter-9s4wk\" (UID: \"2073a8c8-6f07-4817-b7b3-468b327e2712\") " pod="openshift-network-operator/iptables-alerter-9s4wk"
Nov 03 20:04:27.184383 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.183924    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2073a8c8-6f07-4817-b7b3-468b327e2712-host-slash\") pod \"iptables-alerter-9s4wk\" (UID: \"2073a8c8-6f07-4817-b7b3-468b327e2712\") " pod="openshift-network-operator/iptables-alerter-9s4wk"
Nov 03 20:04:27.184383 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:27.183953    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/db895326-506d-479d-a526-6bf71be7f967-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mhtmw\" (UID: \"db895326-506d-479d-a526-6bf71be7f967\") " pod="openshift-multus/multus-additional-cni-plugins-mhtmw"
Nov 03 20:04:28.081575 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:28.081541    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-5bd4bfb68d-9449x"
Nov 03 20:04:28.872628 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:28.872590    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-7zl2f"
Nov 03 20:04:32.325821 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:32.325788    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6sv7j"
Nov 03 20:04:34.328513 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:34.328483    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6sv7j"
Nov 03 20:04:34.329010 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:34.328833    5570 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 20:04:36.602750 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:36.602710    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-6dfr5"
Nov 03 20:04:39.911450 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:39.911417    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-5p48g"
Nov 03 20:04:41.687885 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:41.687852    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-654b7fcdc8-l6l9f"
Nov 03 20:04:46.832070 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:46.832034    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 20:04:46.832482 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:46.832373    5570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f7d30630-43af-4157-b90f-d7eddef25085" containerName="alertmanager" containerID="cri-o://0ff023e230ba0d28d23d2743e9b0aba7c73e86105e55e4944b6cb93cc8a89c71" gracePeriod=120
Nov 03 20:04:46.832482 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:46.832397    5570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f7d30630-43af-4157-b90f-d7eddef25085" containerName="kube-rbac-proxy-metric" containerID="cri-o://960a709deb8999b7ce32bdc8df50230a2b1c396c9f4b7fec4565ce238051471c" gracePeriod=120
Nov 03 20:04:46.832482 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:46.832427    5570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f7d30630-43af-4157-b90f-d7eddef25085" containerName="prom-label-proxy" containerID="cri-o://e2504655bc35f9bba51ad6d1bfb0b7293d1790ba3ebcd18215ab07aea2386a41" gracePeriod=120
Nov 03 20:04:46.832482 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:46.832449    5570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f7d30630-43af-4157-b90f-d7eddef25085" containerName="kube-rbac-proxy-web" containerID="cri-o://6bbb13e30ca02316f2656a078c3631aa8a4bee6760823627343e629d39e7dda7" gracePeriod=120
Nov 03 20:04:46.832729 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:46.832458    5570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f7d30630-43af-4157-b90f-d7eddef25085" containerName="config-reloader" containerID="cri-o://26307d79ee47d0ad8409c23eb27f0d2e6e77b80f6bcc80c48a111753adc9ab9c" gracePeriod=120
Nov 03 20:04:46.832729 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:46.832495    5570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f7d30630-43af-4157-b90f-d7eddef25085" containerName="kube-rbac-proxy" containerID="cri-o://014278ced716c2a52d7b544d736e420472f4440af29a73b56a93341b99fc7d51" gracePeriod=120
Nov 03 20:04:47.091253 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:47.091163    5570 generic.go:356] "Generic (PLEG): container finished" podID="f7d30630-43af-4157-b90f-d7eddef25085" containerID="e2504655bc35f9bba51ad6d1bfb0b7293d1790ba3ebcd18215ab07aea2386a41" exitCode=0
Nov 03 20:04:47.091253 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:47.091194    5570 generic.go:356] "Generic (PLEG): container finished" podID="f7d30630-43af-4157-b90f-d7eddef25085" containerID="960a709deb8999b7ce32bdc8df50230a2b1c396c9f4b7fec4565ce238051471c" exitCode=0
Nov 03 20:04:47.091253 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:47.091204    5570 generic.go:356] "Generic (PLEG): container finished" podID="f7d30630-43af-4157-b90f-d7eddef25085" containerID="014278ced716c2a52d7b544d736e420472f4440af29a73b56a93341b99fc7d51" exitCode=0
Nov 03 20:04:47.091253 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:47.091210    5570 generic.go:356] "Generic (PLEG): container finished" podID="f7d30630-43af-4157-b90f-d7eddef25085" containerID="26307d79ee47d0ad8409c23eb27f0d2e6e77b80f6bcc80c48a111753adc9ab9c" exitCode=0
Nov 03 20:04:47.091253 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:47.091216    5570 generic.go:356] "Generic (PLEG): container finished" podID="f7d30630-43af-4157-b90f-d7eddef25085" containerID="0ff023e230ba0d28d23d2743e9b0aba7c73e86105e55e4944b6cb93cc8a89c71" exitCode=0
Nov 03 20:04:47.091253 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:47.091235    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f7d30630-43af-4157-b90f-d7eddef25085","Type":"ContainerDied","Data":"e2504655bc35f9bba51ad6d1bfb0b7293d1790ba3ebcd18215ab07aea2386a41"}
Nov 03 20:04:47.091593 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:47.091315    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f7d30630-43af-4157-b90f-d7eddef25085","Type":"ContainerDied","Data":"960a709deb8999b7ce32bdc8df50230a2b1c396c9f4b7fec4565ce238051471c"}
Nov 03 20:04:47.091593 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:47.091351    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f7d30630-43af-4157-b90f-d7eddef25085","Type":"ContainerDied","Data":"014278ced716c2a52d7b544d736e420472f4440af29a73b56a93341b99fc7d51"}
Nov 03 20:04:47.091593 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:47.091389    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f7d30630-43af-4157-b90f-d7eddef25085","Type":"ContainerDied","Data":"26307d79ee47d0ad8409c23eb27f0d2e6e77b80f6bcc80c48a111753adc9ab9c"}
Nov 03 20:04:47.091593 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:47.091423    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f7d30630-43af-4157-b90f-d7eddef25085","Type":"ContainerDied","Data":"0ff023e230ba0d28d23d2743e9b0aba7c73e86105e55e4944b6cb93cc8a89c71"}
Nov 03 20:04:48.078309 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:48.078285    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:04:48.097123 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:48.097091    5570 generic.go:356] "Generic (PLEG): container finished" podID="f7d30630-43af-4157-b90f-d7eddef25085" containerID="6bbb13e30ca02316f2656a078c3631aa8a4bee6760823627343e629d39e7dda7" exitCode=0
Nov 03 20:04:48.097257 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:48.097176    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:04:48.097257 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:48.097168    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f7d30630-43af-4157-b90f-d7eddef25085","Type":"ContainerDied","Data":"6bbb13e30ca02316f2656a078c3631aa8a4bee6760823627343e629d39e7dda7"}
Nov 03 20:04:48.097345 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:48.097331    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f7d30630-43af-4157-b90f-d7eddef25085","Type":"ContainerDied","Data":"8ef0ef34d0b75303f53cea650dd4280a76b4f1e5813cd730e128a5fd3b3b726e"}
Nov 03 20:04:48.097387 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:48.097365    5570 scope.go:122] "RemoveContainer" containerID="e2504655bc35f9bba51ad6d1bfb0b7293d1790ba3ebcd18215ab07aea2386a41"
Nov 03 20:04:48.105176 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:48.105158    5570 scope.go:122] "RemoveContainer" containerID="960a709deb8999b7ce32bdc8df50230a2b1c396c9f4b7fec4565ce238051471c"
Nov 03 20:04:48.112055 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:48.112040    5570 scope.go:122] "RemoveContainer" containerID="014278ced716c2a52d7b544d736e420472f4440af29a73b56a93341b99fc7d51"
Nov 03 20:04:48.118494 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:48.118479    5570 scope.go:122] "RemoveContainer" containerID="6bbb13e30ca02316f2656a078c3631aa8a4bee6760823627343e629d39e7dda7"
Nov 03 20:04:48.124948 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:48.124929    5570 scope.go:122] "RemoveContainer" containerID="26307d79ee47d0ad8409c23eb27f0d2e6e77b80f6bcc80c48a111753adc9ab9c"
Nov 03 20:04:48.131663 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:48.131648    5570 scope.go:122] "RemoveContainer" containerID="0ff023e230ba0d28d23d2743e9b0aba7c73e86105e55e4944b6cb93cc8a89c71"
Nov 03 20:04:48.138380 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:48.138360    5570 scope.go:122] "RemoveContainer" containerID="785eedffac63800711a89514aec53147c740e3e22adcbae196ce94713ef27782"
Nov 03 20:04:48.142512 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:48.142496    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f7d30630-43af-4157-b90f-d7eddef25085-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f7d30630-43af-4157-b90f-d7eddef25085-secret-alertmanager-kube-rbac-proxy-metric\") pod \"f7d30630-43af-4157-b90f-d7eddef25085\" (UID: \"f7d30630-43af-4157-b90f-d7eddef25085\") "
Nov 03 20:04:48.142614 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:48.142526    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7d30630-43af-4157-b90f-d7eddef25085-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f7d30630-43af-4157-b90f-d7eddef25085-alertmanager-main-db\") pod \"f7d30630-43af-4157-b90f-d7eddef25085\" (UID: \"f7d30630-43af-4157-b90f-d7eddef25085\") "
Nov 03 20:04:48.142614 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:48.142560    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7d30630-43af-4157-b90f-d7eddef25085-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f7d30630-43af-4157-b90f-d7eddef25085-metrics-client-ca\") pod \"f7d30630-43af-4157-b90f-d7eddef25085\" (UID: \"f7d30630-43af-4157-b90f-d7eddef25085\") "
Nov 03 20:04:48.142614 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:48.142586    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f7d30630-43af-4157-b90f-d7eddef25085-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f7d30630-43af-4157-b90f-d7eddef25085-secret-alertmanager-main-tls\") pod \"f7d30630-43af-4157-b90f-d7eddef25085\" (UID: \"f7d30630-43af-4157-b90f-d7eddef25085\") "
Nov 03 20:04:48.142791 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:48.142612    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7d30630-43af-4157-b90f-d7eddef25085-tls-assets\" (UniqueName: \"kubernetes.io/projected/f7d30630-43af-4157-b90f-d7eddef25085-tls-assets\") pod \"f7d30630-43af-4157-b90f-d7eddef25085\" (UID: \"f7d30630-43af-4157-b90f-d7eddef25085\") "
Nov 03 20:04:48.142791 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:48.142642    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f7d30630-43af-4157-b90f-d7eddef25085-config-volume\" (UniqueName: \"kubernetes.io/secret/f7d30630-43af-4157-b90f-d7eddef25085-config-volume\") pod \"f7d30630-43af-4157-b90f-d7eddef25085\" (UID: \"f7d30630-43af-4157-b90f-d7eddef25085\") "
Nov 03 20:04:48.142791 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:48.142691    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7d30630-43af-4157-b90f-d7eddef25085-kube-api-access-z67jw\" (UniqueName: \"kubernetes.io/projected/f7d30630-43af-4157-b90f-d7eddef25085-kube-api-access-z67jw\") pod \"f7d30630-43af-4157-b90f-d7eddef25085\" (UID: \"f7d30630-43af-4157-b90f-d7eddef25085\") "
Nov 03 20:04:48.142791 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:48.142723    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7d30630-43af-4157-b90f-d7eddef25085-config-out\" (UniqueName: \"kubernetes.io/empty-dir/f7d30630-43af-4157-b90f-d7eddef25085-config-out\") pod \"f7d30630-43af-4157-b90f-d7eddef25085\" (UID: \"f7d30630-43af-4157-b90f-d7eddef25085\") "
Nov 03 20:04:48.142791 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:48.142756    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f7d30630-43af-4157-b90f-d7eddef25085-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f7d30630-43af-4157-b90f-d7eddef25085-secret-alertmanager-kube-rbac-proxy-web\") pod \"f7d30630-43af-4157-b90f-d7eddef25085\" (UID: \"f7d30630-43af-4157-b90f-d7eddef25085\") "
Nov 03 20:04:48.143097 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:48.142898    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f7d30630-43af-4157-b90f-d7eddef25085-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f7d30630-43af-4157-b90f-d7eddef25085-cluster-tls-config\") pod \"f7d30630-43af-4157-b90f-d7eddef25085\" (UID: \"f7d30630-43af-4157-b90f-d7eddef25085\") "
Nov 03 20:04:48.143097 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:48.142949    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f7d30630-43af-4157-b90f-d7eddef25085-web-config\" (UniqueName: \"kubernetes.io/secret/f7d30630-43af-4157-b90f-d7eddef25085-web-config\") pod \"f7d30630-43af-4157-b90f-d7eddef25085\" (UID: \"f7d30630-43af-4157-b90f-d7eddef25085\") "
Nov 03 20:04:48.143097 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:48.142989    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f7d30630-43af-4157-b90f-d7eddef25085-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f7d30630-43af-4157-b90f-d7eddef25085-secret-alertmanager-kube-rbac-proxy\") pod \"f7d30630-43af-4157-b90f-d7eddef25085\" (UID: \"f7d30630-43af-4157-b90f-d7eddef25085\") "
Nov 03 20:04:48.143097 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:48.143036    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7d30630-43af-4157-b90f-d7eddef25085-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f7d30630-43af-4157-b90f-d7eddef25085-alertmanager-trusted-ca-bundle\") pod \"f7d30630-43af-4157-b90f-d7eddef25085\" (UID: \"f7d30630-43af-4157-b90f-d7eddef25085\") "
Nov 03 20:04:48.143754 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:48.142893    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7d30630-43af-4157-b90f-d7eddef25085-alertmanager-main-db" pod "f7d30630-43af-4157-b90f-d7eddef25085" (UID: "f7d30630-43af-4157-b90f-d7eddef25085"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:04:48.143754 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:48.142933    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7d30630-43af-4157-b90f-d7eddef25085-metrics-client-ca" pod "f7d30630-43af-4157-b90f-d7eddef25085" (UID: "f7d30630-43af-4157-b90f-d7eddef25085"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:04:48.144353 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:48.143974    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7d30630-43af-4157-b90f-d7eddef25085-alertmanager-trusted-ca-bundle" pod "f7d30630-43af-4157-b90f-d7eddef25085" (UID: "f7d30630-43af-4157-b90f-d7eddef25085"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:04:48.146065 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:48.146003    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7d30630-43af-4157-b90f-d7eddef25085-secret-alertmanager-kube-rbac-proxy-web" pod "f7d30630-43af-4157-b90f-d7eddef25085" (UID: "f7d30630-43af-4157-b90f-d7eddef25085"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:04:48.146277 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:48.146253    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7d30630-43af-4157-b90f-d7eddef25085-secret-alertmanager-main-tls" pod "f7d30630-43af-4157-b90f-d7eddef25085" (UID: "f7d30630-43af-4157-b90f-d7eddef25085"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:04:48.146392 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:48.146329    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7d30630-43af-4157-b90f-d7eddef25085-kube-api-access-z67jw" pod "f7d30630-43af-4157-b90f-d7eddef25085" (UID: "f7d30630-43af-4157-b90f-d7eddef25085"). InnerVolumeSpecName "kube-api-access-z67jw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:04:48.146561 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:48.146439    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7d30630-43af-4157-b90f-d7eddef25085-config-volume" pod "f7d30630-43af-4157-b90f-d7eddef25085" (UID: "f7d30630-43af-4157-b90f-d7eddef25085"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:04:48.146561 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:48.146458    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7d30630-43af-4157-b90f-d7eddef25085-tls-assets" pod "f7d30630-43af-4157-b90f-d7eddef25085" (UID: "f7d30630-43af-4157-b90f-d7eddef25085"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:04:48.146561 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:48.146504    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7d30630-43af-4157-b90f-d7eddef25085-secret-alertmanager-kube-rbac-proxy-metric" pod "f7d30630-43af-4157-b90f-d7eddef25085" (UID: "f7d30630-43af-4157-b90f-d7eddef25085"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:04:48.146983 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:48.146609    5570 scope.go:122] "RemoveContainer" containerID="e2504655bc35f9bba51ad6d1bfb0b7293d1790ba3ebcd18215ab07aea2386a41"
Nov 03 20:04:48.146983 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:04:48.146942    5570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2504655bc35f9bba51ad6d1bfb0b7293d1790ba3ebcd18215ab07aea2386a41\": container with ID starting with e2504655bc35f9bba51ad6d1bfb0b7293d1790ba3ebcd18215ab07aea2386a41 not found: ID does not exist" containerID="e2504655bc35f9bba51ad6d1bfb0b7293d1790ba3ebcd18215ab07aea2386a41"
Nov 03 20:04:48.147102 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:48.146982    5570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2504655bc35f9bba51ad6d1bfb0b7293d1790ba3ebcd18215ab07aea2386a41"} err="failed to get container status \"e2504655bc35f9bba51ad6d1bfb0b7293d1790ba3ebcd18215ab07aea2386a41\": rpc error: code = NotFound desc = could not find container \"e2504655bc35f9bba51ad6d1bfb0b7293d1790ba3ebcd18215ab07aea2386a41\": container with ID starting with e2504655bc35f9bba51ad6d1bfb0b7293d1790ba3ebcd18215ab07aea2386a41 not found: ID does not exist"
Nov 03 20:04:48.147102 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:48.147027    5570 scope.go:122] "RemoveContainer" containerID="960a709deb8999b7ce32bdc8df50230a2b1c396c9f4b7fec4565ce238051471c"
Nov 03 20:04:48.147335 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:04:48.147312    5570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"960a709deb8999b7ce32bdc8df50230a2b1c396c9f4b7fec4565ce238051471c\": container with ID starting with 960a709deb8999b7ce32bdc8df50230a2b1c396c9f4b7fec4565ce238051471c not found: ID does not exist" containerID="960a709deb8999b7ce32bdc8df50230a2b1c396c9f4b7fec4565ce238051471c"
Nov 03 20:04:48.147414 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:48.147345    5570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"960a709deb8999b7ce32bdc8df50230a2b1c396c9f4b7fec4565ce238051471c"} err="failed to get container status \"960a709deb8999b7ce32bdc8df50230a2b1c396c9f4b7fec4565ce238051471c\": rpc error: code = NotFound desc = could not find container \"960a709deb8999b7ce32bdc8df50230a2b1c396c9f4b7fec4565ce238051471c\": container with ID starting with 960a709deb8999b7ce32bdc8df50230a2b1c396c9f4b7fec4565ce238051471c not found: ID does not exist"
Nov 03 20:04:48.147414 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:48.147367    5570 scope.go:122] "RemoveContainer" containerID="014278ced716c2a52d7b544d736e420472f4440af29a73b56a93341b99fc7d51"
Nov 03 20:04:48.147414 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:48.147375    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7d30630-43af-4157-b90f-d7eddef25085-secret-alertmanager-kube-rbac-proxy" pod "f7d30630-43af-4157-b90f-d7eddef25085" (UID: "f7d30630-43af-4157-b90f-d7eddef25085"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:04:48.147637 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:04:48.147613    5570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"014278ced716c2a52d7b544d736e420472f4440af29a73b56a93341b99fc7d51\": container with ID starting with 014278ced716c2a52d7b544d736e420472f4440af29a73b56a93341b99fc7d51 not found: ID does not exist" containerID="014278ced716c2a52d7b544d736e420472f4440af29a73b56a93341b99fc7d51"
Nov 03 20:04:48.147706 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:48.147647    5570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"014278ced716c2a52d7b544d736e420472f4440af29a73b56a93341b99fc7d51"} err="failed to get container status \"014278ced716c2a52d7b544d736e420472f4440af29a73b56a93341b99fc7d51\": rpc error: code = NotFound desc = could not find container \"014278ced716c2a52d7b544d736e420472f4440af29a73b56a93341b99fc7d51\": container with ID starting with 014278ced716c2a52d7b544d736e420472f4440af29a73b56a93341b99fc7d51 not found: ID does not exist"
Nov 03 20:04:48.147706 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:48.147668    5570 scope.go:122] "RemoveContainer" containerID="6bbb13e30ca02316f2656a078c3631aa8a4bee6760823627343e629d39e7dda7"
Nov 03 20:04:48.147952 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:04:48.147930    5570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6bbb13e30ca02316f2656a078c3631aa8a4bee6760823627343e629d39e7dda7\": container with ID starting with 6bbb13e30ca02316f2656a078c3631aa8a4bee6760823627343e629d39e7dda7 not found: ID does not exist" containerID="6bbb13e30ca02316f2656a078c3631aa8a4bee6760823627343e629d39e7dda7"
Nov 03 20:04:48.148016 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:48.147961    5570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bbb13e30ca02316f2656a078c3631aa8a4bee6760823627343e629d39e7dda7"} err="failed to get container status \"6bbb13e30ca02316f2656a078c3631aa8a4bee6760823627343e629d39e7dda7\": rpc error: code = NotFound desc = could not find container \"6bbb13e30ca02316f2656a078c3631aa8a4bee6760823627343e629d39e7dda7\": container with ID starting with 6bbb13e30ca02316f2656a078c3631aa8a4bee6760823627343e629d39e7dda7 not found: ID does not exist"
Nov 03 20:04:48.148016 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:48.147982    5570 scope.go:122] "RemoveContainer" containerID="26307d79ee47d0ad8409c23eb27f0d2e6e77b80f6bcc80c48a111753adc9ab9c"
Nov 03 20:04:48.148260 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:48.148194    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7d30630-43af-4157-b90f-d7eddef25085-config-out" pod "f7d30630-43af-4157-b90f-d7eddef25085" (UID: "f7d30630-43af-4157-b90f-d7eddef25085"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:04:48.148368 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:04:48.148268    5570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26307d79ee47d0ad8409c23eb27f0d2e6e77b80f6bcc80c48a111753adc9ab9c\": container with ID starting with 26307d79ee47d0ad8409c23eb27f0d2e6e77b80f6bcc80c48a111753adc9ab9c not found: ID does not exist" containerID="26307d79ee47d0ad8409c23eb27f0d2e6e77b80f6bcc80c48a111753adc9ab9c"
Nov 03 20:04:48.148368 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:48.148295    5570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26307d79ee47d0ad8409c23eb27f0d2e6e77b80f6bcc80c48a111753adc9ab9c"} err="failed to get container status \"26307d79ee47d0ad8409c23eb27f0d2e6e77b80f6bcc80c48a111753adc9ab9c\": rpc error: code = NotFound desc = could not find container \"26307d79ee47d0ad8409c23eb27f0d2e6e77b80f6bcc80c48a111753adc9ab9c\": container with ID starting with 26307d79ee47d0ad8409c23eb27f0d2e6e77b80f6bcc80c48a111753adc9ab9c not found: ID does not exist"
Nov 03 20:04:48.148368 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:48.148314    5570 scope.go:122] "RemoveContainer" containerID="0ff023e230ba0d28d23d2743e9b0aba7c73e86105e55e4944b6cb93cc8a89c71"
Nov 03 20:04:48.148636 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:04:48.148612    5570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ff023e230ba0d28d23d2743e9b0aba7c73e86105e55e4944b6cb93cc8a89c71\": container with ID starting with 0ff023e230ba0d28d23d2743e9b0aba7c73e86105e55e4944b6cb93cc8a89c71 not found: ID does not exist" containerID="0ff023e230ba0d28d23d2743e9b0aba7c73e86105e55e4944b6cb93cc8a89c71"
Nov 03 20:04:48.148845 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:48.148767    5570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ff023e230ba0d28d23d2743e9b0aba7c73e86105e55e4944b6cb93cc8a89c71"} err="failed to get container status \"0ff023e230ba0d28d23d2743e9b0aba7c73e86105e55e4944b6cb93cc8a89c71\": rpc error: code = NotFound desc = could not find container \"0ff023e230ba0d28d23d2743e9b0aba7c73e86105e55e4944b6cb93cc8a89c71\": container with ID starting with 0ff023e230ba0d28d23d2743e9b0aba7c73e86105e55e4944b6cb93cc8a89c71 not found: ID does not exist"
Nov 03 20:04:48.148953 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:48.148846    5570 scope.go:122] "RemoveContainer" containerID="785eedffac63800711a89514aec53147c740e3e22adcbae196ce94713ef27782"
Nov 03 20:04:48.149157 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:04:48.149135    5570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"785eedffac63800711a89514aec53147c740e3e22adcbae196ce94713ef27782\": container with ID starting with 785eedffac63800711a89514aec53147c740e3e22adcbae196ce94713ef27782 not found: ID does not exist" containerID="785eedffac63800711a89514aec53147c740e3e22adcbae196ce94713ef27782"
Nov 03 20:04:48.149219 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:48.149164    5570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"785eedffac63800711a89514aec53147c740e3e22adcbae196ce94713ef27782"} err="failed to get container status \"785eedffac63800711a89514aec53147c740e3e22adcbae196ce94713ef27782\": rpc error: code = NotFound desc = could not find container \"785eedffac63800711a89514aec53147c740e3e22adcbae196ce94713ef27782\": container with ID starting with 785eedffac63800711a89514aec53147c740e3e22adcbae196ce94713ef27782 not found: ID does not exist"
Nov 03 20:04:48.152090 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:48.152064    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7d30630-43af-4157-b90f-d7eddef25085-cluster-tls-config" pod "f7d30630-43af-4157-b90f-d7eddef25085" (UID: "f7d30630-43af-4157-b90f-d7eddef25085"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:04:48.158431 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:48.158411    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7d30630-43af-4157-b90f-d7eddef25085-web-config" pod "f7d30630-43af-4157-b90f-d7eddef25085" (UID: "f7d30630-43af-4157-b90f-d7eddef25085"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:04:48.244237 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:48.244199    5570 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f7d30630-43af-4157-b90f-d7eddef25085-config-volume\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:04:48.244237 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:48.244230    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z67jw\" (UniqueName: \"kubernetes.io/projected/f7d30630-43af-4157-b90f-d7eddef25085-kube-api-access-z67jw\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:04:48.244237 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:48.244241    5570 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f7d30630-43af-4157-b90f-d7eddef25085-config-out\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:04:48.244483 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:48.244251    5570 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f7d30630-43af-4157-b90f-d7eddef25085-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:04:48.244483 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:48.244260    5570 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f7d30630-43af-4157-b90f-d7eddef25085-cluster-tls-config\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:04:48.244483 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:48.244292    5570 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f7d30630-43af-4157-b90f-d7eddef25085-web-config\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:04:48.244483 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:48.244302    5570 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f7d30630-43af-4157-b90f-d7eddef25085-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:04:48.244483 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:48.244311    5570 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f7d30630-43af-4157-b90f-d7eddef25085-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:04:48.244483 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:48.244320    5570 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f7d30630-43af-4157-b90f-d7eddef25085-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:04:48.244483 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:48.244330    5570 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f7d30630-43af-4157-b90f-d7eddef25085-alertmanager-main-db\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:04:48.244483 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:48.244339    5570 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f7d30630-43af-4157-b90f-d7eddef25085-metrics-client-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:04:48.244483 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:48.244347    5570 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f7d30630-43af-4157-b90f-d7eddef25085-secret-alertmanager-main-tls\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:04:48.244483 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:48.244355    5570 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f7d30630-43af-4157-b90f-d7eddef25085-tls-assets\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:04:48.442871 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:48.442840    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 20:04:48.455145 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:48.455118    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 20:04:48.469034 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:48.468988    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 20:04:48.469300 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:48.469284    5570 cpu_manager.go:414] "containerMap: removing container" podUID="f7d30630-43af-4157-b90f-d7eddef25085" containerName="kube-rbac-proxy"
Nov 03 20:04:48.469362 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:48.469303    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7d30630-43af-4157-b90f-d7eddef25085" containerName="kube-rbac-proxy"
Nov 03 20:04:48.469362 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:48.469317    5570 cpu_manager.go:414] "containerMap: removing container" podUID="f7d30630-43af-4157-b90f-d7eddef25085" containerName="config-reloader"
Nov 03 20:04:48.469362 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:48.469325    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7d30630-43af-4157-b90f-d7eddef25085" containerName="config-reloader"
Nov 03 20:04:48.469362 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:48.469345    5570 cpu_manager.go:414] "containerMap: removing container" podUID="f7d30630-43af-4157-b90f-d7eddef25085" containerName="alertmanager"
Nov 03 20:04:48.469362 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:48.469350    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7d30630-43af-4157-b90f-d7eddef25085" containerName="alertmanager"
Nov 03 20:04:48.469362 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:48.469360    5570 cpu_manager.go:414] "containerMap: removing container" podUID="f7d30630-43af-4157-b90f-d7eddef25085" containerName="kube-rbac-proxy-metric"
Nov 03 20:04:48.469561 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:48.469365    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7d30630-43af-4157-b90f-d7eddef25085" containerName="kube-rbac-proxy-metric"
Nov 03 20:04:48.469561 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:48.469378    5570 cpu_manager.go:414] "containerMap: removing container" podUID="f7d30630-43af-4157-b90f-d7eddef25085" containerName="init-config-reloader"
Nov 03 20:04:48.469561 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:48.469387    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7d30630-43af-4157-b90f-d7eddef25085" containerName="init-config-reloader"
Nov 03 20:04:48.469561 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:48.469397    5570 cpu_manager.go:414] "containerMap: removing container" podUID="f7d30630-43af-4157-b90f-d7eddef25085" containerName="kube-rbac-proxy-web"
Nov 03 20:04:48.469561 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:48.469401    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7d30630-43af-4157-b90f-d7eddef25085" containerName="kube-rbac-proxy-web"
Nov 03 20:04:48.469561 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:48.469408    5570 cpu_manager.go:414] "containerMap: removing container" podUID="f7d30630-43af-4157-b90f-d7eddef25085" containerName="prom-label-proxy"
Nov 03 20:04:48.469561 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:48.469414    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7d30630-43af-4157-b90f-d7eddef25085" containerName="prom-label-proxy"
Nov 03 20:04:48.469561 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:48.469459    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7d30630-43af-4157-b90f-d7eddef25085" containerName="alertmanager"
Nov 03 20:04:48.469561 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:48.469469    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7d30630-43af-4157-b90f-d7eddef25085" containerName="kube-rbac-proxy-metric"
Nov 03 20:04:48.469561 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:48.469475    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7d30630-43af-4157-b90f-d7eddef25085" containerName="init-config-reloader"
Nov 03 20:04:48.469561 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:48.469481    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7d30630-43af-4157-b90f-d7eddef25085" containerName="kube-rbac-proxy-web"
Nov 03 20:04:48.469561 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:48.469486    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7d30630-43af-4157-b90f-d7eddef25085" containerName="prom-label-proxy"
Nov 03 20:04:48.469561 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:48.469492    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7d30630-43af-4157-b90f-d7eddef25085" containerName="kube-rbac-proxy"
Nov 03 20:04:48.469561 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:48.469498    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7d30630-43af-4157-b90f-d7eddef25085" containerName="config-reloader"
Nov 03 20:04:48.473137 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:48.473119    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:04:48.488629 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:48.488603    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Nov 03 20:04:48.488732 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:48.488632    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Nov 03 20:04:48.488732 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:48.488634    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Nov 03 20:04:48.488732 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:48.488727    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Nov 03 20:04:48.488931 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:48.488742    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Nov 03 20:04:48.488931 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:48.488746    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Nov 03 20:04:48.488931 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:48.488765    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Nov 03 20:04:48.488931 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:48.488799    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Nov 03 20:04:48.489068 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:48.488972    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-rnjcd\""
Nov 03 20:04:48.501524 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:48.501503    5570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Nov 03 20:04:48.502086 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:48.502065    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 20:04:48.546283 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:48.546251    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4727cec2-c7e7-4965-ae91-a78dfc4ec7f5-tls-assets\") pod \"alertmanager-main-0\" (UID: \"4727cec2-c7e7-4965-ae91-a78dfc4ec7f5\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:04:48.546435 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:48.546293    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4727cec2-c7e7-4965-ae91-a78dfc4ec7f5-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"4727cec2-c7e7-4965-ae91-a78dfc4ec7f5\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:04:48.546435 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:48.546323    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/4727cec2-c7e7-4965-ae91-a78dfc4ec7f5-config-volume\") pod \"alertmanager-main-0\" (UID: \"4727cec2-c7e7-4965-ae91-a78dfc4ec7f5\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:04:48.546516 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:48.546429    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/4727cec2-c7e7-4965-ae91-a78dfc4ec7f5-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"4727cec2-c7e7-4965-ae91-a78dfc4ec7f5\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:04:48.546516 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:48.546475    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mwz4\" (UniqueName: \"kubernetes.io/projected/4727cec2-c7e7-4965-ae91-a78dfc4ec7f5-kube-api-access-5mwz4\") pod \"alertmanager-main-0\" (UID: \"4727cec2-c7e7-4965-ae91-a78dfc4ec7f5\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:04:48.546516 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:48.546498    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4727cec2-c7e7-4965-ae91-a78dfc4ec7f5-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"4727cec2-c7e7-4965-ae91-a78dfc4ec7f5\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:04:48.546516 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:48.546514    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4727cec2-c7e7-4965-ae91-a78dfc4ec7f5-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"4727cec2-c7e7-4965-ae91-a78dfc4ec7f5\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:04:48.546661 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:48.546541    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4727cec2-c7e7-4965-ae91-a78dfc4ec7f5-config-out\") pod \"alertmanager-main-0\" (UID: \"4727cec2-c7e7-4965-ae91-a78dfc4ec7f5\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:04:48.546661 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:48.546627    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4727cec2-c7e7-4965-ae91-a78dfc4ec7f5-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"4727cec2-c7e7-4965-ae91-a78dfc4ec7f5\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:04:48.546725 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:48.546685    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4727cec2-c7e7-4965-ae91-a78dfc4ec7f5-web-config\") pod \"alertmanager-main-0\" (UID: \"4727cec2-c7e7-4965-ae91-a78dfc4ec7f5\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:04:48.546725 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:48.546704    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4727cec2-c7e7-4965-ae91-a78dfc4ec7f5-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"4727cec2-c7e7-4965-ae91-a78dfc4ec7f5\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:04:48.546835 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:48.546726    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/4727cec2-c7e7-4965-ae91-a78dfc4ec7f5-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"4727cec2-c7e7-4965-ae91-a78dfc4ec7f5\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:04:48.546835 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:48.546800    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/4727cec2-c7e7-4965-ae91-a78dfc4ec7f5-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"4727cec2-c7e7-4965-ae91-a78dfc4ec7f5\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:04:48.647604 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:48.647569    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4727cec2-c7e7-4965-ae91-a78dfc4ec7f5-tls-assets\") pod \"alertmanager-main-0\" (UID: \"4727cec2-c7e7-4965-ae91-a78dfc4ec7f5\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:04:48.647604 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:48.647608    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4727cec2-c7e7-4965-ae91-a78dfc4ec7f5-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"4727cec2-c7e7-4965-ae91-a78dfc4ec7f5\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:04:48.647886 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:48.647625    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/4727cec2-c7e7-4965-ae91-a78dfc4ec7f5-config-volume\") pod \"alertmanager-main-0\" (UID: \"4727cec2-c7e7-4965-ae91-a78dfc4ec7f5\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:04:48.647886 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:48.647644    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/4727cec2-c7e7-4965-ae91-a78dfc4ec7f5-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"4727cec2-c7e7-4965-ae91-a78dfc4ec7f5\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:04:48.647886 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:48.647673    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mwz4\" (UniqueName: \"kubernetes.io/projected/4727cec2-c7e7-4965-ae91-a78dfc4ec7f5-kube-api-access-5mwz4\") pod \"alertmanager-main-0\" (UID: \"4727cec2-c7e7-4965-ae91-a78dfc4ec7f5\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:04:48.647886 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:48.647696    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4727cec2-c7e7-4965-ae91-a78dfc4ec7f5-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"4727cec2-c7e7-4965-ae91-a78dfc4ec7f5\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:04:48.647886 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:48.647721    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4727cec2-c7e7-4965-ae91-a78dfc4ec7f5-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"4727cec2-c7e7-4965-ae91-a78dfc4ec7f5\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:04:48.647886 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:48.647751    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4727cec2-c7e7-4965-ae91-a78dfc4ec7f5-config-out\") pod \"alertmanager-main-0\" (UID: \"4727cec2-c7e7-4965-ae91-a78dfc4ec7f5\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:04:48.647886 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:48.647800    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4727cec2-c7e7-4965-ae91-a78dfc4ec7f5-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"4727cec2-c7e7-4965-ae91-a78dfc4ec7f5\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:04:48.647886 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:48.647830    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4727cec2-c7e7-4965-ae91-a78dfc4ec7f5-web-config\") pod \"alertmanager-main-0\" (UID: \"4727cec2-c7e7-4965-ae91-a78dfc4ec7f5\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:04:48.647886 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:48.647857    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4727cec2-c7e7-4965-ae91-a78dfc4ec7f5-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"4727cec2-c7e7-4965-ae91-a78dfc4ec7f5\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:04:48.647886 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:48.647889    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/4727cec2-c7e7-4965-ae91-a78dfc4ec7f5-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"4727cec2-c7e7-4965-ae91-a78dfc4ec7f5\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:04:48.648355 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:48.647911    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/4727cec2-c7e7-4965-ae91-a78dfc4ec7f5-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"4727cec2-c7e7-4965-ae91-a78dfc4ec7f5\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:04:48.648355 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:48.648115    5570 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 03 20:04:48.648355 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:48.648296    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/4727cec2-c7e7-4965-ae91-a78dfc4ec7f5-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"4727cec2-c7e7-4965-ae91-a78dfc4ec7f5\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:04:48.648961 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:48.648933    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4727cec2-c7e7-4965-ae91-a78dfc4ec7f5-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"4727cec2-c7e7-4965-ae91-a78dfc4ec7f5\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:04:48.650887 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:48.650863    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4727cec2-c7e7-4965-ae91-a78dfc4ec7f5-config-out\") pod \"alertmanager-main-0\" (UID: \"4727cec2-c7e7-4965-ae91-a78dfc4ec7f5\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:04:48.650983 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:48.650968    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4727cec2-c7e7-4965-ae91-a78dfc4ec7f5-tls-assets\") pod \"alertmanager-main-0\" (UID: \"4727cec2-c7e7-4965-ae91-a78dfc4ec7f5\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:04:48.651041 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:48.650992    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4727cec2-c7e7-4965-ae91-a78dfc4ec7f5-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"4727cec2-c7e7-4965-ae91-a78dfc4ec7f5\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:04:48.651198 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:48.651170    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4727cec2-c7e7-4965-ae91-a78dfc4ec7f5-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"4727cec2-c7e7-4965-ae91-a78dfc4ec7f5\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:04:48.651315 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:48.651205    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/4727cec2-c7e7-4965-ae91-a78dfc4ec7f5-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"4727cec2-c7e7-4965-ae91-a78dfc4ec7f5\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:04:48.651315 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:48.651204    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4727cec2-c7e7-4965-ae91-a78dfc4ec7f5-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"4727cec2-c7e7-4965-ae91-a78dfc4ec7f5\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:04:48.651315 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:48.651272    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4727cec2-c7e7-4965-ae91-a78dfc4ec7f5-web-config\") pod \"alertmanager-main-0\" (UID: \"4727cec2-c7e7-4965-ae91-a78dfc4ec7f5\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:04:48.651475 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:48.651397    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4727cec2-c7e7-4965-ae91-a78dfc4ec7f5-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"4727cec2-c7e7-4965-ae91-a78dfc4ec7f5\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:04:48.651593 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:48.651574    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/4727cec2-c7e7-4965-ae91-a78dfc4ec7f5-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"4727cec2-c7e7-4965-ae91-a78dfc4ec7f5\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:04:48.652835 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:48.652811    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/4727cec2-c7e7-4965-ae91-a78dfc4ec7f5-config-volume\") pod \"alertmanager-main-0\" (UID: \"4727cec2-c7e7-4965-ae91-a78dfc4ec7f5\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:04:48.666979 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:48.666930    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mwz4\" (UniqueName: \"kubernetes.io/projected/4727cec2-c7e7-4965-ae91-a78dfc4ec7f5-kube-api-access-5mwz4\") pod \"alertmanager-main-0\" (UID: \"4727cec2-c7e7-4965-ae91-a78dfc4ec7f5\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:04:48.790160 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:48.790110    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 03 20:04:49.113616 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:49.113555    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 20:04:49.117957 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:49.117919    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4727cec2_c7e7_4965_ae91_a78dfc4ec7f5.slice/crio-083a76d1a18e3afeec094aba064f3dd54fa0c1a29d6324f423c8ec8e437ca24f WatchSource:0}: Error finding container 083a76d1a18e3afeec094aba064f3dd54fa0c1a29d6324f423c8ec8e437ca24f: Status 404 returned error can't find the container with id 083a76d1a18e3afeec094aba064f3dd54fa0c1a29d6324f423c8ec8e437ca24f
Nov 03 20:04:49.961914 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:49.961879    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7d30630-43af-4157-b90f-d7eddef25085" path="/var/lib/kubelet/pods/f7d30630-43af-4157-b90f-d7eddef25085/volumes"
Nov 03 20:04:50.050020 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:50.049985    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6sv7j"
Nov 03 20:04:50.107280 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:50.107244    5570 generic.go:356] "Generic (PLEG): container finished" podID="4727cec2-c7e7-4965-ae91-a78dfc4ec7f5" containerID="743b92c425f8d56f26ea6f32f9100bdd6e7daf96436feea1781e898d69213678" exitCode=0
Nov 03 20:04:50.107444 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:50.107336    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4727cec2-c7e7-4965-ae91-a78dfc4ec7f5","Type":"ContainerDied","Data":"743b92c425f8d56f26ea6f32f9100bdd6e7daf96436feea1781e898d69213678"}
Nov 03 20:04:50.107444 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:50.107410    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4727cec2-c7e7-4965-ae91-a78dfc4ec7f5","Type":"ContainerStarted","Data":"083a76d1a18e3afeec094aba064f3dd54fa0c1a29d6324f423c8ec8e437ca24f"}
Nov 03 20:04:50.774616 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:50.774580    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-567ccc7984-qnsvp"]
Nov 03 20:04:50.777022 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:50.777002    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-567ccc7984-qnsvp"
Nov 03 20:04:50.791927 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:50.791905    5570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Nov 03 20:04:50.792151 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:50.792126    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Nov 03 20:04:50.792280 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:50.792165    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Nov 03 20:04:50.804860 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:50.804832    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Nov 03 20:04:50.804978 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:50.804930    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-8vmf6\""
Nov 03 20:04:50.805083 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:50.805065    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Nov 03 20:04:50.821395 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:50.819425    5570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\""
Nov 03 20:04:50.823143 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:50.821899    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-567ccc7984-qnsvp"]
Nov 03 20:04:50.863460 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:50.863427    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4f0d1d22-fa21-4f7c-8330-363e32444e3c-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-567ccc7984-qnsvp\" (UID: \"4f0d1d22-fa21-4f7c-8330-363e32444e3c\") " pod="openshift-monitoring/telemeter-client-567ccc7984-qnsvp"
Nov 03 20:04:50.863609 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:50.863469    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4f0d1d22-fa21-4f7c-8330-363e32444e3c-metrics-client-ca\") pod \"telemeter-client-567ccc7984-qnsvp\" (UID: \"4f0d1d22-fa21-4f7c-8330-363e32444e3c\") " pod="openshift-monitoring/telemeter-client-567ccc7984-qnsvp"
Nov 03 20:04:50.863609 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:50.863504    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4f0d1d22-fa21-4f7c-8330-363e32444e3c-telemeter-trusted-ca-bundle\") pod \"telemeter-client-567ccc7984-qnsvp\" (UID: \"4f0d1d22-fa21-4f7c-8330-363e32444e3c\") " pod="openshift-monitoring/telemeter-client-567ccc7984-qnsvp"
Nov 03 20:04:50.863609 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:50.863556    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5jpk\" (UniqueName: \"kubernetes.io/projected/4f0d1d22-fa21-4f7c-8330-363e32444e3c-kube-api-access-w5jpk\") pod \"telemeter-client-567ccc7984-qnsvp\" (UID: \"4f0d1d22-fa21-4f7c-8330-363e32444e3c\") " pod="openshift-monitoring/telemeter-client-567ccc7984-qnsvp"
Nov 03 20:04:50.863709 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:50.863611    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4f0d1d22-fa21-4f7c-8330-363e32444e3c-serving-certs-ca-bundle\") pod \"telemeter-client-567ccc7984-qnsvp\" (UID: \"4f0d1d22-fa21-4f7c-8330-363e32444e3c\") " pod="openshift-monitoring/telemeter-client-567ccc7984-qnsvp"
Nov 03 20:04:50.863709 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:50.863639    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/4f0d1d22-fa21-4f7c-8330-363e32444e3c-federate-client-tls\") pod \"telemeter-client-567ccc7984-qnsvp\" (UID: \"4f0d1d22-fa21-4f7c-8330-363e32444e3c\") " pod="openshift-monitoring/telemeter-client-567ccc7984-qnsvp"
Nov 03 20:04:50.863709 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:50.863661    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/4f0d1d22-fa21-4f7c-8330-363e32444e3c-secret-telemeter-client\") pod \"telemeter-client-567ccc7984-qnsvp\" (UID: \"4f0d1d22-fa21-4f7c-8330-363e32444e3c\") " pod="openshift-monitoring/telemeter-client-567ccc7984-qnsvp"
Nov 03 20:04:50.863845 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:50.863711    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/4f0d1d22-fa21-4f7c-8330-363e32444e3c-telemeter-client-tls\") pod \"telemeter-client-567ccc7984-qnsvp\" (UID: \"4f0d1d22-fa21-4f7c-8330-363e32444e3c\") " pod="openshift-monitoring/telemeter-client-567ccc7984-qnsvp"
Nov 03 20:04:50.964905 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:50.964869    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4f0d1d22-fa21-4f7c-8330-363e32444e3c-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-567ccc7984-qnsvp\" (UID: \"4f0d1d22-fa21-4f7c-8330-363e32444e3c\") " pod="openshift-monitoring/telemeter-client-567ccc7984-qnsvp"
Nov 03 20:04:50.964905 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:50.964911    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4f0d1d22-fa21-4f7c-8330-363e32444e3c-metrics-client-ca\") pod \"telemeter-client-567ccc7984-qnsvp\" (UID: \"4f0d1d22-fa21-4f7c-8330-363e32444e3c\") " pod="openshift-monitoring/telemeter-client-567ccc7984-qnsvp"
Nov 03 20:04:50.965151 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:50.964939    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4f0d1d22-fa21-4f7c-8330-363e32444e3c-telemeter-trusted-ca-bundle\") pod \"telemeter-client-567ccc7984-qnsvp\" (UID: \"4f0d1d22-fa21-4f7c-8330-363e32444e3c\") " pod="openshift-monitoring/telemeter-client-567ccc7984-qnsvp"
Nov 03 20:04:50.965151 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:50.964957    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5jpk\" (UniqueName: \"kubernetes.io/projected/4f0d1d22-fa21-4f7c-8330-363e32444e3c-kube-api-access-w5jpk\") pod \"telemeter-client-567ccc7984-qnsvp\" (UID: \"4f0d1d22-fa21-4f7c-8330-363e32444e3c\") " pod="openshift-monitoring/telemeter-client-567ccc7984-qnsvp"
Nov 03 20:04:50.965151 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:50.964994    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4f0d1d22-fa21-4f7c-8330-363e32444e3c-serving-certs-ca-bundle\") pod \"telemeter-client-567ccc7984-qnsvp\" (UID: \"4f0d1d22-fa21-4f7c-8330-363e32444e3c\") " pod="openshift-monitoring/telemeter-client-567ccc7984-qnsvp"
Nov 03 20:04:50.965151 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:50.965024    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/4f0d1d22-fa21-4f7c-8330-363e32444e3c-federate-client-tls\") pod \"telemeter-client-567ccc7984-qnsvp\" (UID: \"4f0d1d22-fa21-4f7c-8330-363e32444e3c\") " pod="openshift-monitoring/telemeter-client-567ccc7984-qnsvp"
Nov 03 20:04:50.965151 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:50.965051    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/4f0d1d22-fa21-4f7c-8330-363e32444e3c-secret-telemeter-client\") pod \"telemeter-client-567ccc7984-qnsvp\" (UID: \"4f0d1d22-fa21-4f7c-8330-363e32444e3c\") " pod="openshift-monitoring/telemeter-client-567ccc7984-qnsvp"
Nov 03 20:04:50.965151 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:50.965077    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/4f0d1d22-fa21-4f7c-8330-363e32444e3c-telemeter-client-tls\") pod \"telemeter-client-567ccc7984-qnsvp\" (UID: \"4f0d1d22-fa21-4f7c-8330-363e32444e3c\") " pod="openshift-monitoring/telemeter-client-567ccc7984-qnsvp"
Nov 03 20:04:50.965735 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:50.965703    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4f0d1d22-fa21-4f7c-8330-363e32444e3c-metrics-client-ca\") pod \"telemeter-client-567ccc7984-qnsvp\" (UID: \"4f0d1d22-fa21-4f7c-8330-363e32444e3c\") " pod="openshift-monitoring/telemeter-client-567ccc7984-qnsvp"
Nov 03 20:04:50.965867 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:50.965807    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4f0d1d22-fa21-4f7c-8330-363e32444e3c-serving-certs-ca-bundle\") pod \"telemeter-client-567ccc7984-qnsvp\" (UID: \"4f0d1d22-fa21-4f7c-8330-363e32444e3c\") " pod="openshift-monitoring/telemeter-client-567ccc7984-qnsvp"
Nov 03 20:04:50.965867 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:50.965828    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4f0d1d22-fa21-4f7c-8330-363e32444e3c-telemeter-trusted-ca-bundle\") pod \"telemeter-client-567ccc7984-qnsvp\" (UID: \"4f0d1d22-fa21-4f7c-8330-363e32444e3c\") " pod="openshift-monitoring/telemeter-client-567ccc7984-qnsvp"
Nov 03 20:04:50.967684 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:50.967656    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/4f0d1d22-fa21-4f7c-8330-363e32444e3c-secret-telemeter-client\") pod \"telemeter-client-567ccc7984-qnsvp\" (UID: \"4f0d1d22-fa21-4f7c-8330-363e32444e3c\") " pod="openshift-monitoring/telemeter-client-567ccc7984-qnsvp"
Nov 03 20:04:50.967793 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:50.967729    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/4f0d1d22-fa21-4f7c-8330-363e32444e3c-telemeter-client-tls\") pod \"telemeter-client-567ccc7984-qnsvp\" (UID: \"4f0d1d22-fa21-4f7c-8330-363e32444e3c\") " pod="openshift-monitoring/telemeter-client-567ccc7984-qnsvp"
Nov 03 20:04:50.968028 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:50.968009    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/4f0d1d22-fa21-4f7c-8330-363e32444e3c-federate-client-tls\") pod \"telemeter-client-567ccc7984-qnsvp\" (UID: \"4f0d1d22-fa21-4f7c-8330-363e32444e3c\") " pod="openshift-monitoring/telemeter-client-567ccc7984-qnsvp"
Nov 03 20:04:50.968067 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:50.968030    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4f0d1d22-fa21-4f7c-8330-363e32444e3c-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-567ccc7984-qnsvp\" (UID: \"4f0d1d22-fa21-4f7c-8330-363e32444e3c\") " pod="openshift-monitoring/telemeter-client-567ccc7984-qnsvp"
Nov 03 20:04:50.984121 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:50.984095    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5jpk\" (UniqueName: \"kubernetes.io/projected/4f0d1d22-fa21-4f7c-8330-363e32444e3c-kube-api-access-w5jpk\") pod \"telemeter-client-567ccc7984-qnsvp\" (UID: \"4f0d1d22-fa21-4f7c-8330-363e32444e3c\") " pod="openshift-monitoring/telemeter-client-567ccc7984-qnsvp"
Nov 03 20:04:51.089035 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:51.088956    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-567ccc7984-qnsvp"
Nov 03 20:04:51.114962 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:51.114929    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4727cec2-c7e7-4965-ae91-a78dfc4ec7f5","Type":"ContainerStarted","Data":"a05e678ebb725ab96d1fbdca8954ebf5e0d744f5dca55bca21a538df8385a855"}
Nov 03 20:04:51.115085 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:51.115003    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4727cec2-c7e7-4965-ae91-a78dfc4ec7f5","Type":"ContainerStarted","Data":"5bdeafa5eb6f34572b2295a84b82d44e671969825b7b4861b8f27abd159dfa01"}
Nov 03 20:04:51.115085 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:51.115033    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4727cec2-c7e7-4965-ae91-a78dfc4ec7f5","Type":"ContainerStarted","Data":"f6c49de15773fbb54f7a1965276b402e2276aee713ee38543a01bfaa09fee263"}
Nov 03 20:04:51.115085 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:51.115056    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4727cec2-c7e7-4965-ae91-a78dfc4ec7f5","Type":"ContainerStarted","Data":"545a82decdfacee2dd6824f2b3045a5f63eaa81660901863362c8cf88c532a71"}
Nov 03 20:04:51.115085 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:51.115077    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4727cec2-c7e7-4965-ae91-a78dfc4ec7f5","Type":"ContainerStarted","Data":"feec4c4c3b6417e2afa80989f85f682c91a4507e37a47d6816466832dc37cc24"}
Nov 03 20:04:51.115205 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:51.115099    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4727cec2-c7e7-4965-ae91-a78dfc4ec7f5","Type":"ContainerStarted","Data":"4d839e806701d353147359601bd813d7d619309a35d4aea6d464fd81dd177f6b"}
Nov 03 20:04:51.161362 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:51.161290    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.150549831 podStartE2EDuration="3.161271539s" totalImagesPullingTime="0s" totalInitContainerRuntime="10.721708ms" isStatefulPod=false podCreationTimestamp="2026-11-03 20:04:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:04:51.160499504 +0000 UTC m=+25.313161392" watchObservedRunningTime="2026-11-03 20:04:51.161271539 +0000 UTC m=+25.313933429"
Nov 03 20:04:51.401841 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:51.401741    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-567ccc7984-qnsvp"]
Nov 03 20:04:51.405446 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:04:51.405395    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f0d1d22_fa21_4f7c_8330_363e32444e3c.slice/crio-7e7dd24d930b36ba4c1bb5a66e0f1dee5549dbe0c924b507f2d183bcab51faef WatchSource:0}: Error finding container 7e7dd24d930b36ba4c1bb5a66e0f1dee5549dbe0c924b507f2d183bcab51faef: Status 404 returned error can't find the container with id 7e7dd24d930b36ba4c1bb5a66e0f1dee5549dbe0c924b507f2d183bcab51faef
Nov 03 20:04:51.407682 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:51.407662    5570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 20:04:52.121151 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:52.121115    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-567ccc7984-qnsvp" event={"ID":"4f0d1d22-fa21-4f7c-8330-363e32444e3c","Type":"ContainerStarted","Data":"7e7dd24d930b36ba4c1bb5a66e0f1dee5549dbe0c924b507f2d183bcab51faef"}
Nov 03 20:04:53.125917 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:53.125888    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-567ccc7984-qnsvp" event={"ID":"4f0d1d22-fa21-4f7c-8330-363e32444e3c","Type":"ContainerStarted","Data":"93c77ff3689b69e2adcb2f51bd0c55fb0d0bec0d368c2a4b00175356ae6f6d65"}
Nov 03 20:04:53.126250 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:53.125949    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-567ccc7984-qnsvp" event={"ID":"4f0d1d22-fa21-4f7c-8330-363e32444e3c","Type":"ContainerStarted","Data":"4e52581c7f539496cbae5228c4379a892b1819d9d479876c064acd114c8cc68b"}
Nov 03 20:04:54.131594 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:54.131552    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-567ccc7984-qnsvp" event={"ID":"4f0d1d22-fa21-4f7c-8330-363e32444e3c","Type":"ContainerStarted","Data":"f078048e944f95f2e27f190e0e13eaf96725e2efb1e6d7267b37fa3894fb64e4"}
Nov 03 20:04:54.200532 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:54.200476    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-567ccc7984-qnsvp" podStartSLOduration=2.606075921 podStartE2EDuration="4.200463927s" totalImagesPullingTime="1.594388006s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:04:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:04:54.200149517 +0000 UTC m=+28.352811403" watchObservedRunningTime="2026-11-03 20:04:54.200463927 +0000 UTC m=+28.353125807"
Nov 03 20:04:56.605649 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:04:56.605618    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-6dfr5"
Nov 03 20:04:57.225865 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:04:57.225829    5570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8bwhz" containerName="extractor"
Nov 03 20:05:27.225403 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:05:27.225367    5570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8bwhz" containerName="extractor"
Nov 03 20:05:57.226250 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:05:57.226214    5570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8bwhz" containerName="extractor"
Nov 03 20:06:27.226027 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:06:27.225989    5570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8bwhz" containerName="extractor"
Nov 03 20:06:57.226337 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:06:57.226292    5570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8bwhz" containerName="extractor"
Nov 03 20:07:27.226115 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:07:27.226074    5570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8bwhz" containerName="extractor"
Nov 03 20:07:57.225295 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:07:57.225212    5570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8bwhz" containerName="extractor"
Nov 03 20:08:27.226046 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:08:27.226001    5570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8bwhz" containerName="extractor"
Nov 03 20:08:57.226733 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:08:57.226692    5570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8bwhz" containerName="extractor"
Nov 03 20:09:27.226034 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:09:27.225988    5570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8bwhz" containerName="extractor"
Nov 03 20:09:50.701235 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:09:50.701203    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zlxws"]
Nov 03 20:09:50.704756 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:09:50.704734    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zlxws"
Nov 03 20:09:50.722101 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:09:50.722076    5570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Nov 03 20:09:50.734270 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:09:50.734250    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-t2v8n\""
Nov 03 20:09:50.734270 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:09:50.734250    5570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Nov 03 20:09:50.737013 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:09:50.736992    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zlxws"]
Nov 03 20:09:50.777175 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:09:50.777150    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b3c9db4b-3d94-4f8a-95db-e669d8063eda-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zlxws\" (UID: \"b3c9db4b-3d94-4f8a-95db-e669d8063eda\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zlxws"
Nov 03 20:09:50.777285 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:09:50.777179    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7mq2\" (UniqueName: \"kubernetes.io/projected/b3c9db4b-3d94-4f8a-95db-e669d8063eda-kube-api-access-l7mq2\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zlxws\" (UID: \"b3c9db4b-3d94-4f8a-95db-e669d8063eda\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zlxws"
Nov 03 20:09:50.777285 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:09:50.777208    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b3c9db4b-3d94-4f8a-95db-e669d8063eda-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zlxws\" (UID: \"b3c9db4b-3d94-4f8a-95db-e669d8063eda\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zlxws"
Nov 03 20:09:50.826651 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:09:50.826627    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cg89lf"]
Nov 03 20:09:50.830191 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:09:50.830174    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cg89lf"
Nov 03 20:09:50.848103 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:09:50.848075    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-z69q8\""
Nov 03 20:09:50.848910 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:09:50.848886    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cg89lf"]
Nov 03 20:09:50.878610 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:09:50.878583    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b3c9db4b-3d94-4f8a-95db-e669d8063eda-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zlxws\" (UID: \"b3c9db4b-3d94-4f8a-95db-e669d8063eda\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zlxws"
Nov 03 20:09:50.878784 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:09:50.878637    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e4ad193d-769c-448a-b60c-5e66b05e9716-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cg89lf\" (UID: \"e4ad193d-769c-448a-b60c-5e66b05e9716\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cg89lf"
Nov 03 20:09:50.878784 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:09:50.878678    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e4ad193d-769c-448a-b60c-5e66b05e9716-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cg89lf\" (UID: \"e4ad193d-769c-448a-b60c-5e66b05e9716\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cg89lf"
Nov 03 20:09:50.878784 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:09:50.878698    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9fkf\" (UniqueName: \"kubernetes.io/projected/e4ad193d-769c-448a-b60c-5e66b05e9716-kube-api-access-m9fkf\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cg89lf\" (UID: \"e4ad193d-769c-448a-b60c-5e66b05e9716\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cg89lf"
Nov 03 20:09:50.878784 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:09:50.878721    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b3c9db4b-3d94-4f8a-95db-e669d8063eda-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zlxws\" (UID: \"b3c9db4b-3d94-4f8a-95db-e669d8063eda\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zlxws"
Nov 03 20:09:50.879030 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:09:50.878810    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7mq2\" (UniqueName: \"kubernetes.io/projected/b3c9db4b-3d94-4f8a-95db-e669d8063eda-kube-api-access-l7mq2\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zlxws\" (UID: \"b3c9db4b-3d94-4f8a-95db-e669d8063eda\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zlxws"
Nov 03 20:09:50.879090 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:09:50.879038    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b3c9db4b-3d94-4f8a-95db-e669d8063eda-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zlxws\" (UID: \"b3c9db4b-3d94-4f8a-95db-e669d8063eda\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zlxws"
Nov 03 20:09:50.879090 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:09:50.879071    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b3c9db4b-3d94-4f8a-95db-e669d8063eda-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zlxws\" (UID: \"b3c9db4b-3d94-4f8a-95db-e669d8063eda\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zlxws"
Nov 03 20:09:50.899663 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:09:50.899642    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7mq2\" (UniqueName: \"kubernetes.io/projected/b3c9db4b-3d94-4f8a-95db-e669d8063eda-kube-api-access-l7mq2\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zlxws\" (UID: \"b3c9db4b-3d94-4f8a-95db-e669d8063eda\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zlxws"
Nov 03 20:09:50.928115 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:09:50.928091    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-lh8mg"]
Nov 03 20:09:50.931426 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:09:50.931410    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lh8mg"
Nov 03 20:09:50.952016 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:09:50.951972    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Nov 03 20:09:50.952016 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:09:50.951974    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Nov 03 20:09:50.952297 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:09:50.952286    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Nov 03 20:09:50.952510 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:09:50.952485    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-gwj92\""
Nov 03 20:09:50.952797 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:09:50.952742    5570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Nov 03 20:09:50.952971 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:09:50.952838    5570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Nov 03 20:09:50.953046 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:09:50.952975    5570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Nov 03 20:09:50.954145 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:09:50.954126    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-lh8mg"]
Nov 03 20:09:50.979548 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:09:50.979517    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a89fe83e-c27c-4124-98e3-04cf40ce08e8-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-lh8mg\" (UID: \"a89fe83e-c27c-4124-98e3-04cf40ce08e8\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lh8mg"
Nov 03 20:09:50.979669 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:09:50.979569    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a89fe83e-c27c-4124-98e3-04cf40ce08e8-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-lh8mg\" (UID: \"a89fe83e-c27c-4124-98e3-04cf40ce08e8\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lh8mg"
Nov 03 20:09:50.979669 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:09:50.979604    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e4ad193d-769c-448a-b60c-5e66b05e9716-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cg89lf\" (UID: \"e4ad193d-769c-448a-b60c-5e66b05e9716\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cg89lf"
Nov 03 20:09:50.979669 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:09:50.979631    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a89fe83e-c27c-4124-98e3-04cf40ce08e8-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-lh8mg\" (UID: \"a89fe83e-c27c-4124-98e3-04cf40ce08e8\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lh8mg"
Nov 03 20:09:50.979669 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:09:50.979664    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9fkf\" (UniqueName: \"kubernetes.io/projected/e4ad193d-769c-448a-b60c-5e66b05e9716-kube-api-access-m9fkf\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cg89lf\" (UID: \"e4ad193d-769c-448a-b60c-5e66b05e9716\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cg89lf"
Nov 03 20:09:50.979879 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:09:50.979697    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a89fe83e-c27c-4124-98e3-04cf40ce08e8-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-lh8mg\" (UID: \"a89fe83e-c27c-4124-98e3-04cf40ce08e8\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lh8mg"
Nov 03 20:09:50.979879 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:09:50.979727    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a89fe83e-c27c-4124-98e3-04cf40ce08e8-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-lh8mg\" (UID: \"a89fe83e-c27c-4124-98e3-04cf40ce08e8\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lh8mg"
Nov 03 20:09:50.979945 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:09:50.979875    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a89fe83e-c27c-4124-98e3-04cf40ce08e8-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-lh8mg\" (UID: \"a89fe83e-c27c-4124-98e3-04cf40ce08e8\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lh8mg"
Nov 03 20:09:50.979945 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:09:50.979932    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zm59b\" (UniqueName: \"kubernetes.io/projected/a89fe83e-c27c-4124-98e3-04cf40ce08e8-kube-api-access-zm59b\") pod \"istiod-openshift-gateway-7876bfcb74-lh8mg\" (UID: \"a89fe83e-c27c-4124-98e3-04cf40ce08e8\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lh8mg"
Nov 03 20:09:50.980009 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:09:50.979954    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e4ad193d-769c-448a-b60c-5e66b05e9716-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cg89lf\" (UID: \"e4ad193d-769c-448a-b60c-5e66b05e9716\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cg89lf"
Nov 03 20:09:50.980078 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:09:50.980061    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e4ad193d-769c-448a-b60c-5e66b05e9716-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cg89lf\" (UID: \"e4ad193d-769c-448a-b60c-5e66b05e9716\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cg89lf"
Nov 03 20:09:50.980318 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:09:50.980304    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e4ad193d-769c-448a-b60c-5e66b05e9716-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cg89lf\" (UID: \"e4ad193d-769c-448a-b60c-5e66b05e9716\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cg89lf"
Nov 03 20:09:50.999969 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:09:50.999941    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9fkf\" (UniqueName: \"kubernetes.io/projected/e4ad193d-769c-448a-b60c-5e66b05e9716-kube-api-access-m9fkf\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cg89lf\" (UID: \"e4ad193d-769c-448a-b60c-5e66b05e9716\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cg89lf"
Nov 03 20:09:51.019739 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:09:51.019718    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zlxws"
Nov 03 20:09:51.083801 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:09:51.081318    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a89fe83e-c27c-4124-98e3-04cf40ce08e8-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-lh8mg\" (UID: \"a89fe83e-c27c-4124-98e3-04cf40ce08e8\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lh8mg"
Nov 03 20:09:51.083801 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:09:51.081381    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a89fe83e-c27c-4124-98e3-04cf40ce08e8-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-lh8mg\" (UID: \"a89fe83e-c27c-4124-98e3-04cf40ce08e8\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lh8mg"
Nov 03 20:09:51.083801 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:09:51.081418    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a89fe83e-c27c-4124-98e3-04cf40ce08e8-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-lh8mg\" (UID: \"a89fe83e-c27c-4124-98e3-04cf40ce08e8\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lh8mg"
Nov 03 20:09:51.083801 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:09:51.081454    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a89fe83e-c27c-4124-98e3-04cf40ce08e8-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-lh8mg\" (UID: \"a89fe83e-c27c-4124-98e3-04cf40ce08e8\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lh8mg"
Nov 03 20:09:51.083801 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:09:51.081482    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a89fe83e-c27c-4124-98e3-04cf40ce08e8-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-lh8mg\" (UID: \"a89fe83e-c27c-4124-98e3-04cf40ce08e8\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lh8mg"
Nov 03 20:09:51.083801 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:09:51.081518    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a89fe83e-c27c-4124-98e3-04cf40ce08e8-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-lh8mg\" (UID: \"a89fe83e-c27c-4124-98e3-04cf40ce08e8\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lh8mg"
Nov 03 20:09:51.083801 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:09:51.081548    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zm59b\" (UniqueName: \"kubernetes.io/projected/a89fe83e-c27c-4124-98e3-04cf40ce08e8-kube-api-access-zm59b\") pod \"istiod-openshift-gateway-7876bfcb74-lh8mg\" (UID: \"a89fe83e-c27c-4124-98e3-04cf40ce08e8\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lh8mg"
Nov 03 20:09:51.083801 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:09:51.082106    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a89fe83e-c27c-4124-98e3-04cf40ce08e8-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-lh8mg\" (UID: \"a89fe83e-c27c-4124-98e3-04cf40ce08e8\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lh8mg"
Nov 03 20:09:51.085467 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:09:51.084973    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a89fe83e-c27c-4124-98e3-04cf40ce08e8-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-lh8mg\" (UID: \"a89fe83e-c27c-4124-98e3-04cf40ce08e8\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lh8mg"
Nov 03 20:09:51.085467 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:09:51.085103    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a89fe83e-c27c-4124-98e3-04cf40ce08e8-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-lh8mg\" (UID: \"a89fe83e-c27c-4124-98e3-04cf40ce08e8\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lh8mg"
Nov 03 20:09:51.086057 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:09:51.086030    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a89fe83e-c27c-4124-98e3-04cf40ce08e8-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-lh8mg\" (UID: \"a89fe83e-c27c-4124-98e3-04cf40ce08e8\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lh8mg"
Nov 03 20:09:51.086196 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:09:51.086179    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a89fe83e-c27c-4124-98e3-04cf40ce08e8-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-lh8mg\" (UID: \"a89fe83e-c27c-4124-98e3-04cf40ce08e8\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lh8mg"
Nov 03 20:09:51.101062 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:09:51.101037    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zm59b\" (UniqueName: \"kubernetes.io/projected/a89fe83e-c27c-4124-98e3-04cf40ce08e8-kube-api-access-zm59b\") pod \"istiod-openshift-gateway-7876bfcb74-lh8mg\" (UID: \"a89fe83e-c27c-4124-98e3-04cf40ce08e8\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lh8mg"
Nov 03 20:09:51.101666 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:09:51.101649    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a89fe83e-c27c-4124-98e3-04cf40ce08e8-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-lh8mg\" (UID: \"a89fe83e-c27c-4124-98e3-04cf40ce08e8\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lh8mg"
Nov 03 20:09:51.145452 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:09:51.145430    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cg89lf"
Nov 03 20:09:51.242439 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:09:51.242354    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lh8mg"
Nov 03 20:09:51.321983 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:09:51.321952    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zlxws"]
Nov 03 20:09:51.327178 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:09:51.327150    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3c9db4b_3d94_4f8a_95db_e669d8063eda.slice/crio-0c70cdadc7f6885706949a3397a594ea3bfb107e779613a15f7090eb2727f203 WatchSource:0}: Error finding container 0c70cdadc7f6885706949a3397a594ea3bfb107e779613a15f7090eb2727f203: Status 404 returned error can't find the container with id 0c70cdadc7f6885706949a3397a594ea3bfb107e779613a15f7090eb2727f203
Nov 03 20:09:51.439493 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:09:51.439458    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cg89lf"]
Nov 03 20:09:51.443700 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:09:51.443669    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4ad193d_769c_448a_b60c_5e66b05e9716.slice/crio-b577d5d3aa8fc6d345a0b29abbb5ad5bb909ce8603d435d7fc74664c8d23f78b WatchSource:0}: Error finding container b577d5d3aa8fc6d345a0b29abbb5ad5bb909ce8603d435d7fc74664c8d23f78b: Status 404 returned error can't find the container with id b577d5d3aa8fc6d345a0b29abbb5ad5bb909ce8603d435d7fc74664c8d23f78b
Nov 03 20:09:51.445724 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:09:51.445708    5570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 20:09:51.541359 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:09:51.541287    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-lh8mg"]
Nov 03 20:09:51.545435 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:09:51.545407    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda89fe83e_c27c_4124_98e3_04cf40ce08e8.slice/crio-385737090e64e424d5fd6e3a2a3b1b13390a78f6e7eac9222ca85eb0fbd14072 WatchSource:0}: Error finding container 385737090e64e424d5fd6e3a2a3b1b13390a78f6e7eac9222ca85eb0fbd14072: Status 404 returned error can't find the container with id 385737090e64e424d5fd6e3a2a3b1b13390a78f6e7eac9222ca85eb0fbd14072
Nov 03 20:09:52.134567 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:09:52.134525    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cg89lf" event={"ID":"e4ad193d-769c-448a-b60c-5e66b05e9716","Type":"ContainerStarted","Data":"b577d5d3aa8fc6d345a0b29abbb5ad5bb909ce8603d435d7fc74664c8d23f78b"}
Nov 03 20:09:52.135944 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:09:52.135889    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zlxws" event={"ID":"b3c9db4b-3d94-4f8a-95db-e669d8063eda","Type":"ContainerStarted","Data":"0c70cdadc7f6885706949a3397a594ea3bfb107e779613a15f7090eb2727f203"}
Nov 03 20:09:52.137364 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:09:52.137338    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lh8mg" event={"ID":"a89fe83e-c27c-4124-98e3-04cf40ce08e8","Type":"ContainerStarted","Data":"385737090e64e424d5fd6e3a2a3b1b13390a78f6e7eac9222ca85eb0fbd14072"}
Nov 03 20:09:54.904466 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:09:54.904429    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:09:54.904893 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:09:54.904525    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:09:55.153375 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:09:55.153295    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lh8mg" event={"ID":"a89fe83e-c27c-4124-98e3-04cf40ce08e8","Type":"ContainerStarted","Data":"358bf923bd3ed783e1976c92afb6c96fa26224257b001b7004029d14da3f853b"}
Nov 03 20:09:55.153620 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:09:55.153558    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lh8mg"
Nov 03 20:09:55.155484 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:09:55.155441    5570 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-lh8mg container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 20:09:55.155610 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:09:55.155511    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lh8mg" podUID="a89fe83e-c27c-4124-98e3-04cf40ce08e8" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 20:09:55.211785 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:09:55.210955    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lh8mg" podStartSLOduration=1.8547871009999999 podStartE2EDuration="5.210937614s" totalImagesPullingTime="3.356150513s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:09:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:09:55.209407089 +0000 UTC m=+329.362068978" watchObservedRunningTime="2026-11-03 20:09:55.210937614 +0000 UTC m=+329.363599500"
Nov 03 20:09:56.160433 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:09:56.159890    5570 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-lh8mg container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 20:09:56.160433 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:09:56.159949    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lh8mg" podUID="a89fe83e-c27c-4124-98e3-04cf40ce08e8" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 20:09:57.237166 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:09:57.237128    5570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8bwhz" containerName="extractor"
Nov 03 20:09:59.159262 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:09:59.159230    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lh8mg"
Nov 03 20:09:59.172660 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:09:59.172632    5570 generic.go:356] "Generic (PLEG): container finished" podID="e4ad193d-769c-448a-b60c-5e66b05e9716" containerID="b30c35a2df8449dd4425fa407289f562fc6684be3adbce30838d7c0d0c8a85ee" exitCode=0
Nov 03 20:09:59.172818 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:09:59.172704    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cg89lf" event={"ID":"e4ad193d-769c-448a-b60c-5e66b05e9716","Type":"ContainerDied","Data":"b30c35a2df8449dd4425fa407289f562fc6684be3adbce30838d7c0d0c8a85ee"}
Nov 03 20:09:59.174303 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:09:59.174273    5570 generic.go:356] "Generic (PLEG): container finished" podID="b3c9db4b-3d94-4f8a-95db-e669d8063eda" containerID="dbbb23a7e6f84cee45da27570a61ad83e72a9fcd431f47b3d6a61631cbb10adc" exitCode=0
Nov 03 20:09:59.174404 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:09:59.174380    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zlxws" event={"ID":"b3c9db4b-3d94-4f8a-95db-e669d8063eda","Type":"ContainerDied","Data":"dbbb23a7e6f84cee45da27570a61ad83e72a9fcd431f47b3d6a61631cbb10adc"}
Nov 03 20:10:02.188326 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:10:02.188289    5570 generic.go:356] "Generic (PLEG): container finished" podID="e4ad193d-769c-448a-b60c-5e66b05e9716" containerID="53ffb59586e78c73c6c51158baa3bdad09157b2ea6f53e29dd8c99656bb0f653" exitCode=0
Nov 03 20:10:02.188794 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:10:02.188374    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cg89lf" event={"ID":"e4ad193d-769c-448a-b60c-5e66b05e9716","Type":"ContainerDied","Data":"53ffb59586e78c73c6c51158baa3bdad09157b2ea6f53e29dd8c99656bb0f653"}
Nov 03 20:10:02.190342 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:10:02.190243    5570 generic.go:356] "Generic (PLEG): container finished" podID="b3c9db4b-3d94-4f8a-95db-e669d8063eda" containerID="c4631ba35a15a2f14000c3ed2af26f17ada5e4eb309e06f89d035642a8a3759b" exitCode=0
Nov 03 20:10:02.190342 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:10:02.190275    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zlxws" event={"ID":"b3c9db4b-3d94-4f8a-95db-e669d8063eda","Type":"ContainerDied","Data":"c4631ba35a15a2f14000c3ed2af26f17ada5e4eb309e06f89d035642a8a3759b"}
Nov 03 20:10:09.221949 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:10:09.221909    5570 generic.go:356] "Generic (PLEG): container finished" podID="e4ad193d-769c-448a-b60c-5e66b05e9716" containerID="c55e70319e5c700b7f1f802c4ecd313b9a0fcb7dbf9c1189211296c8968ae4db" exitCode=0
Nov 03 20:10:09.222424 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:10:09.221980    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cg89lf" event={"ID":"e4ad193d-769c-448a-b60c-5e66b05e9716","Type":"ContainerDied","Data":"c55e70319e5c700b7f1f802c4ecd313b9a0fcb7dbf9c1189211296c8968ae4db"}
Nov 03 20:10:09.224099 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:10:09.224076    5570 generic.go:356] "Generic (PLEG): container finished" podID="b3c9db4b-3d94-4f8a-95db-e669d8063eda" containerID="ab9616920ff6dd7a29183e882be940ef244383d90e78bda4509aa96a30849c03" exitCode=0
Nov 03 20:10:09.224202 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:10:09.224121    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zlxws" event={"ID":"b3c9db4b-3d94-4f8a-95db-e669d8063eda","Type":"ContainerDied","Data":"ab9616920ff6dd7a29183e882be940ef244383d90e78bda4509aa96a30849c03"}
Nov 03 20:10:10.369446 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:10:10.369422    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zlxws"
Nov 03 20:10:10.372569 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:10:10.372551    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cg89lf"
Nov 03 20:10:10.465899 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:10:10.465873    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3c9db4b-3d94-4f8a-95db-e669d8063eda-kube-api-access-l7mq2\" (UniqueName: \"kubernetes.io/projected/b3c9db4b-3d94-4f8a-95db-e669d8063eda-kube-api-access-l7mq2\") pod \"b3c9db4b-3d94-4f8a-95db-e669d8063eda\" (UID: \"b3c9db4b-3d94-4f8a-95db-e669d8063eda\") "
Nov 03 20:10:10.466055 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:10:10.465916    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3c9db4b-3d94-4f8a-95db-e669d8063eda-bundle\" (UniqueName: \"kubernetes.io/empty-dir/b3c9db4b-3d94-4f8a-95db-e669d8063eda-bundle\") pod \"b3c9db4b-3d94-4f8a-95db-e669d8063eda\" (UID: \"b3c9db4b-3d94-4f8a-95db-e669d8063eda\") "
Nov 03 20:10:10.466055 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:10:10.465968    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3c9db4b-3d94-4f8a-95db-e669d8063eda-util\" (UniqueName: \"kubernetes.io/empty-dir/b3c9db4b-3d94-4f8a-95db-e669d8063eda-util\") pod \"b3c9db4b-3d94-4f8a-95db-e669d8063eda\" (UID: \"b3c9db4b-3d94-4f8a-95db-e669d8063eda\") "
Nov 03 20:10:10.466055 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:10:10.465990    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4ad193d-769c-448a-b60c-5e66b05e9716-bundle\" (UniqueName: \"kubernetes.io/empty-dir/e4ad193d-769c-448a-b60c-5e66b05e9716-bundle\") pod \"e4ad193d-769c-448a-b60c-5e66b05e9716\" (UID: \"e4ad193d-769c-448a-b60c-5e66b05e9716\") "
Nov 03 20:10:10.466055 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:10:10.466012    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4ad193d-769c-448a-b60c-5e66b05e9716-kube-api-access-m9fkf\" (UniqueName: \"kubernetes.io/projected/e4ad193d-769c-448a-b60c-5e66b05e9716-kube-api-access-m9fkf\") pod \"e4ad193d-769c-448a-b60c-5e66b05e9716\" (UID: \"e4ad193d-769c-448a-b60c-5e66b05e9716\") "
Nov 03 20:10:10.466055 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:10:10.466047    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4ad193d-769c-448a-b60c-5e66b05e9716-util\" (UniqueName: \"kubernetes.io/empty-dir/e4ad193d-769c-448a-b60c-5e66b05e9716-util\") pod \"e4ad193d-769c-448a-b60c-5e66b05e9716\" (UID: \"e4ad193d-769c-448a-b60c-5e66b05e9716\") "
Nov 03 20:10:10.466510 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:10:10.466476    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3c9db4b-3d94-4f8a-95db-e669d8063eda-bundle" pod "b3c9db4b-3d94-4f8a-95db-e669d8063eda" (UID: "b3c9db4b-3d94-4f8a-95db-e669d8063eda"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:10:10.466767 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:10:10.466746    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4ad193d-769c-448a-b60c-5e66b05e9716-bundle" pod "e4ad193d-769c-448a-b60c-5e66b05e9716" (UID: "e4ad193d-769c-448a-b60c-5e66b05e9716"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:10:10.468378 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:10:10.468334    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4ad193d-769c-448a-b60c-5e66b05e9716-kube-api-access-m9fkf" pod "e4ad193d-769c-448a-b60c-5e66b05e9716" (UID: "e4ad193d-769c-448a-b60c-5e66b05e9716"). InnerVolumeSpecName "kube-api-access-m9fkf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:10:10.468477 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:10:10.468437    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3c9db4b-3d94-4f8a-95db-e669d8063eda-kube-api-access-l7mq2" pod "b3c9db4b-3d94-4f8a-95db-e669d8063eda" (UID: "b3c9db4b-3d94-4f8a-95db-e669d8063eda"). InnerVolumeSpecName "kube-api-access-l7mq2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:10:10.470254 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:10:10.470226    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4ad193d-769c-448a-b60c-5e66b05e9716-util" pod "e4ad193d-769c-448a-b60c-5e66b05e9716" (UID: "e4ad193d-769c-448a-b60c-5e66b05e9716"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:10:10.472301 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:10:10.472279    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3c9db4b-3d94-4f8a-95db-e669d8063eda-util" pod "b3c9db4b-3d94-4f8a-95db-e669d8063eda" (UID: "b3c9db4b-3d94-4f8a-95db-e669d8063eda"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:10:10.567041 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:10:10.566957    5570 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b3c9db4b-3d94-4f8a-95db-e669d8063eda-util\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:10:10.567041 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:10:10.566996    5570 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e4ad193d-769c-448a-b60c-5e66b05e9716-bundle\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:10:10.567041 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:10:10.567007    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9fkf\" (UniqueName: \"kubernetes.io/projected/e4ad193d-769c-448a-b60c-5e66b05e9716-kube-api-access-m9fkf\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:10:10.567041 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:10:10.567015    5570 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e4ad193d-769c-448a-b60c-5e66b05e9716-util\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:10:10.567041 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:10:10.567024    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7mq2\" (UniqueName: \"kubernetes.io/projected/b3c9db4b-3d94-4f8a-95db-e669d8063eda-kube-api-access-l7mq2\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:10:10.567041 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:10:10.567032    5570 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b3c9db4b-3d94-4f8a-95db-e669d8063eda-bundle\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:10:11.233163 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:10:11.233122    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cg89lf" event={"ID":"e4ad193d-769c-448a-b60c-5e66b05e9716","Type":"ContainerDied","Data":"b577d5d3aa8fc6d345a0b29abbb5ad5bb909ce8603d435d7fc74664c8d23f78b"}
Nov 03 20:10:11.233351 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:10:11.233181    5570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b577d5d3aa8fc6d345a0b29abbb5ad5bb909ce8603d435d7fc74664c8d23f78b"
Nov 03 20:10:11.233351 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:10:11.233140    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cg89lf"
Nov 03 20:10:11.235063 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:10:11.235036    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zlxws" event={"ID":"b3c9db4b-3d94-4f8a-95db-e669d8063eda","Type":"ContainerDied","Data":"0c70cdadc7f6885706949a3397a594ea3bfb107e779613a15f7090eb2727f203"}
Nov 03 20:10:11.235180 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:10:11.235076    5570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0c70cdadc7f6885706949a3397a594ea3bfb107e779613a15f7090eb2727f203"
Nov 03 20:10:11.235180 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:10:11.235125    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644zlxws"
Nov 03 20:10:27.225695 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:10:27.225658    5570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8bwhz" containerName="extractor"
Nov 03 20:10:27.476762 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:10:27.476689    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-6fd567655f-mghpw"]
Nov 03 20:10:27.477007 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:10:27.476994    5570 cpu_manager.go:414] "containerMap: removing container" podUID="b3c9db4b-3d94-4f8a-95db-e669d8063eda" containerName="util"
Nov 03 20:10:27.477050 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:10:27.477008    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3c9db4b-3d94-4f8a-95db-e669d8063eda" containerName="util"
Nov 03 20:10:27.477050 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:10:27.477026    5570 cpu_manager.go:414] "containerMap: removing container" podUID="e4ad193d-769c-448a-b60c-5e66b05e9716" containerName="pull"
Nov 03 20:10:27.477050 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:10:27.477031    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4ad193d-769c-448a-b60c-5e66b05e9716" containerName="pull"
Nov 03 20:10:27.477050 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:10:27.477039    5570 cpu_manager.go:414] "containerMap: removing container" podUID="e4ad193d-769c-448a-b60c-5e66b05e9716" containerName="util"
Nov 03 20:10:27.477050 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:10:27.477043    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4ad193d-769c-448a-b60c-5e66b05e9716" containerName="util"
Nov 03 20:10:27.477050 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:10:27.477050    5570 cpu_manager.go:414] "containerMap: removing container" podUID="b3c9db4b-3d94-4f8a-95db-e669d8063eda" containerName="extract"
Nov 03 20:10:27.477229 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:10:27.477055    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3c9db4b-3d94-4f8a-95db-e669d8063eda" containerName="extract"
Nov 03 20:10:27.477229 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:10:27.477062    5570 cpu_manager.go:414] "containerMap: removing container" podUID="b3c9db4b-3d94-4f8a-95db-e669d8063eda" containerName="pull"
Nov 03 20:10:27.477229 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:10:27.477067    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3c9db4b-3d94-4f8a-95db-e669d8063eda" containerName="pull"
Nov 03 20:10:27.477229 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:10:27.477078    5570 cpu_manager.go:414] "containerMap: removing container" podUID="e4ad193d-769c-448a-b60c-5e66b05e9716" containerName="extract"
Nov 03 20:10:27.477229 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:10:27.477083    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4ad193d-769c-448a-b60c-5e66b05e9716" containerName="extract"
Nov 03 20:10:27.477229 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:10:27.477130    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4ad193d-769c-448a-b60c-5e66b05e9716" containerName="extract"
Nov 03 20:10:27.477229 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:10:27.477141    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3c9db4b-3d94-4f8a-95db-e669d8063eda" containerName="extract"
Nov 03 20:10:27.482744 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:10:27.482721    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6fd567655f-mghpw"
Nov 03 20:10:27.503094 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:10:27.503073    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Nov 03 20:10:27.503200 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:10:27.503140    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Nov 03 20:10:27.504254 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:10:27.504234    5570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Nov 03 20:10:27.504311 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:10:27.504294    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-rsm5h\""
Nov 03 20:10:27.504488 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:10:27.504477    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Nov 03 20:10:27.504525 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:10:27.504492    5570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Nov 03 20:10:27.505793 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:10:27.505766    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f9182571-ed14-46b1-bc81-3f901876b06a-manifests\") pod \"opendatahub-operator-controller-manager-6fd567655f-mghpw\" (UID: \"f9182571-ed14-46b1-bc81-3f901876b06a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6fd567655f-mghpw"
Nov 03 20:10:27.505841 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:10:27.505803    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f9182571-ed14-46b1-bc81-3f901876b06a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6fd567655f-mghpw\" (UID: \"f9182571-ed14-46b1-bc81-3f901876b06a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6fd567655f-mghpw"
Nov 03 20:10:27.505841 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:10:27.505824    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwpqn\" (UniqueName: \"kubernetes.io/projected/f9182571-ed14-46b1-bc81-3f901876b06a-kube-api-access-zwpqn\") pod \"opendatahub-operator-controller-manager-6fd567655f-mghpw\" (UID: \"f9182571-ed14-46b1-bc81-3f901876b06a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6fd567655f-mghpw"
Nov 03 20:10:27.505958 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:10:27.505941    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f9182571-ed14-46b1-bc81-3f901876b06a-webhook-cert\") pod \"opendatahub-operator-controller-manager-6fd567655f-mghpw\" (UID: \"f9182571-ed14-46b1-bc81-3f901876b06a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6fd567655f-mghpw"
Nov 03 20:10:27.506001 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:10:27.505968    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/f9182571-ed14-46b1-bc81-3f901876b06a-metrics-cert\") pod \"opendatahub-operator-controller-manager-6fd567655f-mghpw\" (UID: \"f9182571-ed14-46b1-bc81-3f901876b06a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6fd567655f-mghpw"
Nov 03 20:10:27.506051 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:10:27.506038    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f9182571-ed14-46b1-bc81-3f901876b06a-tmp\") pod \"opendatahub-operator-controller-manager-6fd567655f-mghpw\" (UID: \"f9182571-ed14-46b1-bc81-3f901876b06a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6fd567655f-mghpw"
Nov 03 20:10:27.588908 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:10:27.588881    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-6fd567655f-mghpw"]
Nov 03 20:10:27.607120 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:10:27.607093    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/f9182571-ed14-46b1-bc81-3f901876b06a-metrics-cert\") pod \"opendatahub-operator-controller-manager-6fd567655f-mghpw\" (UID: \"f9182571-ed14-46b1-bc81-3f901876b06a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6fd567655f-mghpw"
Nov 03 20:10:27.607231 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:10:27.607132    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f9182571-ed14-46b1-bc81-3f901876b06a-tmp\") pod \"opendatahub-operator-controller-manager-6fd567655f-mghpw\" (UID: \"f9182571-ed14-46b1-bc81-3f901876b06a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6fd567655f-mghpw"
Nov 03 20:10:27.607231 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:10:27.607202    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f9182571-ed14-46b1-bc81-3f901876b06a-manifests\") pod \"opendatahub-operator-controller-manager-6fd567655f-mghpw\" (UID: \"f9182571-ed14-46b1-bc81-3f901876b06a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6fd567655f-mghpw"
Nov 03 20:10:27.607332 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:10:27.607228    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f9182571-ed14-46b1-bc81-3f901876b06a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6fd567655f-mghpw\" (UID: \"f9182571-ed14-46b1-bc81-3f901876b06a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6fd567655f-mghpw"
Nov 03 20:10:27.607332 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:10:27.607258    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwpqn\" (UniqueName: \"kubernetes.io/projected/f9182571-ed14-46b1-bc81-3f901876b06a-kube-api-access-zwpqn\") pod \"opendatahub-operator-controller-manager-6fd567655f-mghpw\" (UID: \"f9182571-ed14-46b1-bc81-3f901876b06a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6fd567655f-mghpw"
Nov 03 20:10:27.607332 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:10:27.607303    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f9182571-ed14-46b1-bc81-3f901876b06a-webhook-cert\") pod \"opendatahub-operator-controller-manager-6fd567655f-mghpw\" (UID: \"f9182571-ed14-46b1-bc81-3f901876b06a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6fd567655f-mghpw"
Nov 03 20:10:27.607662 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:10:27.607647    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f9182571-ed14-46b1-bc81-3f901876b06a-manifests\") pod \"opendatahub-operator-controller-manager-6fd567655f-mghpw\" (UID: \"f9182571-ed14-46b1-bc81-3f901876b06a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6fd567655f-mghpw"
Nov 03 20:10:27.609431 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:10:27.609408    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f9182571-ed14-46b1-bc81-3f901876b06a-tmp\") pod \"opendatahub-operator-controller-manager-6fd567655f-mghpw\" (UID: \"f9182571-ed14-46b1-bc81-3f901876b06a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6fd567655f-mghpw"
Nov 03 20:10:27.609567 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:10:27.609550    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f9182571-ed14-46b1-bc81-3f901876b06a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6fd567655f-mghpw\" (UID: \"f9182571-ed14-46b1-bc81-3f901876b06a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6fd567655f-mghpw"
Nov 03 20:10:27.610056 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:10:27.610036    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f9182571-ed14-46b1-bc81-3f901876b06a-webhook-cert\") pod \"opendatahub-operator-controller-manager-6fd567655f-mghpw\" (UID: \"f9182571-ed14-46b1-bc81-3f901876b06a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6fd567655f-mghpw"
Nov 03 20:10:27.610056 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:10:27.610047    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/f9182571-ed14-46b1-bc81-3f901876b06a-metrics-cert\") pod \"opendatahub-operator-controller-manager-6fd567655f-mghpw\" (UID: \"f9182571-ed14-46b1-bc81-3f901876b06a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6fd567655f-mghpw"
Nov 03 20:10:27.628022 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:10:27.627994    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwpqn\" (UniqueName: \"kubernetes.io/projected/f9182571-ed14-46b1-bc81-3f901876b06a-kube-api-access-zwpqn\") pod \"opendatahub-operator-controller-manager-6fd567655f-mghpw\" (UID: \"f9182571-ed14-46b1-bc81-3f901876b06a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6fd567655f-mghpw"
Nov 03 20:10:27.794478 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:10:27.794408    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6fd567655f-mghpw"
Nov 03 20:10:28.142255 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:10:28.142180    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-6fd567655f-mghpw"]
Nov 03 20:10:28.145154 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:10:28.145121    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9182571_ed14_46b1_bc81_3f901876b06a.slice/crio-e52abbb19d387bb35864a3c56c0ba0dac3dbd93140456c4d6bb6433bf2cf6023 WatchSource:0}: Error finding container e52abbb19d387bb35864a3c56c0ba0dac3dbd93140456c4d6bb6433bf2cf6023: Status 404 returned error can't find the container with id e52abbb19d387bb35864a3c56c0ba0dac3dbd93140456c4d6bb6433bf2cf6023
Nov 03 20:10:28.302724 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:10:28.297920    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6fd567655f-mghpw" event={"ID":"f9182571-ed14-46b1-bc81-3f901876b06a","Type":"ContainerStarted","Data":"e52abbb19d387bb35864a3c56c0ba0dac3dbd93140456c4d6bb6433bf2cf6023"}
Nov 03 20:10:31.313159 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:10:31.313071    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6fd567655f-mghpw" event={"ID":"f9182571-ed14-46b1-bc81-3f901876b06a","Type":"ContainerStarted","Data":"4d88cd8fe9b7d6045acd739ef5659fe0118988e3f6c8465b326bdcb8cd8ce5b0"}
Nov 03 20:10:32.318216 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:10:32.318175    5570 generic.go:356] "Generic (PLEG): container finished" podID="f9182571-ed14-46b1-bc81-3f901876b06a" containerID="4d88cd8fe9b7d6045acd739ef5659fe0118988e3f6c8465b326bdcb8cd8ce5b0" exitCode=0
Nov 03 20:10:32.318588 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:10:32.318264    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6fd567655f-mghpw" event={"ID":"f9182571-ed14-46b1-bc81-3f901876b06a","Type":"ContainerDied","Data":"4d88cd8fe9b7d6045acd739ef5659fe0118988e3f6c8465b326bdcb8cd8ce5b0"}
Nov 03 20:10:33.324068 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:10:33.324032    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6fd567655f-mghpw" event={"ID":"f9182571-ed14-46b1-bc81-3f901876b06a","Type":"ContainerStarted","Data":"480a803cc4fb57e70f48493b61e736fff96329d2fa9a263f57b7246735b142a7"}
Nov 03 20:10:33.324457 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:10:33.324155    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6fd567655f-mghpw"
Nov 03 20:10:33.412333 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:10:33.412281    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6fd567655f-mghpw" podStartSLOduration=2.791922476 podStartE2EDuration="6.412269112s" totalImagesPullingTime="3.119009232s" totalInitContainerRuntime="501.337404ms" isStatefulPod=false podCreationTimestamp="2026-11-03 20:10:27 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:10:33.399511496 +0000 UTC m=+367.552173381" watchObservedRunningTime="2026-11-03 20:10:33.412269112 +0000 UTC m=+367.564930998"
Nov 03 20:10:44.330091 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:10:44.330006    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6fd567655f-mghpw"
Nov 03 20:10:57.225354 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:10:57.225318    5570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8bwhz" containerName="extractor"
Nov 03 20:10:58.914913 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:10:58.914879    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-fbjbc"]
Nov 03 20:10:58.917400 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:10:58.917379    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-fbjbc"
Nov 03 20:10:58.944245 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:10:58.944211    5570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Nov 03 20:10:58.944347 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:10:58.944247    5570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Nov 03 20:10:58.944347 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:10:58.944319    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-w88lr\""
Nov 03 20:10:58.945372 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:10:58.945343    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-fbjbc"]
Nov 03 20:10:59.065223 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:10:59.065193    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8s6w\" (UniqueName: \"kubernetes.io/projected/6308d31e-2cc7-4e66-9f9f-cf2c878b0043-kube-api-access-t8s6w\") pod \"limitador-operator-controller-manager-5778bb4449-fbjbc\" (UID: \"6308d31e-2cc7-4e66-9f9f-cf2c878b0043\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-fbjbc"
Nov 03 20:10:59.166391 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:10:59.166309    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8s6w\" (UniqueName: \"kubernetes.io/projected/6308d31e-2cc7-4e66-9f9f-cf2c878b0043-kube-api-access-t8s6w\") pod \"limitador-operator-controller-manager-5778bb4449-fbjbc\" (UID: \"6308d31e-2cc7-4e66-9f9f-cf2c878b0043\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-fbjbc"
Nov 03 20:10:59.187567 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:10:59.187531    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8s6w\" (UniqueName: \"kubernetes.io/projected/6308d31e-2cc7-4e66-9f9f-cf2c878b0043-kube-api-access-t8s6w\") pod \"limitador-operator-controller-manager-5778bb4449-fbjbc\" (UID: \"6308d31e-2cc7-4e66-9f9f-cf2c878b0043\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-fbjbc"
Nov 03 20:10:59.228415 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:10:59.228372    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-fbjbc"
Nov 03 20:10:59.535690 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:10:59.535418    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-fbjbc"]
Nov 03 20:10:59.982745 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:10:59.982711    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-6fd6dbbcd5-s95zx"]
Nov 03 20:10:59.985587 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:10:59.985567    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6fd6dbbcd5-s95zx"
Nov 03 20:11:00.013560 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:11:00.013523    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6fd6dbbcd5-s95zx"]
Nov 03 20:11:00.074305 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:11:00.074268    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8d9abc3-2f2b-4f6f-a8c9-88b67354781b-tls-certs\") pod \"kube-auth-proxy-6fd6dbbcd5-s95zx\" (UID: \"c8d9abc3-2f2b-4f6f-a8c9-88b67354781b\") " pod="openshift-ingress/kube-auth-proxy-6fd6dbbcd5-s95zx"
Nov 03 20:11:00.074305 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:11:00.074310    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bjpf\" (UniqueName: \"kubernetes.io/projected/c8d9abc3-2f2b-4f6f-a8c9-88b67354781b-kube-api-access-5bjpf\") pod \"kube-auth-proxy-6fd6dbbcd5-s95zx\" (UID: \"c8d9abc3-2f2b-4f6f-a8c9-88b67354781b\") " pod="openshift-ingress/kube-auth-proxy-6fd6dbbcd5-s95zx"
Nov 03 20:11:00.074518 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:11:00.074329    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c8d9abc3-2f2b-4f6f-a8c9-88b67354781b-tmp\") pod \"kube-auth-proxy-6fd6dbbcd5-s95zx\" (UID: \"c8d9abc3-2f2b-4f6f-a8c9-88b67354781b\") " pod="openshift-ingress/kube-auth-proxy-6fd6dbbcd5-s95zx"
Nov 03 20:11:00.096657 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:11:00.096621    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Nov 03 20:11:00.096961 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:11:00.096942    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Nov 03 20:11:00.097024 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:11:00.096946    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-6qpp4\""
Nov 03 20:11:00.175075 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:11:00.175039    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8d9abc3-2f2b-4f6f-a8c9-88b67354781b-tls-certs\") pod \"kube-auth-proxy-6fd6dbbcd5-s95zx\" (UID: \"c8d9abc3-2f2b-4f6f-a8c9-88b67354781b\") " pod="openshift-ingress/kube-auth-proxy-6fd6dbbcd5-s95zx"
Nov 03 20:11:00.175242 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:11:00.175098    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bjpf\" (UniqueName: \"kubernetes.io/projected/c8d9abc3-2f2b-4f6f-a8c9-88b67354781b-kube-api-access-5bjpf\") pod \"kube-auth-proxy-6fd6dbbcd5-s95zx\" (UID: \"c8d9abc3-2f2b-4f6f-a8c9-88b67354781b\") " pod="openshift-ingress/kube-auth-proxy-6fd6dbbcd5-s95zx"
Nov 03 20:11:00.175242 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:11:00.175128    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c8d9abc3-2f2b-4f6f-a8c9-88b67354781b-tmp\") pod \"kube-auth-proxy-6fd6dbbcd5-s95zx\" (UID: \"c8d9abc3-2f2b-4f6f-a8c9-88b67354781b\") " pod="openshift-ingress/kube-auth-proxy-6fd6dbbcd5-s95zx"
Nov 03 20:11:00.177524 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:11:00.177492    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c8d9abc3-2f2b-4f6f-a8c9-88b67354781b-tmp\") pod \"kube-auth-proxy-6fd6dbbcd5-s95zx\" (UID: \"c8d9abc3-2f2b-4f6f-a8c9-88b67354781b\") " pod="openshift-ingress/kube-auth-proxy-6fd6dbbcd5-s95zx"
Nov 03 20:11:00.177653 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:11:00.177623    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8d9abc3-2f2b-4f6f-a8c9-88b67354781b-tls-certs\") pod \"kube-auth-proxy-6fd6dbbcd5-s95zx\" (UID: \"c8d9abc3-2f2b-4f6f-a8c9-88b67354781b\") " pod="openshift-ingress/kube-auth-proxy-6fd6dbbcd5-s95zx"
Nov 03 20:11:00.205985 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:11:00.205954    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bjpf\" (UniqueName: \"kubernetes.io/projected/c8d9abc3-2f2b-4f6f-a8c9-88b67354781b-kube-api-access-5bjpf\") pod \"kube-auth-proxy-6fd6dbbcd5-s95zx\" (UID: \"c8d9abc3-2f2b-4f6f-a8c9-88b67354781b\") " pod="openshift-ingress/kube-auth-proxy-6fd6dbbcd5-s95zx"
Nov 03 20:11:00.297080 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:11:00.296986    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6fd6dbbcd5-s95zx"
Nov 03 20:11:00.421640 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:11:00.421607    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-fbjbc" event={"ID":"6308d31e-2cc7-4e66-9f9f-cf2c878b0043","Type":"ContainerStarted","Data":"3d9c71e5f5d12b712e824dd654ad820425fe543507fdce9e24c5da1a662baafa"}
Nov 03 20:11:00.598075 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:11:00.597972    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6fd6dbbcd5-s95zx"]
Nov 03 20:11:00.603065 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:11:00.603031    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8d9abc3_2f2b_4f6f_a8c9_88b67354781b.slice/crio-64c05adb0e04410fe49d1ef03cae922e1f67b807663f42992d16006d78ea22a1 WatchSource:0}: Error finding container 64c05adb0e04410fe49d1ef03cae922e1f67b807663f42992d16006d78ea22a1: Status 404 returned error can't find the container with id 64c05adb0e04410fe49d1ef03cae922e1f67b807663f42992d16006d78ea22a1
Nov 03 20:11:01.430324 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:11:01.430287    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6fd6dbbcd5-s95zx" event={"ID":"c8d9abc3-2f2b-4f6f-a8c9-88b67354781b","Type":"ContainerStarted","Data":"64c05adb0e04410fe49d1ef03cae922e1f67b807663f42992d16006d78ea22a1"}
Nov 03 20:11:03.441381 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:11:03.441347    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-fbjbc" event={"ID":"6308d31e-2cc7-4e66-9f9f-cf2c878b0043","Type":"ContainerStarted","Data":"663a9d72afa0f6d8da7740ad271127c79049156d42f80a68aa240dc7a542ab48"}
Nov 03 20:11:03.441749 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:11:03.441571    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-fbjbc"
Nov 03 20:11:03.488520 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:11:03.488477    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-fbjbc" podStartSLOduration=2.465079336 podStartE2EDuration="5.488465249s" totalImagesPullingTime="3.023385913s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:10:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:11:03.487892755 +0000 UTC m=+397.640554642" watchObservedRunningTime="2026-11-03 20:11:03.488465249 +0000 UTC m=+397.641127135"
Nov 03 20:11:04.447670 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:11:04.447632    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6fd6dbbcd5-s95zx" event={"ID":"c8d9abc3-2f2b-4f6f-a8c9-88b67354781b","Type":"ContainerStarted","Data":"5535670fdcd25397fab9a6d1c55c2719e1251349546791b18422561f2ce6b4ed"}
Nov 03 20:11:04.485920 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:11:04.485867    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-6fd6dbbcd5-s95zx" podStartSLOduration=2.576702117 podStartE2EDuration="5.48585553s" totalImagesPullingTime="2.909153413s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:10:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:11:04.485071798 +0000 UTC m=+398.637733683" watchObservedRunningTime="2026-11-03 20:11:04.48585553 +0000 UTC m=+398.638517407"
Nov 03 20:11:14.450299 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:11:14.450267    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-fbjbc"
Nov 03 20:11:18.557557 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:11:18.557525    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-4qn8m"]
Nov 03 20:11:18.560900 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:11:18.560869    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-4qn8m"
Nov 03 20:11:18.573756 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:11:18.573733    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-4qn8m"]
Nov 03 20:11:18.584663 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:11:18.584642    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-vhmtn\""
Nov 03 20:11:18.640334 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:11:18.640306    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lpl7\" (UniqueName: \"kubernetes.io/projected/cff8d911-5b38-459d-861b-0908e9bfa631-kube-api-access-6lpl7\") pod \"authorino-57f7f8659f-4qn8m\" (UID: \"cff8d911-5b38-459d-861b-0908e9bfa631\") " pod="kuadrant-system/authorino-57f7f8659f-4qn8m"
Nov 03 20:11:18.741150 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:11:18.741129    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lpl7\" (UniqueName: \"kubernetes.io/projected/cff8d911-5b38-459d-861b-0908e9bfa631-kube-api-access-6lpl7\") pod \"authorino-57f7f8659f-4qn8m\" (UID: \"cff8d911-5b38-459d-861b-0908e9bfa631\") " pod="kuadrant-system/authorino-57f7f8659f-4qn8m"
Nov 03 20:11:18.760262 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:11:18.760240    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lpl7\" (UniqueName: \"kubernetes.io/projected/cff8d911-5b38-459d-861b-0908e9bfa631-kube-api-access-6lpl7\") pod \"authorino-57f7f8659f-4qn8m\" (UID: \"cff8d911-5b38-459d-861b-0908e9bfa631\") " pod="kuadrant-system/authorino-57f7f8659f-4qn8m"
Nov 03 20:11:18.886688 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:11:18.886635    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-4qn8m"
Nov 03 20:11:19.017072 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:11:19.017038    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-n6vk4"]
Nov 03 20:11:19.020452 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:11:19.020431    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-n6vk4"
Nov 03 20:11:19.022542 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:11:19.022516    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g69f8"]
Nov 03 20:11:19.025758 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:11:19.025738    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-g69f8"
Nov 03 20:11:19.038123 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:11:19.038103    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-n6vk4"]
Nov 03 20:11:19.043405 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:11:19.043383    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-hxsl2\""
Nov 03 20:11:19.049097 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:11:19.049074    5570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Nov 03 20:11:19.050082 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:11:19.050062    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g69f8"]
Nov 03 20:11:19.126025 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:11:19.125989    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g69f8"]
Nov 03 20:11:19.144339 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:11:19.144265    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bczq\" (UniqueName: \"kubernetes.io/projected/93d19d0f-57fa-4116-a049-497b592f86f6-kube-api-access-7bczq\") pod \"authorino-56c6c9ddd5-n6vk4\" (UID: \"93d19d0f-57fa-4116-a049-497b592f86f6\") " pod="kuadrant-system/authorino-56c6c9ddd5-n6vk4"
Nov 03 20:11:19.144470 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:11:19.144388    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/818236b5-238c-4342-abce-f375dbebf689-config-file\") pod \"limitador-limitador-7c9b87d574-g69f8\" (UID: \"818236b5-238c-4342-abce-f375dbebf689\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-g69f8"
Nov 03 20:11:19.144470 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:11:19.144444    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hsvs\" (UniqueName: \"kubernetes.io/projected/818236b5-238c-4342-abce-f375dbebf689-kube-api-access-7hsvs\") pod \"limitador-limitador-7c9b87d574-g69f8\" (UID: \"818236b5-238c-4342-abce-f375dbebf689\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-g69f8"
Nov 03 20:11:19.176252 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:11:19.176217    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-4qn8m"]
Nov 03 20:11:19.180865 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:11:19.180835    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcff8d911_5b38_459d_861b_0908e9bfa631.slice/crio-0e5eb87b6c20955cc746c2df37e374e937d25c07cfc51ed4175956269066ac78 WatchSource:0}: Error finding container 0e5eb87b6c20955cc746c2df37e374e937d25c07cfc51ed4175956269066ac78: Status 404 returned error can't find the container with id 0e5eb87b6c20955cc746c2df37e374e937d25c07cfc51ed4175956269066ac78
Nov 03 20:11:19.245495 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:11:19.245464    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/818236b5-238c-4342-abce-f375dbebf689-config-file\") pod \"limitador-limitador-7c9b87d574-g69f8\" (UID: \"818236b5-238c-4342-abce-f375dbebf689\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-g69f8"
Nov 03 20:11:19.245495 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:11:19.245497    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hsvs\" (UniqueName: \"kubernetes.io/projected/818236b5-238c-4342-abce-f375dbebf689-kube-api-access-7hsvs\") pod \"limitador-limitador-7c9b87d574-g69f8\" (UID: \"818236b5-238c-4342-abce-f375dbebf689\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-g69f8"
Nov 03 20:11:19.245715 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:11:19.245552    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bczq\" (UniqueName: \"kubernetes.io/projected/93d19d0f-57fa-4116-a049-497b592f86f6-kube-api-access-7bczq\") pod \"authorino-56c6c9ddd5-n6vk4\" (UID: \"93d19d0f-57fa-4116-a049-497b592f86f6\") " pod="kuadrant-system/authorino-56c6c9ddd5-n6vk4"
Nov 03 20:11:19.246151 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:11:19.246131    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/818236b5-238c-4342-abce-f375dbebf689-config-file\") pod \"limitador-limitador-7c9b87d574-g69f8\" (UID: \"818236b5-238c-4342-abce-f375dbebf689\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-g69f8"
Nov 03 20:11:19.264360 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:11:19.264330    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hsvs\" (UniqueName: \"kubernetes.io/projected/818236b5-238c-4342-abce-f375dbebf689-kube-api-access-7hsvs\") pod \"limitador-limitador-7c9b87d574-g69f8\" (UID: \"818236b5-238c-4342-abce-f375dbebf689\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-g69f8"
Nov 03 20:11:19.264521 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:11:19.264499    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bczq\" (UniqueName: \"kubernetes.io/projected/93d19d0f-57fa-4116-a049-497b592f86f6-kube-api-access-7bczq\") pod \"authorino-56c6c9ddd5-n6vk4\" (UID: \"93d19d0f-57fa-4116-a049-497b592f86f6\") " pod="kuadrant-system/authorino-56c6c9ddd5-n6vk4"
Nov 03 20:11:19.334196 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:11:19.334163    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-n6vk4"
Nov 03 20:11:19.353412 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:11:19.353384    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-g69f8"
Nov 03 20:11:19.519553 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:11:19.519502    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-4qn8m" event={"ID":"cff8d911-5b38-459d-861b-0908e9bfa631","Type":"ContainerStarted","Data":"0e5eb87b6c20955cc746c2df37e374e937d25c07cfc51ed4175956269066ac78"}
Nov 03 20:11:19.631332 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:11:19.631300    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-n6vk4"]
Nov 03 20:11:19.635204 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:11:19.635180    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod93d19d0f_57fa_4116_a049_497b592f86f6.slice/crio-c97feffd39fef87a85383271e6f24d2fb26b880c1a86ec1462e2325a96b61676 WatchSource:0}: Error finding container c97feffd39fef87a85383271e6f24d2fb26b880c1a86ec1462e2325a96b61676: Status 404 returned error can't find the container with id c97feffd39fef87a85383271e6f24d2fb26b880c1a86ec1462e2325a96b61676
Nov 03 20:11:19.663264 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:11:19.663199    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g69f8"]
Nov 03 20:11:19.667697 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:11:19.667670    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod818236b5_238c_4342_abce_f375dbebf689.slice/crio-0f558d0ca91c1b9a73fe9683d3236eb12e0d36f09be2b78d30866e445f50db34 WatchSource:0}: Error finding container 0f558d0ca91c1b9a73fe9683d3236eb12e0d36f09be2b78d30866e445f50db34: Status 404 returned error can't find the container with id 0f558d0ca91c1b9a73fe9683d3236eb12e0d36f09be2b78d30866e445f50db34
Nov 03 20:11:20.541104 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:11:20.541056    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-g69f8" event={"ID":"818236b5-238c-4342-abce-f375dbebf689","Type":"ContainerStarted","Data":"0f558d0ca91c1b9a73fe9683d3236eb12e0d36f09be2b78d30866e445f50db34"}
Nov 03 20:11:20.543291 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:11:20.543247    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-n6vk4" event={"ID":"93d19d0f-57fa-4116-a049-497b592f86f6","Type":"ContainerStarted","Data":"c97feffd39fef87a85383271e6f24d2fb26b880c1a86ec1462e2325a96b61676"}
Nov 03 20:11:21.550238 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:11:21.550158    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-4qn8m" event={"ID":"cff8d911-5b38-459d-861b-0908e9bfa631","Type":"ContainerStarted","Data":"4cc3bef7c3d02da95284e1aae1a4d9b01292aaa8213365e4a847623311702187"}
Nov 03 20:11:21.551682 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:11:21.551658    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-g69f8" event={"ID":"818236b5-238c-4342-abce-f375dbebf689","Type":"ContainerStarted","Data":"6cec3f7ac2b741af74522c17bb97daf9b703e73459dc5ab04cba469c499e097d"}
Nov 03 20:11:21.551802 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:11:21.551723    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-g69f8"
Nov 03 20:11:21.586412 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:11:21.586369    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-4qn8m" podStartSLOduration=1.478761407 podStartE2EDuration="3.586354059s" totalImagesPullingTime="2.107592652s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:11:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:11:21.585851791 +0000 UTC m=+415.738513660" watchObservedRunningTime="2026-11-03 20:11:21.586354059 +0000 UTC m=+415.739015954"
Nov 03 20:11:21.625470 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:11:21.625425    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-g69f8" podStartSLOduration=2.729695022 podStartE2EDuration="3.625411042s" totalImagesPullingTime="895.71602ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:11:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:11:21.624707213 +0000 UTC m=+415.777369100" watchObservedRunningTime="2026-11-03 20:11:21.625411042 +0000 UTC m=+415.778072932"
Nov 03 20:11:22.557436 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:11:22.557393    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-n6vk4" event={"ID":"93d19d0f-57fa-4116-a049-497b592f86f6","Type":"ContainerStarted","Data":"baa75473872efb652e99347b32fab68e345e1eceeba108cba8344ff7717ff353"}
Nov 03 20:11:22.597482 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:11:22.597434    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-n6vk4" podStartSLOduration=2.649992436 podStartE2EDuration="4.597409404s" totalImagesPullingTime="1.947416968s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:11:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:11:22.596279611 +0000 UTC m=+416.748941499" watchObservedRunningTime="2026-11-03 20:11:22.597409404 +0000 UTC m=+416.750071291"
Nov 03 20:11:22.621989 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:11:22.621955    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-4qn8m"]
Nov 03 20:11:23.561283 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:11:23.561239    5570 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-4qn8m" podUID="cff8d911-5b38-459d-861b-0908e9bfa631" containerName="authorino" containerID="cri-o://4cc3bef7c3d02da95284e1aae1a4d9b01292aaa8213365e4a847623311702187" gracePeriod=30
Nov 03 20:11:23.798408 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:11:23.798386    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-4qn8m"
Nov 03 20:11:23.886222 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:11:23.886140    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cff8d911-5b38-459d-861b-0908e9bfa631-kube-api-access-6lpl7\" (UniqueName: \"kubernetes.io/projected/cff8d911-5b38-459d-861b-0908e9bfa631-kube-api-access-6lpl7\") pod \"cff8d911-5b38-459d-861b-0908e9bfa631\" (UID: \"cff8d911-5b38-459d-861b-0908e9bfa631\") "
Nov 03 20:11:23.888326 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:11:23.888297    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cff8d911-5b38-459d-861b-0908e9bfa631-kube-api-access-6lpl7" pod "cff8d911-5b38-459d-861b-0908e9bfa631" (UID: "cff8d911-5b38-459d-861b-0908e9bfa631"). InnerVolumeSpecName "kube-api-access-6lpl7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:11:23.987628 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:11:23.987604    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6lpl7\" (UniqueName: \"kubernetes.io/projected/cff8d911-5b38-459d-861b-0908e9bfa631-kube-api-access-6lpl7\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:11:24.565948 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:11:24.565904    5570 generic.go:356] "Generic (PLEG): container finished" podID="cff8d911-5b38-459d-861b-0908e9bfa631" containerID="4cc3bef7c3d02da95284e1aae1a4d9b01292aaa8213365e4a847623311702187" exitCode=0
Nov 03 20:11:24.565948 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:11:24.565952    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-4qn8m"
Nov 03 20:11:24.566497 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:11:24.565989    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-4qn8m" event={"ID":"cff8d911-5b38-459d-861b-0908e9bfa631","Type":"ContainerDied","Data":"4cc3bef7c3d02da95284e1aae1a4d9b01292aaa8213365e4a847623311702187"}
Nov 03 20:11:24.566497 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:11:24.566035    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-4qn8m" event={"ID":"cff8d911-5b38-459d-861b-0908e9bfa631","Type":"ContainerDied","Data":"0e5eb87b6c20955cc746c2df37e374e937d25c07cfc51ed4175956269066ac78"}
Nov 03 20:11:24.566497 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:11:24.566052    5570 scope.go:122] "RemoveContainer" containerID="4cc3bef7c3d02da95284e1aae1a4d9b01292aaa8213365e4a847623311702187"
Nov 03 20:11:24.574034 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:11:24.574006    5570 scope.go:122] "RemoveContainer" containerID="4cc3bef7c3d02da95284e1aae1a4d9b01292aaa8213365e4a847623311702187"
Nov 03 20:11:24.574289 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:11:24.574266    5570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4cc3bef7c3d02da95284e1aae1a4d9b01292aaa8213365e4a847623311702187\": container with ID starting with 4cc3bef7c3d02da95284e1aae1a4d9b01292aaa8213365e4a847623311702187 not found: ID does not exist" containerID="4cc3bef7c3d02da95284e1aae1a4d9b01292aaa8213365e4a847623311702187"
Nov 03 20:11:24.574344 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:11:24.574302    5570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4cc3bef7c3d02da95284e1aae1a4d9b01292aaa8213365e4a847623311702187"} err="failed to get container status \"4cc3bef7c3d02da95284e1aae1a4d9b01292aaa8213365e4a847623311702187\": rpc error: code = NotFound desc = could not find container \"4cc3bef7c3d02da95284e1aae1a4d9b01292aaa8213365e4a847623311702187\": container with ID starting with 4cc3bef7c3d02da95284e1aae1a4d9b01292aaa8213365e4a847623311702187 not found: ID does not exist"
Nov 03 20:11:24.613380 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:11:24.613352    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-4qn8m"]
Nov 03 20:11:24.616823 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:11:24.616799    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-4qn8m"]
Nov 03 20:11:25.962906 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:11:25.962875    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cff8d911-5b38-459d-861b-0908e9bfa631" path="/var/lib/kubelet/pods/cff8d911-5b38-459d-861b-0908e9bfa631/volumes"
Nov 03 20:11:27.226193 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:11:27.226157    5570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8bwhz" containerName="extractor"
Nov 03 20:11:32.558767 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:11:32.558733    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-g69f8"
Nov 03 20:11:57.226905 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:11:57.226865    5570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8bwhz" containerName="extractor"
Nov 03 20:12:03.427277 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:12:03.427244    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5mwhf"]
Nov 03 20:12:03.427734 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:12:03.427716    5570 cpu_manager.go:414] "containerMap: removing container" podUID="cff8d911-5b38-459d-861b-0908e9bfa631" containerName="authorino"
Nov 03 20:12:03.427802 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:12:03.427736    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cff8d911-5b38-459d-861b-0908e9bfa631" containerName="authorino"
Nov 03 20:12:03.427849 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:12:03.427839    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="cff8d911-5b38-459d-861b-0908e9bfa631" containerName="authorino"
Nov 03 20:12:03.432937 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:12:03.432918    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5mwhf"
Nov 03 20:12:03.470462 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:12:03.470427    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5mwhf"]
Nov 03 20:12:03.525264 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:12:03.525227    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8dnf\" (UniqueName: \"kubernetes.io/projected/be4487de-c7d5-4a34-9064-a38ecfee6810-kube-api-access-m8dnf\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5mwhf\" (UID: \"be4487de-c7d5-4a34-9064-a38ecfee6810\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5mwhf"
Nov 03 20:12:03.525437 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:12:03.525291    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/be4487de-c7d5-4a34-9064-a38ecfee6810-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5mwhf\" (UID: \"be4487de-c7d5-4a34-9064-a38ecfee6810\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5mwhf"
Nov 03 20:12:03.525437 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:12:03.525364    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/be4487de-c7d5-4a34-9064-a38ecfee6810-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5mwhf\" (UID: \"be4487de-c7d5-4a34-9064-a38ecfee6810\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5mwhf"
Nov 03 20:12:03.525437 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:12:03.525386    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be4487de-c7d5-4a34-9064-a38ecfee6810-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5mwhf\" (UID: \"be4487de-c7d5-4a34-9064-a38ecfee6810\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5mwhf"
Nov 03 20:12:03.525437 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:12:03.525410    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/be4487de-c7d5-4a34-9064-a38ecfee6810-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5mwhf\" (UID: \"be4487de-c7d5-4a34-9064-a38ecfee6810\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5mwhf"
Nov 03 20:12:03.525591 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:12:03.525444    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/be4487de-c7d5-4a34-9064-a38ecfee6810-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5mwhf\" (UID: \"be4487de-c7d5-4a34-9064-a38ecfee6810\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5mwhf"
Nov 03 20:12:03.525591 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:12:03.525528    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/be4487de-c7d5-4a34-9064-a38ecfee6810-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5mwhf\" (UID: \"be4487de-c7d5-4a34-9064-a38ecfee6810\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5mwhf"
Nov 03 20:12:03.626289 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:12:03.626256    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/be4487de-c7d5-4a34-9064-a38ecfee6810-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5mwhf\" (UID: \"be4487de-c7d5-4a34-9064-a38ecfee6810\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5mwhf"
Nov 03 20:12:03.626464 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:12:03.626303    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/be4487de-c7d5-4a34-9064-a38ecfee6810-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5mwhf\" (UID: \"be4487de-c7d5-4a34-9064-a38ecfee6810\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5mwhf"
Nov 03 20:12:03.626464 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:12:03.626415    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be4487de-c7d5-4a34-9064-a38ecfee6810-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5mwhf\" (UID: \"be4487de-c7d5-4a34-9064-a38ecfee6810\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5mwhf"
Nov 03 20:12:03.626464 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:12:03.626452    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/be4487de-c7d5-4a34-9064-a38ecfee6810-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5mwhf\" (UID: \"be4487de-c7d5-4a34-9064-a38ecfee6810\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5mwhf"
Nov 03 20:12:03.626647 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:12:03.626622    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/be4487de-c7d5-4a34-9064-a38ecfee6810-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5mwhf\" (UID: \"be4487de-c7d5-4a34-9064-a38ecfee6810\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5mwhf"
Nov 03 20:12:03.626711 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:12:03.626671    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/be4487de-c7d5-4a34-9064-a38ecfee6810-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5mwhf\" (UID: \"be4487de-c7d5-4a34-9064-a38ecfee6810\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5mwhf"
Nov 03 20:12:03.626882 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:12:03.626765    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8dnf\" (UniqueName: \"kubernetes.io/projected/be4487de-c7d5-4a34-9064-a38ecfee6810-kube-api-access-m8dnf\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5mwhf\" (UID: \"be4487de-c7d5-4a34-9064-a38ecfee6810\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5mwhf"
Nov 03 20:12:03.627308 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:12:03.627280    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/be4487de-c7d5-4a34-9064-a38ecfee6810-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5mwhf\" (UID: \"be4487de-c7d5-4a34-9064-a38ecfee6810\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5mwhf"
Nov 03 20:12:03.628880 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:12:03.628858    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/be4487de-c7d5-4a34-9064-a38ecfee6810-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5mwhf\" (UID: \"be4487de-c7d5-4a34-9064-a38ecfee6810\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5mwhf"
Nov 03 20:12:03.628972 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:12:03.628880    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/be4487de-c7d5-4a34-9064-a38ecfee6810-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5mwhf\" (UID: \"be4487de-c7d5-4a34-9064-a38ecfee6810\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5mwhf"
Nov 03 20:12:03.629063 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:12:03.629042    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/be4487de-c7d5-4a34-9064-a38ecfee6810-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5mwhf\" (UID: \"be4487de-c7d5-4a34-9064-a38ecfee6810\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5mwhf"
Nov 03 20:12:03.629103 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:12:03.629072    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/be4487de-c7d5-4a34-9064-a38ecfee6810-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5mwhf\" (UID: \"be4487de-c7d5-4a34-9064-a38ecfee6810\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5mwhf"
Nov 03 20:12:03.645457 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:12:03.645430    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be4487de-c7d5-4a34-9064-a38ecfee6810-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5mwhf\" (UID: \"be4487de-c7d5-4a34-9064-a38ecfee6810\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5mwhf"
Nov 03 20:12:03.645805 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:12:03.645749    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8dnf\" (UniqueName: \"kubernetes.io/projected/be4487de-c7d5-4a34-9064-a38ecfee6810-kube-api-access-m8dnf\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5mwhf\" (UID: \"be4487de-c7d5-4a34-9064-a38ecfee6810\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5mwhf"
Nov 03 20:12:03.745829 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:12:03.745797    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5mwhf"
Nov 03 20:12:04.057205 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:12:04.057104    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5mwhf"]
Nov 03 20:12:04.066318 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:12:04.061718    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe4487de_c7d5_4a34_9064_a38ecfee6810.slice/crio-f51a6104a1fcfc8af5e2ff53143c4a764a3d45c840d88ade1bd4d28629aa25eb WatchSource:0}: Error finding container f51a6104a1fcfc8af5e2ff53143c4a764a3d45c840d88ade1bd4d28629aa25eb: Status 404 returned error can't find the container with id f51a6104a1fcfc8af5e2ff53143c4a764a3d45c840d88ade1bd4d28629aa25eb
Nov 03 20:12:04.067176 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:12:04.067142    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:12:04.067289 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:12:04.067204    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:12:04.741220 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:12:04.741184    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5mwhf" event={"ID":"be4487de-c7d5-4a34-9064-a38ecfee6810","Type":"ContainerStarted","Data":"a73ae5305cd5793ca7d8289bda536cebbbb7712bf6f220b3643ce275191a90bc"}
Nov 03 20:12:04.741741 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:12:04.741265    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5mwhf" event={"ID":"be4487de-c7d5-4a34-9064-a38ecfee6810","Type":"ContainerStarted","Data":"f51a6104a1fcfc8af5e2ff53143c4a764a3d45c840d88ade1bd4d28629aa25eb"}
Nov 03 20:12:04.741741 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:12:04.741301    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5mwhf"
Nov 03 20:12:04.805855 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:12:04.805801    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5mwhf" podStartSLOduration=1.8057618130000002 podStartE2EDuration="1.805761813s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:12:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:12:04.793466074 +0000 UTC m=+458.946127962" watchObservedRunningTime="2026-11-03 20:12:04.805761813 +0000 UTC m=+458.958423699"
Nov 03 20:12:05.747485 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:12:05.747457    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5mwhf"
Nov 03 20:12:05.830415 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:12:05.830362    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-lh8mg"]
Nov 03 20:12:05.830744 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:12:05.830633    5570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lh8mg" podUID="a89fe83e-c27c-4124-98e3-04cf40ce08e8" containerName="discovery" containerID="cri-o://358bf923bd3ed783e1976c92afb6c96fa26224257b001b7004029d14da3f853b" gracePeriod=30
Nov 03 20:12:06.071959 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:12:06.071937    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lh8mg"
Nov 03 20:12:06.145163 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:12:06.145132    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a89fe83e-c27c-4124-98e3-04cf40ce08e8-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a89fe83e-c27c-4124-98e3-04cf40ce08e8-istio-kubeconfig\") pod \"a89fe83e-c27c-4124-98e3-04cf40ce08e8\" (UID: \"a89fe83e-c27c-4124-98e3-04cf40ce08e8\") "
Nov 03 20:12:06.145357 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:12:06.145194    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a89fe83e-c27c-4124-98e3-04cf40ce08e8-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a89fe83e-c27c-4124-98e3-04cf40ce08e8-istio-csr-ca-configmap\") pod \"a89fe83e-c27c-4124-98e3-04cf40ce08e8\" (UID: \"a89fe83e-c27c-4124-98e3-04cf40ce08e8\") "
Nov 03 20:12:06.145357 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:12:06.145227    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a89fe83e-c27c-4124-98e3-04cf40ce08e8-cacerts\" (UniqueName: \"kubernetes.io/secret/a89fe83e-c27c-4124-98e3-04cf40ce08e8-cacerts\") pod \"a89fe83e-c27c-4124-98e3-04cf40ce08e8\" (UID: \"a89fe83e-c27c-4124-98e3-04cf40ce08e8\") "
Nov 03 20:12:06.145357 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:12:06.145277    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a89fe83e-c27c-4124-98e3-04cf40ce08e8-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a89fe83e-c27c-4124-98e3-04cf40ce08e8-istio-csr-dns-cert\") pod \"a89fe83e-c27c-4124-98e3-04cf40ce08e8\" (UID: \"a89fe83e-c27c-4124-98e3-04cf40ce08e8\") "
Nov 03 20:12:06.145357 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:12:06.145329    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a89fe83e-c27c-4124-98e3-04cf40ce08e8-kube-api-access-zm59b\" (UniqueName: \"kubernetes.io/projected/a89fe83e-c27c-4124-98e3-04cf40ce08e8-kube-api-access-zm59b\") pod \"a89fe83e-c27c-4124-98e3-04cf40ce08e8\" (UID: \"a89fe83e-c27c-4124-98e3-04cf40ce08e8\") "
Nov 03 20:12:06.145564 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:12:06.145379    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a89fe83e-c27c-4124-98e3-04cf40ce08e8-istio-token\" (UniqueName: \"kubernetes.io/projected/a89fe83e-c27c-4124-98e3-04cf40ce08e8-istio-token\") pod \"a89fe83e-c27c-4124-98e3-04cf40ce08e8\" (UID: \"a89fe83e-c27c-4124-98e3-04cf40ce08e8\") "
Nov 03 20:12:06.145692 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:12:06.145661    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a89fe83e-c27c-4124-98e3-04cf40ce08e8-istio-csr-ca-configmap" pod "a89fe83e-c27c-4124-98e3-04cf40ce08e8" (UID: "a89fe83e-c27c-4124-98e3-04cf40ce08e8"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:12:06.147611 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:12:06.147581    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a89fe83e-c27c-4124-98e3-04cf40ce08e8-istio-token" pod "a89fe83e-c27c-4124-98e3-04cf40ce08e8" (UID: "a89fe83e-c27c-4124-98e3-04cf40ce08e8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:12:06.147702 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:12:06.147635    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a89fe83e-c27c-4124-98e3-04cf40ce08e8-istio-csr-dns-cert" pod "a89fe83e-c27c-4124-98e3-04cf40ce08e8" (UID: "a89fe83e-c27c-4124-98e3-04cf40ce08e8"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:12:06.147702 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:12:06.147666    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a89fe83e-c27c-4124-98e3-04cf40ce08e8-istio-kubeconfig" pod "a89fe83e-c27c-4124-98e3-04cf40ce08e8" (UID: "a89fe83e-c27c-4124-98e3-04cf40ce08e8"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:12:06.147887 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:12:06.147859    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a89fe83e-c27c-4124-98e3-04cf40ce08e8-kube-api-access-zm59b" pod "a89fe83e-c27c-4124-98e3-04cf40ce08e8" (UID: "a89fe83e-c27c-4124-98e3-04cf40ce08e8"). InnerVolumeSpecName "kube-api-access-zm59b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:12:06.147887 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:12:06.147874    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a89fe83e-c27c-4124-98e3-04cf40ce08e8-cacerts" pod "a89fe83e-c27c-4124-98e3-04cf40ce08e8" (UID: "a89fe83e-c27c-4124-98e3-04cf40ce08e8"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:12:06.245876 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:12:06.245847    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a89fe83e-c27c-4124-98e3-04cf40ce08e8-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a89fe83e-c27c-4124-98e3-04cf40ce08e8-local-certs\") pod \"a89fe83e-c27c-4124-98e3-04cf40ce08e8\" (UID: \"a89fe83e-c27c-4124-98e3-04cf40ce08e8\") "
Nov 03 20:12:06.246069 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:12:06.246052    5570 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a89fe83e-c27c-4124-98e3-04cf40ce08e8-istio-csr-ca-configmap\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:12:06.246113 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:12:06.246076    5570 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a89fe83e-c27c-4124-98e3-04cf40ce08e8-cacerts\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:12:06.246113 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:12:06.246092    5570 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a89fe83e-c27c-4124-98e3-04cf40ce08e8-istio-csr-dns-cert\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:12:06.246113 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:12:06.246105    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zm59b\" (UniqueName: \"kubernetes.io/projected/a89fe83e-c27c-4124-98e3-04cf40ce08e8-kube-api-access-zm59b\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:12:06.246213 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:12:06.246117    5570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a89fe83e-c27c-4124-98e3-04cf40ce08e8-istio-token\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:12:06.246213 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:12:06.246132    5570 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a89fe83e-c27c-4124-98e3-04cf40ce08e8-istio-kubeconfig\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:12:06.247921 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:12:06.247896    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a89fe83e-c27c-4124-98e3-04cf40ce08e8-local-certs" pod "a89fe83e-c27c-4124-98e3-04cf40ce08e8" (UID: "a89fe83e-c27c-4124-98e3-04cf40ce08e8"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:12:06.347242 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:12:06.347167    5570 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a89fe83e-c27c-4124-98e3-04cf40ce08e8-local-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:12:06.750423 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:12:06.750391    5570 generic.go:356] "Generic (PLEG): container finished" podID="a89fe83e-c27c-4124-98e3-04cf40ce08e8" containerID="358bf923bd3ed783e1976c92afb6c96fa26224257b001b7004029d14da3f853b" exitCode=0
Nov 03 20:12:06.750866 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:12:06.750445    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lh8mg"
Nov 03 20:12:06.750866 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:12:06.750461    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lh8mg" event={"ID":"a89fe83e-c27c-4124-98e3-04cf40ce08e8","Type":"ContainerDied","Data":"358bf923bd3ed783e1976c92afb6c96fa26224257b001b7004029d14da3f853b"}
Nov 03 20:12:06.750866 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:12:06.750512    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lh8mg" event={"ID":"a89fe83e-c27c-4124-98e3-04cf40ce08e8","Type":"ContainerDied","Data":"385737090e64e424d5fd6e3a2a3b1b13390a78f6e7eac9222ca85eb0fbd14072"}
Nov 03 20:12:06.750866 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:12:06.750536    5570 scope.go:122] "RemoveContainer" containerID="358bf923bd3ed783e1976c92afb6c96fa26224257b001b7004029d14da3f853b"
Nov 03 20:12:06.759325 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:12:06.759307    5570 scope.go:122] "RemoveContainer" containerID="358bf923bd3ed783e1976c92afb6c96fa26224257b001b7004029d14da3f853b"
Nov 03 20:12:06.759608 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:12:06.759590    5570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"358bf923bd3ed783e1976c92afb6c96fa26224257b001b7004029d14da3f853b\": container with ID starting with 358bf923bd3ed783e1976c92afb6c96fa26224257b001b7004029d14da3f853b not found: ID does not exist" containerID="358bf923bd3ed783e1976c92afb6c96fa26224257b001b7004029d14da3f853b"
Nov 03 20:12:06.759656 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:12:06.759618    5570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"358bf923bd3ed783e1976c92afb6c96fa26224257b001b7004029d14da3f853b"} err="failed to get container status \"358bf923bd3ed783e1976c92afb6c96fa26224257b001b7004029d14da3f853b\": rpc error: code = NotFound desc = could not find container \"358bf923bd3ed783e1976c92afb6c96fa26224257b001b7004029d14da3f853b\": container with ID starting with 358bf923bd3ed783e1976c92afb6c96fa26224257b001b7004029d14da3f853b not found: ID does not exist"
Nov 03 20:12:06.810406 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:12:06.810376    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-lh8mg"]
Nov 03 20:12:06.822424 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:12:06.822392    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-lh8mg"]
Nov 03 20:12:07.963725 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:12:07.963682    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a89fe83e-c27c-4124-98e3-04cf40ce08e8" path="/var/lib/kubelet/pods/a89fe83e-c27c-4124-98e3-04cf40ce08e8/volumes"
Nov 03 20:12:27.226665 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:12:27.226629    5570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8bwhz" containerName="extractor"
Nov 03 20:12:35.569250 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:12:35.569210    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g69f8"]
Nov 03 20:12:43.819615 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:12:43.819580    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-hk5gj"]
Nov 03 20:12:43.820118 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:12:43.819961    5570 cpu_manager.go:414] "containerMap: removing container" podUID="a89fe83e-c27c-4124-98e3-04cf40ce08e8" containerName="discovery"
Nov 03 20:12:43.820118 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:12:43.819975    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a89fe83e-c27c-4124-98e3-04cf40ce08e8" containerName="discovery"
Nov 03 20:12:43.820118 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:12:43.820042    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="a89fe83e-c27c-4124-98e3-04cf40ce08e8" containerName="discovery"
Nov 03 20:12:43.823104 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:12:43.823086    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-hk5gj"
Nov 03 20:12:43.837982 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:12:43.837952    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-hk5gj"]
Nov 03 20:12:43.980746 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:12:43.980713    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbhb5\" (UniqueName: \"kubernetes.io/projected/227ed64b-cb2f-40ec-9893-7535c002b66c-kube-api-access-cbhb5\") pod \"authorino-6f6944bcdd-hk5gj\" (UID: \"227ed64b-cb2f-40ec-9893-7535c002b66c\") " pod="kuadrant-system/authorino-6f6944bcdd-hk5gj"
Nov 03 20:12:44.081724 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:12:44.081616    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbhb5\" (UniqueName: \"kubernetes.io/projected/227ed64b-cb2f-40ec-9893-7535c002b66c-kube-api-access-cbhb5\") pod \"authorino-6f6944bcdd-hk5gj\" (UID: \"227ed64b-cb2f-40ec-9893-7535c002b66c\") " pod="kuadrant-system/authorino-6f6944bcdd-hk5gj"
Nov 03 20:12:44.102107 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:12:44.102074    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbhb5\" (UniqueName: \"kubernetes.io/projected/227ed64b-cb2f-40ec-9893-7535c002b66c-kube-api-access-cbhb5\") pod \"authorino-6f6944bcdd-hk5gj\" (UID: \"227ed64b-cb2f-40ec-9893-7535c002b66c\") " pod="kuadrant-system/authorino-6f6944bcdd-hk5gj"
Nov 03 20:12:44.134924 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:12:44.134891    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-hk5gj"
Nov 03 20:12:44.151029 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:12:44.151002    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-hk5gj"]
Nov 03 20:12:44.182385 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:12:44.181462    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7c96554b5c-jfjnh"]
Nov 03 20:12:44.185519 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:12:44.185492    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7c96554b5c-jfjnh"
Nov 03 20:12:44.202863 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:12:44.202756    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7c96554b5c-jfjnh"]
Nov 03 20:12:44.284310 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:12:44.284272    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrjn5\" (UniqueName: \"kubernetes.io/projected/4665dd4d-da21-4946-9d74-c57654293ac7-kube-api-access-nrjn5\") pod \"authorino-7c96554b5c-jfjnh\" (UID: \"4665dd4d-da21-4946-9d74-c57654293ac7\") " pod="kuadrant-system/authorino-7c96554b5c-jfjnh"
Nov 03 20:12:44.384922 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:12:44.384834    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrjn5\" (UniqueName: \"kubernetes.io/projected/4665dd4d-da21-4946-9d74-c57654293ac7-kube-api-access-nrjn5\") pod \"authorino-7c96554b5c-jfjnh\" (UID: \"4665dd4d-da21-4946-9d74-c57654293ac7\") " pod="kuadrant-system/authorino-7c96554b5c-jfjnh"
Nov 03 20:12:44.403316 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:12:44.403288    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrjn5\" (UniqueName: \"kubernetes.io/projected/4665dd4d-da21-4946-9d74-c57654293ac7-kube-api-access-nrjn5\") pod \"authorino-7c96554b5c-jfjnh\" (UID: \"4665dd4d-da21-4946-9d74-c57654293ac7\") " pod="kuadrant-system/authorino-7c96554b5c-jfjnh"
Nov 03 20:12:44.425977 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:12:44.425942    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-hk5gj"]
Nov 03 20:12:44.430785 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:12:44.430743    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod227ed64b_cb2f_40ec_9893_7535c002b66c.slice/crio-2a63c6f0742a858fdffc2ab96da126cd685001329eab0b86f757fdbbc0142f49 WatchSource:0}: Error finding container 2a63c6f0742a858fdffc2ab96da126cd685001329eab0b86f757fdbbc0142f49: Status 404 returned error can't find the container with id 2a63c6f0742a858fdffc2ab96da126cd685001329eab0b86f757fdbbc0142f49
Nov 03 20:12:44.501237 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:12:44.501202    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7c96554b5c-jfjnh"
Nov 03 20:12:44.732959 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:12:44.732921    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7c96554b5c-jfjnh"]
Nov 03 20:12:44.758928 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:12:44.758896    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d9f5db59-vggm4"]
Nov 03 20:12:44.763386 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:12:44.763364    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d9f5db59-vggm4"
Nov 03 20:12:44.777099 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:12:44.777070    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d9f5db59-vggm4"]
Nov 03 20:12:44.788225 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:12:44.788199    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Nov 03 20:12:44.792161 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:12:44.792137    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7c96554b5c-jfjnh"]
Nov 03 20:12:44.796563 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:12:44.796535    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4665dd4d_da21_4946_9d74_c57654293ac7.slice/crio-7d3f7040aafe85c93e86da6a613f7a4577ffaf94d930bfdaf01d5ec72101f764 WatchSource:0}: Error finding container 7d3f7040aafe85c93e86da6a613f7a4577ffaf94d930bfdaf01d5ec72101f764: Status 404 returned error can't find the container with id 7d3f7040aafe85c93e86da6a613f7a4577ffaf94d930bfdaf01d5ec72101f764
Nov 03 20:12:44.890004 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:12:44.889972    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ca3c6734-bb2a-487b-9c6d-c2836a8eea39-tls-cert\") pod \"authorino-75d9f5db59-vggm4\" (UID: \"ca3c6734-bb2a-487b-9c6d-c2836a8eea39\") " pod="kuadrant-system/authorino-75d9f5db59-vggm4"
Nov 03 20:12:44.890424 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:12:44.890015    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqw9c\" (UniqueName: \"kubernetes.io/projected/ca3c6734-bb2a-487b-9c6d-c2836a8eea39-kube-api-access-cqw9c\") pod \"authorino-75d9f5db59-vggm4\" (UID: \"ca3c6734-bb2a-487b-9c6d-c2836a8eea39\") " pod="kuadrant-system/authorino-75d9f5db59-vggm4"
Nov 03 20:12:44.915456 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:12:44.915418    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-hk5gj" event={"ID":"227ed64b-cb2f-40ec-9893-7535c002b66c","Type":"ContainerStarted","Data":"2a63c6f0742a858fdffc2ab96da126cd685001329eab0b86f757fdbbc0142f49"}
Nov 03 20:12:44.916560 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:12:44.916524    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7c96554b5c-jfjnh" event={"ID":"4665dd4d-da21-4946-9d74-c57654293ac7","Type":"ContainerStarted","Data":"7d3f7040aafe85c93e86da6a613f7a4577ffaf94d930bfdaf01d5ec72101f764"}
Nov 03 20:12:44.991015 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:12:44.990925    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ca3c6734-bb2a-487b-9c6d-c2836a8eea39-tls-cert\") pod \"authorino-75d9f5db59-vggm4\" (UID: \"ca3c6734-bb2a-487b-9c6d-c2836a8eea39\") " pod="kuadrant-system/authorino-75d9f5db59-vggm4"
Nov 03 20:12:44.991015 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:12:44.990974    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqw9c\" (UniqueName: \"kubernetes.io/projected/ca3c6734-bb2a-487b-9c6d-c2836a8eea39-kube-api-access-cqw9c\") pod \"authorino-75d9f5db59-vggm4\" (UID: \"ca3c6734-bb2a-487b-9c6d-c2836a8eea39\") " pod="kuadrant-system/authorino-75d9f5db59-vggm4"
Nov 03 20:12:44.993565 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:12:44.993540    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ca3c6734-bb2a-487b-9c6d-c2836a8eea39-tls-cert\") pod \"authorino-75d9f5db59-vggm4\" (UID: \"ca3c6734-bb2a-487b-9c6d-c2836a8eea39\") " pod="kuadrant-system/authorino-75d9f5db59-vggm4"
Nov 03 20:12:45.009546 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:12:45.009517    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqw9c\" (UniqueName: \"kubernetes.io/projected/ca3c6734-bb2a-487b-9c6d-c2836a8eea39-kube-api-access-cqw9c\") pod \"authorino-75d9f5db59-vggm4\" (UID: \"ca3c6734-bb2a-487b-9c6d-c2836a8eea39\") " pod="kuadrant-system/authorino-75d9f5db59-vggm4"
Nov 03 20:12:45.076048 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:12:45.076009    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d9f5db59-vggm4"
Nov 03 20:12:45.379439 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:12:45.379403    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d9f5db59-vggm4"]
Nov 03 20:12:45.385011 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:12:45.384976    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podca3c6734_bb2a_487b_9c6d_c2836a8eea39.slice/crio-6913e6afb4b60a03b7bb8a9dbbb5ee6562504d4eb1e5171006058d6985d92a49 WatchSource:0}: Error finding container 6913e6afb4b60a03b7bb8a9dbbb5ee6562504d4eb1e5171006058d6985d92a49: Status 404 returned error can't find the container with id 6913e6afb4b60a03b7bb8a9dbbb5ee6562504d4eb1e5171006058d6985d92a49
Nov 03 20:12:45.926445 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:12:45.926405    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d9f5db59-vggm4" event={"ID":"ca3c6734-bb2a-487b-9c6d-c2836a8eea39","Type":"ContainerStarted","Data":"6913e6afb4b60a03b7bb8a9dbbb5ee6562504d4eb1e5171006058d6985d92a49"}
Nov 03 20:12:45.928760 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:12:45.928140    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-hk5gj" event={"ID":"227ed64b-cb2f-40ec-9893-7535c002b66c","Type":"ContainerStarted","Data":"f40ac8735f87a91337558d92a66a6b714cf63d9336278a2841521523434ea8d8"}
Nov 03 20:12:45.932822 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:12:45.928912    5570 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-6f6944bcdd-hk5gj" podUID="227ed64b-cb2f-40ec-9893-7535c002b66c" containerName="authorino" containerID="cri-o://f40ac8735f87a91337558d92a66a6b714cf63d9336278a2841521523434ea8d8" gracePeriod=30
Nov 03 20:12:45.932957 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:12:45.931575    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7c96554b5c-jfjnh" event={"ID":"4665dd4d-da21-4946-9d74-c57654293ac7","Type":"ContainerStarted","Data":"82ad9104018c34362662e8e414ae4c5e719f9f031508b90be4506957ff4a52d1"}
Nov 03 20:12:45.932957 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:12:45.931655    5570 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-7c96554b5c-jfjnh" podUID="4665dd4d-da21-4946-9d74-c57654293ac7" containerName="authorino" containerID="cri-o://82ad9104018c34362662e8e414ae4c5e719f9f031508b90be4506957ff4a52d1" gracePeriod=30
Nov 03 20:12:45.967316 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:12:45.967267    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6f6944bcdd-hk5gj" podStartSLOduration=2.30697176 podStartE2EDuration="2.967253472s" totalImagesPullingTime="660.281712ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:12:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:12:45.966363719 +0000 UTC m=+500.119025610" watchObservedRunningTime="2026-11-03 20:12:45.967253472 +0000 UTC m=+500.119915358"
Nov 03 20:12:46.005939 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:12:46.005880    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-7c96554b5c-jfjnh" podStartSLOduration=1.510028127 podStartE2EDuration="2.005865156s" totalImagesPullingTime="495.837029ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:12:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:12:46.004986679 +0000 UTC m=+500.157648567" watchObservedRunningTime="2026-11-03 20:12:46.005865156 +0000 UTC m=+500.158527069"
Nov 03 20:12:46.228373 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:12:46.228350    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-hk5gj"
Nov 03 20:12:46.231871 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:12:46.231848    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7c96554b5c-jfjnh"
Nov 03 20:12:46.414505 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:12:46.414413    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4665dd4d-da21-4946-9d74-c57654293ac7-kube-api-access-nrjn5\" (UniqueName: \"kubernetes.io/projected/4665dd4d-da21-4946-9d74-c57654293ac7-kube-api-access-nrjn5\") pod \"4665dd4d-da21-4946-9d74-c57654293ac7\" (UID: \"4665dd4d-da21-4946-9d74-c57654293ac7\") "
Nov 03 20:12:46.414684 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:12:46.414514    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/227ed64b-cb2f-40ec-9893-7535c002b66c-kube-api-access-cbhb5\" (UniqueName: \"kubernetes.io/projected/227ed64b-cb2f-40ec-9893-7535c002b66c-kube-api-access-cbhb5\") pod \"227ed64b-cb2f-40ec-9893-7535c002b66c\" (UID: \"227ed64b-cb2f-40ec-9893-7535c002b66c\") "
Nov 03 20:12:46.416877 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:12:46.416809    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4665dd4d-da21-4946-9d74-c57654293ac7-kube-api-access-nrjn5" pod "4665dd4d-da21-4946-9d74-c57654293ac7" (UID: "4665dd4d-da21-4946-9d74-c57654293ac7"). InnerVolumeSpecName "kube-api-access-nrjn5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:12:46.417007 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:12:46.416904    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/227ed64b-cb2f-40ec-9893-7535c002b66c-kube-api-access-cbhb5" pod "227ed64b-cb2f-40ec-9893-7535c002b66c" (UID: "227ed64b-cb2f-40ec-9893-7535c002b66c"). InnerVolumeSpecName "kube-api-access-cbhb5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:12:46.516992 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:12:46.516948    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrjn5\" (UniqueName: \"kubernetes.io/projected/4665dd4d-da21-4946-9d74-c57654293ac7-kube-api-access-nrjn5\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:12:46.516992 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:12:46.516982    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbhb5\" (UniqueName: \"kubernetes.io/projected/227ed64b-cb2f-40ec-9893-7535c002b66c-kube-api-access-cbhb5\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:12:46.937791 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:12:46.937742    5570 generic.go:356] "Generic (PLEG): container finished" podID="4665dd4d-da21-4946-9d74-c57654293ac7" containerID="82ad9104018c34362662e8e414ae4c5e719f9f031508b90be4506957ff4a52d1" exitCode=0
Nov 03 20:12:46.938272 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:12:46.937802    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7c96554b5c-jfjnh" event={"ID":"4665dd4d-da21-4946-9d74-c57654293ac7","Type":"ContainerDied","Data":"82ad9104018c34362662e8e414ae4c5e719f9f031508b90be4506957ff4a52d1"}
Nov 03 20:12:46.938272 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:12:46.937817    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7c96554b5c-jfjnh"
Nov 03 20:12:46.938272 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:12:46.937847    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7c96554b5c-jfjnh" event={"ID":"4665dd4d-da21-4946-9d74-c57654293ac7","Type":"ContainerDied","Data":"7d3f7040aafe85c93e86da6a613f7a4577ffaf94d930bfdaf01d5ec72101f764"}
Nov 03 20:12:46.938272 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:12:46.937864    5570 scope.go:122] "RemoveContainer" containerID="82ad9104018c34362662e8e414ae4c5e719f9f031508b90be4506957ff4a52d1"
Nov 03 20:12:46.939592 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:12:46.939558    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d9f5db59-vggm4" event={"ID":"ca3c6734-bb2a-487b-9c6d-c2836a8eea39","Type":"ContainerStarted","Data":"cca0c77fdc4cb76a64e6919ba4c6f79723ee7df9c075424e4f07fa6b4ced2d51"}
Nov 03 20:12:46.940915 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:12:46.940889    5570 generic.go:356] "Generic (PLEG): container finished" podID="227ed64b-cb2f-40ec-9893-7535c002b66c" containerID="f40ac8735f87a91337558d92a66a6b714cf63d9336278a2841521523434ea8d8" exitCode=0
Nov 03 20:12:46.941020 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:12:46.940932    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-hk5gj"
Nov 03 20:12:46.941020 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:12:46.940953    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-hk5gj" event={"ID":"227ed64b-cb2f-40ec-9893-7535c002b66c","Type":"ContainerDied","Data":"f40ac8735f87a91337558d92a66a6b714cf63d9336278a2841521523434ea8d8"}
Nov 03 20:12:46.941020 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:12:46.940986    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-hk5gj" event={"ID":"227ed64b-cb2f-40ec-9893-7535c002b66c","Type":"ContainerDied","Data":"2a63c6f0742a858fdffc2ab96da126cd685001329eab0b86f757fdbbc0142f49"}
Nov 03 20:12:46.946338 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:12:46.946316    5570 scope.go:122] "RemoveContainer" containerID="82ad9104018c34362662e8e414ae4c5e719f9f031508b90be4506957ff4a52d1"
Nov 03 20:12:46.946621 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:12:46.946601    5570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82ad9104018c34362662e8e414ae4c5e719f9f031508b90be4506957ff4a52d1\": container with ID starting with 82ad9104018c34362662e8e414ae4c5e719f9f031508b90be4506957ff4a52d1 not found: ID does not exist" containerID="82ad9104018c34362662e8e414ae4c5e719f9f031508b90be4506957ff4a52d1"
Nov 03 20:12:46.946706 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:12:46.946629    5570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82ad9104018c34362662e8e414ae4c5e719f9f031508b90be4506957ff4a52d1"} err="failed to get container status \"82ad9104018c34362662e8e414ae4c5e719f9f031508b90be4506957ff4a52d1\": rpc error: code = NotFound desc = could not find container \"82ad9104018c34362662e8e414ae4c5e719f9f031508b90be4506957ff4a52d1\": container with ID starting with 82ad9104018c34362662e8e414ae4c5e719f9f031508b90be4506957ff4a52d1 not found: ID does not exist"
Nov 03 20:12:46.946706 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:12:46.946647    5570 scope.go:122] "RemoveContainer" containerID="f40ac8735f87a91337558d92a66a6b714cf63d9336278a2841521523434ea8d8"
Nov 03 20:12:46.955157 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:12:46.955141    5570 scope.go:122] "RemoveContainer" containerID="f40ac8735f87a91337558d92a66a6b714cf63d9336278a2841521523434ea8d8"
Nov 03 20:12:46.955423 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:12:46.955396    5570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f40ac8735f87a91337558d92a66a6b714cf63d9336278a2841521523434ea8d8\": container with ID starting with f40ac8735f87a91337558d92a66a6b714cf63d9336278a2841521523434ea8d8 not found: ID does not exist" containerID="f40ac8735f87a91337558d92a66a6b714cf63d9336278a2841521523434ea8d8"
Nov 03 20:12:46.955515 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:12:46.955431    5570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f40ac8735f87a91337558d92a66a6b714cf63d9336278a2841521523434ea8d8"} err="failed to get container status \"f40ac8735f87a91337558d92a66a6b714cf63d9336278a2841521523434ea8d8\": rpc error: code = NotFound desc = could not find container \"f40ac8735f87a91337558d92a66a6b714cf63d9336278a2841521523434ea8d8\": container with ID starting with f40ac8735f87a91337558d92a66a6b714cf63d9336278a2841521523434ea8d8 not found: ID does not exist"
Nov 03 20:12:46.985287 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:12:46.985258    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-hk5gj"]
Nov 03 20:12:46.988538 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:12:46.988515    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-hk5gj"]
Nov 03 20:12:47.020944 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:12:47.020913    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7c96554b5c-jfjnh"]
Nov 03 20:12:47.024531 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:12:47.024503    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7c96554b5c-jfjnh"]
Nov 03 20:12:47.060793 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:12:47.060721    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-75d9f5db59-vggm4" podStartSLOduration=2.575638562 podStartE2EDuration="3.060707352s" totalImagesPullingTime="485.06879ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:12:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:12:47.05982905 +0000 UTC m=+501.212490936" watchObservedRunningTime="2026-11-03 20:12:47.060707352 +0000 UTC m=+501.213369232"
Nov 03 20:12:47.090066 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:12:47.089868    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-n6vk4"]
Nov 03 20:12:47.090299 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:12:47.090136    5570 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-n6vk4" podUID="93d19d0f-57fa-4116-a049-497b592f86f6" containerName="authorino" containerID="cri-o://baa75473872efb652e99347b32fab68e345e1eceeba108cba8344ff7717ff353" gracePeriod=30
Nov 03 20:12:47.330229 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:12:47.330206    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-n6vk4"
Nov 03 20:12:47.425963 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:12:47.425927    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93d19d0f-57fa-4116-a049-497b592f86f6-kube-api-access-7bczq\" (UniqueName: \"kubernetes.io/projected/93d19d0f-57fa-4116-a049-497b592f86f6-kube-api-access-7bczq\") pod \"93d19d0f-57fa-4116-a049-497b592f86f6\" (UID: \"93d19d0f-57fa-4116-a049-497b592f86f6\") "
Nov 03 20:12:47.427999 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:12:47.427965    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93d19d0f-57fa-4116-a049-497b592f86f6-kube-api-access-7bczq" pod "93d19d0f-57fa-4116-a049-497b592f86f6" (UID: "93d19d0f-57fa-4116-a049-497b592f86f6"). InnerVolumeSpecName "kube-api-access-7bczq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:12:47.527427 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:12:47.527334    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7bczq\" (UniqueName: \"kubernetes.io/projected/93d19d0f-57fa-4116-a049-497b592f86f6-kube-api-access-7bczq\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:12:47.947529 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:12:47.947494    5570 generic.go:356] "Generic (PLEG): container finished" podID="93d19d0f-57fa-4116-a049-497b592f86f6" containerID="baa75473872efb652e99347b32fab68e345e1eceeba108cba8344ff7717ff353" exitCode=0
Nov 03 20:12:47.948031 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:12:47.947541    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-n6vk4"
Nov 03 20:12:47.948031 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:12:47.947585    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-n6vk4" event={"ID":"93d19d0f-57fa-4116-a049-497b592f86f6","Type":"ContainerDied","Data":"baa75473872efb652e99347b32fab68e345e1eceeba108cba8344ff7717ff353"}
Nov 03 20:12:47.948031 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:12:47.947635    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-n6vk4" event={"ID":"93d19d0f-57fa-4116-a049-497b592f86f6","Type":"ContainerDied","Data":"c97feffd39fef87a85383271e6f24d2fb26b880c1a86ec1462e2325a96b61676"}
Nov 03 20:12:47.948031 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:12:47.947660    5570 scope.go:122] "RemoveContainer" containerID="baa75473872efb652e99347b32fab68e345e1eceeba108cba8344ff7717ff353"
Nov 03 20:12:47.957937 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:12:47.957918    5570 scope.go:122] "RemoveContainer" containerID="baa75473872efb652e99347b32fab68e345e1eceeba108cba8344ff7717ff353"
Nov 03 20:12:47.958407 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:12:47.958385    5570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"baa75473872efb652e99347b32fab68e345e1eceeba108cba8344ff7717ff353\": container with ID starting with baa75473872efb652e99347b32fab68e345e1eceeba108cba8344ff7717ff353 not found: ID does not exist" containerID="baa75473872efb652e99347b32fab68e345e1eceeba108cba8344ff7717ff353"
Nov 03 20:12:47.958476 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:12:47.958421    5570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"baa75473872efb652e99347b32fab68e345e1eceeba108cba8344ff7717ff353"} err="failed to get container status \"baa75473872efb652e99347b32fab68e345e1eceeba108cba8344ff7717ff353\": rpc error: code = NotFound desc = could not find container \"baa75473872efb652e99347b32fab68e345e1eceeba108cba8344ff7717ff353\": container with ID starting with baa75473872efb652e99347b32fab68e345e1eceeba108cba8344ff7717ff353 not found: ID does not exist"
Nov 03 20:12:47.962409 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:12:47.962386    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="227ed64b-cb2f-40ec-9893-7535c002b66c" path="/var/lib/kubelet/pods/227ed64b-cb2f-40ec-9893-7535c002b66c/volumes"
Nov 03 20:12:47.962703 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:12:47.962691    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4665dd4d-da21-4946-9d74-c57654293ac7" path="/var/lib/kubelet/pods/4665dd4d-da21-4946-9d74-c57654293ac7/volumes"
Nov 03 20:12:47.995459 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:12:47.995431    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-n6vk4"]
Nov 03 20:12:48.007420 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:12:48.007394    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-n6vk4"]
Nov 03 20:12:49.966951 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:12:49.966915    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93d19d0f-57fa-4116-a049-497b592f86f6" path="/var/lib/kubelet/pods/93d19d0f-57fa-4116-a049-497b592f86f6/volumes"
Nov 03 20:12:57.226425 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:12:57.226387    5570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8bwhz" containerName="extractor"
Nov 03 20:13:27.226942 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:13:27.226824    5570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8bwhz" containerName="extractor"
Nov 03 20:13:57.226036 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:13:57.225931    5570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8bwhz" containerName="extractor"
Nov 03 20:14:01.235275 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:01.235235    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g69f8"]
Nov 03 20:14:02.037379 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:02.037340    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-6nh8m"]
Nov 03 20:14:02.037943 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:02.037916    5570 cpu_manager.go:414] "containerMap: removing container" podUID="93d19d0f-57fa-4116-a049-497b592f86f6" containerName="authorino"
Nov 03 20:14:02.038105 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:02.037997    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93d19d0f-57fa-4116-a049-497b592f86f6" containerName="authorino"
Nov 03 20:14:02.038105 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:02.038015    5570 cpu_manager.go:414] "containerMap: removing container" podUID="4665dd4d-da21-4946-9d74-c57654293ac7" containerName="authorino"
Nov 03 20:14:02.038105 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:02.038024    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4665dd4d-da21-4946-9d74-c57654293ac7" containerName="authorino"
Nov 03 20:14:02.038105 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:02.038044    5570 cpu_manager.go:414] "containerMap: removing container" podUID="227ed64b-cb2f-40ec-9893-7535c002b66c" containerName="authorino"
Nov 03 20:14:02.038105 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:02.038051    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="227ed64b-cb2f-40ec-9893-7535c002b66c" containerName="authorino"
Nov 03 20:14:02.038366 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:02.038166    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="93d19d0f-57fa-4116-a049-497b592f86f6" containerName="authorino"
Nov 03 20:14:02.038366 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:02.038180    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="4665dd4d-da21-4946-9d74-c57654293ac7" containerName="authorino"
Nov 03 20:14:02.038366 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:02.038190    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="227ed64b-cb2f-40ec-9893-7535c002b66c" containerName="authorino"
Nov 03 20:14:02.041706 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:02.041679    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-6nh8m"
Nov 03 20:14:02.056458 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:02.056423    5570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Nov 03 20:14:02.068405 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:02.068376    5570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Nov 03 20:14:02.068569 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:02.068376    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-kschj\""
Nov 03 20:14:02.068569 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:02.068473    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Nov 03 20:14:02.069530 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:02.069511    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-6nh8m"]
Nov 03 20:14:02.156292 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:02.156254    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b31639d5-0583-4706-b580-f5752fa502a4-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-6nh8m\" (UID: \"b31639d5-0583-4706-b580-f5752fa502a4\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-6nh8m"
Nov 03 20:14:02.156292 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:02.156297    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b31639d5-0583-4706-b580-f5752fa502a4-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-6nh8m\" (UID: \"b31639d5-0583-4706-b580-f5752fa502a4\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-6nh8m"
Nov 03 20:14:02.156548 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:02.156454    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b31639d5-0583-4706-b580-f5752fa502a4-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-6nh8m\" (UID: \"b31639d5-0583-4706-b580-f5752fa502a4\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-6nh8m"
Nov 03 20:14:02.156548 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:02.156504    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b31639d5-0583-4706-b580-f5752fa502a4-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-6nh8m\" (UID: \"b31639d5-0583-4706-b580-f5752fa502a4\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-6nh8m"
Nov 03 20:14:02.156548 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:02.156527    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9pfk\" (UniqueName: \"kubernetes.io/projected/b31639d5-0583-4706-b580-f5752fa502a4-kube-api-access-m9pfk\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-6nh8m\" (UID: \"b31639d5-0583-4706-b580-f5752fa502a4\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-6nh8m"
Nov 03 20:14:02.156674 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:02.156588    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b31639d5-0583-4706-b580-f5752fa502a4-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-6nh8m\" (UID: \"b31639d5-0583-4706-b580-f5752fa502a4\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-6nh8m"
Nov 03 20:14:02.257364 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:02.257316    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b31639d5-0583-4706-b580-f5752fa502a4-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-6nh8m\" (UID: \"b31639d5-0583-4706-b580-f5752fa502a4\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-6nh8m"
Nov 03 20:14:02.257364 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:02.257364    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9pfk\" (UniqueName: \"kubernetes.io/projected/b31639d5-0583-4706-b580-f5752fa502a4-kube-api-access-m9pfk\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-6nh8m\" (UID: \"b31639d5-0583-4706-b580-f5752fa502a4\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-6nh8m"
Nov 03 20:14:02.257939 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:02.257388    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b31639d5-0583-4706-b580-f5752fa502a4-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-6nh8m\" (UID: \"b31639d5-0583-4706-b580-f5752fa502a4\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-6nh8m"
Nov 03 20:14:02.257939 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:02.257428    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b31639d5-0583-4706-b580-f5752fa502a4-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-6nh8m\" (UID: \"b31639d5-0583-4706-b580-f5752fa502a4\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-6nh8m"
Nov 03 20:14:02.257939 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:02.257451    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b31639d5-0583-4706-b580-f5752fa502a4-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-6nh8m\" (UID: \"b31639d5-0583-4706-b580-f5752fa502a4\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-6nh8m"
Nov 03 20:14:02.257939 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:02.257612    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b31639d5-0583-4706-b580-f5752fa502a4-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-6nh8m\" (UID: \"b31639d5-0583-4706-b580-f5752fa502a4\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-6nh8m"
Nov 03 20:14:02.257939 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:02.257846    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b31639d5-0583-4706-b580-f5752fa502a4-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-6nh8m\" (UID: \"b31639d5-0583-4706-b580-f5752fa502a4\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-6nh8m"
Nov 03 20:14:02.257939 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:02.257904    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b31639d5-0583-4706-b580-f5752fa502a4-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-6nh8m\" (UID: \"b31639d5-0583-4706-b580-f5752fa502a4\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-6nh8m"
Nov 03 20:14:02.257939 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:02.257924    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b31639d5-0583-4706-b580-f5752fa502a4-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-6nh8m\" (UID: \"b31639d5-0583-4706-b580-f5752fa502a4\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-6nh8m"
Nov 03 20:14:02.259833 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:02.259809    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b31639d5-0583-4706-b580-f5752fa502a4-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-6nh8m\" (UID: \"b31639d5-0583-4706-b580-f5752fa502a4\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-6nh8m"
Nov 03 20:14:02.259976 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:02.259960    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b31639d5-0583-4706-b580-f5752fa502a4-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-6nh8m\" (UID: \"b31639d5-0583-4706-b580-f5752fa502a4\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-6nh8m"
Nov 03 20:14:02.275688 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:02.275660    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9pfk\" (UniqueName: \"kubernetes.io/projected/b31639d5-0583-4706-b580-f5752fa502a4-kube-api-access-m9pfk\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-6nh8m\" (UID: \"b31639d5-0583-4706-b580-f5752fa502a4\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-6nh8m"
Nov 03 20:14:02.358710 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:02.358621    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-6nh8m"
Nov 03 20:14:02.652098 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:02.652010    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-6nh8m"]
Nov 03 20:14:02.656557 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:14:02.656504    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb31639d5_0583_4706_b580_f5752fa502a4.slice/crio-bd696df85cede7d82118ca7afa413926564aece93e4c4ea1d9731ea5ec4512a4 WatchSource:0}: Error finding container bd696df85cede7d82118ca7afa413926564aece93e4c4ea1d9731ea5ec4512a4: Status 404 returned error can't find the container with id bd696df85cede7d82118ca7afa413926564aece93e4c4ea1d9731ea5ec4512a4
Nov 03 20:14:03.265634 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:03.265592    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-6nh8m" event={"ID":"b31639d5-0583-4706-b580-f5752fa502a4","Type":"ContainerStarted","Data":"bd696df85cede7d82118ca7afa413926564aece93e4c4ea1d9731ea5ec4512a4"}
Nov 03 20:14:03.302618 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:03.302580    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g69f8"]
Nov 03 20:14:04.116599 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:04.116551    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz"]
Nov 03 20:14:04.120601 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:04.120576    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz"
Nov 03 20:14:04.139090 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:04.139062    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz"]
Nov 03 20:14:04.149342 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:04.149315    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Nov 03 20:14:04.175627 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:04.175593    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a715ef1-38e9-4386-9eec-e10e8df11a10-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz\" (UID: \"3a715ef1-38e9-4386-9eec-e10e8df11a10\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz"
Nov 03 20:14:04.175627 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:04.175628    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a715ef1-38e9-4386-9eec-e10e8df11a10-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz\" (UID: \"3a715ef1-38e9-4386-9eec-e10e8df11a10\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz"
Nov 03 20:14:04.175926 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:04.175672    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a715ef1-38e9-4386-9eec-e10e8df11a10-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz\" (UID: \"3a715ef1-38e9-4386-9eec-e10e8df11a10\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz"
Nov 03 20:14:04.175926 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:04.175737    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a715ef1-38e9-4386-9eec-e10e8df11a10-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz\" (UID: \"3a715ef1-38e9-4386-9eec-e10e8df11a10\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz"
Nov 03 20:14:04.175926 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:04.175856    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zp5xb\" (UniqueName: \"kubernetes.io/projected/3a715ef1-38e9-4386-9eec-e10e8df11a10-kube-api-access-zp5xb\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz\" (UID: \"3a715ef1-38e9-4386-9eec-e10e8df11a10\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz"
Nov 03 20:14:04.175926 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:04.175896    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a715ef1-38e9-4386-9eec-e10e8df11a10-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz\" (UID: \"3a715ef1-38e9-4386-9eec-e10e8df11a10\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz"
Nov 03 20:14:04.271466 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:04.271379    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-6nh8m" event={"ID":"b31639d5-0583-4706-b580-f5752fa502a4","Type":"ContainerStarted","Data":"1c7fe434db45fdcc4b826079f910d69ad9f0fdd202017fd2ff86f035c4e3911c"}
Nov 03 20:14:04.277050 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:04.277009    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zp5xb\" (UniqueName: \"kubernetes.io/projected/3a715ef1-38e9-4386-9eec-e10e8df11a10-kube-api-access-zp5xb\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz\" (UID: \"3a715ef1-38e9-4386-9eec-e10e8df11a10\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz"
Nov 03 20:14:04.277175 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:04.277074    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a715ef1-38e9-4386-9eec-e10e8df11a10-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz\" (UID: \"3a715ef1-38e9-4386-9eec-e10e8df11a10\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz"
Nov 03 20:14:04.277175 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:04.277142    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a715ef1-38e9-4386-9eec-e10e8df11a10-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz\" (UID: \"3a715ef1-38e9-4386-9eec-e10e8df11a10\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz"
Nov 03 20:14:04.277285 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:04.277241    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a715ef1-38e9-4386-9eec-e10e8df11a10-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz\" (UID: \"3a715ef1-38e9-4386-9eec-e10e8df11a10\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz"
Nov 03 20:14:04.277347 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:04.277308    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a715ef1-38e9-4386-9eec-e10e8df11a10-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz\" (UID: \"3a715ef1-38e9-4386-9eec-e10e8df11a10\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz"
Nov 03 20:14:04.277347 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:04.277338    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a715ef1-38e9-4386-9eec-e10e8df11a10-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz\" (UID: \"3a715ef1-38e9-4386-9eec-e10e8df11a10\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz"
Nov 03 20:14:04.277655 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:04.277622    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a715ef1-38e9-4386-9eec-e10e8df11a10-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz\" (UID: \"3a715ef1-38e9-4386-9eec-e10e8df11a10\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz"
Nov 03 20:14:04.277769 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:04.277626    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a715ef1-38e9-4386-9eec-e10e8df11a10-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz\" (UID: \"3a715ef1-38e9-4386-9eec-e10e8df11a10\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz"
Nov 03 20:14:04.277769 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:04.277710    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a715ef1-38e9-4386-9eec-e10e8df11a10-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz\" (UID: \"3a715ef1-38e9-4386-9eec-e10e8df11a10\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz"
Nov 03 20:14:04.279367 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:04.279348    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a715ef1-38e9-4386-9eec-e10e8df11a10-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz\" (UID: \"3a715ef1-38e9-4386-9eec-e10e8df11a10\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz"
Nov 03 20:14:04.279855 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:04.279833    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a715ef1-38e9-4386-9eec-e10e8df11a10-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz\" (UID: \"3a715ef1-38e9-4386-9eec-e10e8df11a10\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz"
Nov 03 20:14:04.297307 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:04.297276    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zp5xb\" (UniqueName: \"kubernetes.io/projected/3a715ef1-38e9-4386-9eec-e10e8df11a10-kube-api-access-zp5xb\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz\" (UID: \"3a715ef1-38e9-4386-9eec-e10e8df11a10\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz"
Nov 03 20:14:04.322395 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:04.322340    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-6nh8m" podStartSLOduration=1.0377754750000001 podStartE2EDuration="2.322326463s" totalImagesPullingTime="1.284550988s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:14:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:14:04.321614922 +0000 UTC m=+578.474276808" watchObservedRunningTime="2026-11-03 20:14:04.322326463 +0000 UTC m=+578.474988349"
Nov 03 20:14:04.359756 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:04.359729    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-6nh8m"
Nov 03 20:14:04.434804 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:04.434674    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz"
Nov 03 20:14:04.738007 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:04.737968    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz"]
Nov 03 20:14:04.743085 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:14:04.743049    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3a715ef1_38e9_4386_9eec_e10e8df11a10.slice/crio-60f1a42ceeee6a494cccbde3c81bdc3a786ac2ac2b6b72d4a79822b6dbf23048 WatchSource:0}: Error finding container 60f1a42ceeee6a494cccbde3c81bdc3a786ac2ac2b6b72d4a79822b6dbf23048: Status 404 returned error can't find the container with id 60f1a42ceeee6a494cccbde3c81bdc3a786ac2ac2b6b72d4a79822b6dbf23048
Nov 03 20:14:05.283590 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:05.283491    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz" event={"ID":"3a715ef1-38e9-4386-9eec-e10e8df11a10","Type":"ContainerStarted","Data":"022fa12d0591ab3ed31deb0639cfdfa87ffba811c8c1e5747443c9a48d2025ca"}
Nov 03 20:14:05.283590 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:05.283548    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz" event={"ID":"3a715ef1-38e9-4386-9eec-e10e8df11a10","Type":"ContainerStarted","Data":"60f1a42ceeee6a494cccbde3c81bdc3a786ac2ac2b6b72d4a79822b6dbf23048"}
Nov 03 20:14:05.323509 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:05.323451    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz" podStartSLOduration=1.050542249 podStartE2EDuration="1.323438706s" totalImagesPullingTime="272.896457ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:14:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:14:05.322112897 +0000 UTC m=+579.474774783" watchObservedRunningTime="2026-11-03 20:14:05.323438706 +0000 UTC m=+579.476100593"
Nov 03 20:14:05.435606 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:05.435570    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz"
Nov 03 20:14:07.344381 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:07.344338    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g69f8"]
Nov 03 20:14:08.895270 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:08.895236    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-cpfzl"]
Nov 03 20:14:08.899338 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:08.899305    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-cpfzl"
Nov 03 20:14:08.922221 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:08.922185    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-cpfzl"]
Nov 03 20:14:08.928020 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:08.927998    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Nov 03 20:14:08.928219 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:08.928198    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30b21f09-fd9e-4836-9fc3-c19f818c9022-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-cpfzl\" (UID: \"30b21f09-fd9e-4836-9fc3-c19f818c9022\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-cpfzl"
Nov 03 20:14:08.928277 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:08.928238    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30b21f09-fd9e-4836-9fc3-c19f818c9022-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-cpfzl\" (UID: \"30b21f09-fd9e-4836-9fc3-c19f818c9022\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-cpfzl"
Nov 03 20:14:08.928277 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:08.928259    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30b21f09-fd9e-4836-9fc3-c19f818c9022-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-cpfzl\" (UID: \"30b21f09-fd9e-4836-9fc3-c19f818c9022\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-cpfzl"
Nov 03 20:14:08.928419 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:08.928399    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4qc5\" (UniqueName: \"kubernetes.io/projected/30b21f09-fd9e-4836-9fc3-c19f818c9022-kube-api-access-x4qc5\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-cpfzl\" (UID: \"30b21f09-fd9e-4836-9fc3-c19f818c9022\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-cpfzl"
Nov 03 20:14:08.928485 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:08.928463    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30b21f09-fd9e-4836-9fc3-c19f818c9022-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-cpfzl\" (UID: \"30b21f09-fd9e-4836-9fc3-c19f818c9022\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-cpfzl"
Nov 03 20:14:08.928588 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:08.928567    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30b21f09-fd9e-4836-9fc3-c19f818c9022-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-cpfzl\" (UID: \"30b21f09-fd9e-4836-9fc3-c19f818c9022\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-cpfzl"
Nov 03 20:14:09.029373 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:09.029333    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30b21f09-fd9e-4836-9fc3-c19f818c9022-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-cpfzl\" (UID: \"30b21f09-fd9e-4836-9fc3-c19f818c9022\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-cpfzl"
Nov 03 20:14:09.029373 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:09.029375    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30b21f09-fd9e-4836-9fc3-c19f818c9022-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-cpfzl\" (UID: \"30b21f09-fd9e-4836-9fc3-c19f818c9022\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-cpfzl"
Nov 03 20:14:09.029647 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:09.029402    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4qc5\" (UniqueName: \"kubernetes.io/projected/30b21f09-fd9e-4836-9fc3-c19f818c9022-kube-api-access-x4qc5\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-cpfzl\" (UID: \"30b21f09-fd9e-4836-9fc3-c19f818c9022\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-cpfzl"
Nov 03 20:14:09.029647 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:09.029434    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30b21f09-fd9e-4836-9fc3-c19f818c9022-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-cpfzl\" (UID: \"30b21f09-fd9e-4836-9fc3-c19f818c9022\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-cpfzl"
Nov 03 20:14:09.029647 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:09.029459    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30b21f09-fd9e-4836-9fc3-c19f818c9022-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-cpfzl\" (UID: \"30b21f09-fd9e-4836-9fc3-c19f818c9022\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-cpfzl"
Nov 03 20:14:09.029647 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:09.029492    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30b21f09-fd9e-4836-9fc3-c19f818c9022-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-cpfzl\" (UID: \"30b21f09-fd9e-4836-9fc3-c19f818c9022\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-cpfzl"
Nov 03 20:14:09.029909 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:09.029858    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30b21f09-fd9e-4836-9fc3-c19f818c9022-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-cpfzl\" (UID: \"30b21f09-fd9e-4836-9fc3-c19f818c9022\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-cpfzl"
Nov 03 20:14:09.029965 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:09.029943    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30b21f09-fd9e-4836-9fc3-c19f818c9022-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-cpfzl\" (UID: \"30b21f09-fd9e-4836-9fc3-c19f818c9022\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-cpfzl"
Nov 03 20:14:09.030000 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:09.029959    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30b21f09-fd9e-4836-9fc3-c19f818c9022-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-cpfzl\" (UID: \"30b21f09-fd9e-4836-9fc3-c19f818c9022\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-cpfzl"
Nov 03 20:14:09.031721 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:09.031700    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30b21f09-fd9e-4836-9fc3-c19f818c9022-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-cpfzl\" (UID: \"30b21f09-fd9e-4836-9fc3-c19f818c9022\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-cpfzl"
Nov 03 20:14:09.031925 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:09.031891    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30b21f09-fd9e-4836-9fc3-c19f818c9022-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-cpfzl\" (UID: \"30b21f09-fd9e-4836-9fc3-c19f818c9022\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-cpfzl"
Nov 03 20:14:09.049145 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:09.049107    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4qc5\" (UniqueName: \"kubernetes.io/projected/30b21f09-fd9e-4836-9fc3-c19f818c9022-kube-api-access-x4qc5\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-cpfzl\" (UID: \"30b21f09-fd9e-4836-9fc3-c19f818c9022\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-cpfzl"
Nov 03 20:14:09.219671 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:09.219632    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-cpfzl"
Nov 03 20:14:09.516436 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:09.516335    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-cpfzl"]
Nov 03 20:14:09.520233 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:14:09.520195    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30b21f09_fd9e_4836_9fc3_c19f818c9022.slice/crio-5df9ec6dcfcb55b1a7b90a6207ea3f0b77dca9c57b5c4ed9e4dfc85dbd3ce818 WatchSource:0}: Error finding container 5df9ec6dcfcb55b1a7b90a6207ea3f0b77dca9c57b5c4ed9e4dfc85dbd3ce818: Status 404 returned error can't find the container with id 5df9ec6dcfcb55b1a7b90a6207ea3f0b77dca9c57b5c4ed9e4dfc85dbd3ce818
Nov 03 20:14:09.735425 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:14:09.735383    5570 log.go:32] "PullImage from image service failed" err="rpc error: code = Unknown desc = unable to pull image: copying system image from manifest list: parsing image configuration: Get \"https://pkg-containers.githubusercontent.com/ghcr1/blobs/sha256:547b446ce89bf3eb80418748bfc4d1a35b41191c0f4241b1585e587893c1a87e?se=2026-11-03T20%3A20%3A00Z&sig=fIf%2Fd%2Bs3gZX9Piy8NkcAj44M1BV0RnzwcfEvVUpehrg%3D&ske=2026-11-04T12%3A04%3A44Z&skoid=fb3d2a07-ec6c-4fe4-aced-9efe0fd2fe1a&sks=b&skt=2026-11-03T12%3A04%3A44Z&sktid=398a6654-997b-47e9-b12b-9515b896b4de&skv=2025-01-05&sp=r&spr=https&sr=b&sv=2025-01-05&hmac=fa0450ceb02e4eafed18f21e16cf572de348ca09b1abcd30b1d4f98592d30bde\": read tcp 10.0.132.4:35134->185.199.110.154:443: read: connection reset by peer" image="ghcr.io/llm-d/llm-d-inference-sim:v0.8.2"
Nov 03 20:14:09.735607 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:14:09.735438    5570 kuberuntime_image.go:43] "Failed to pull image" err="unable to pull image: copying system image from manifest list: parsing image configuration: Get \"https://pkg-containers.githubusercontent.com/ghcr1/blobs/sha256:547b446ce89bf3eb80418748bfc4d1a35b41191c0f4241b1585e587893c1a87e?se=2026-11-03T20%3A20%3A00Z&sig=fIf%2Fd%2Bs3gZX9Piy8NkcAj44M1BV0RnzwcfEvVUpehrg%3D&ske=2026-11-04T12%3A04%3A44Z&skoid=fb3d2a07-ec6c-4fe4-aced-9efe0fd2fe1a&sks=b&skt=2026-11-03T12%3A04%3A44Z&sktid=398a6654-997b-47e9-b12b-9515b896b4de&skv=2025-01-05&sp=r&spr=https&sr=b&sv=2025-01-05&hmac=fa0450ceb02e4eafed18f21e16cf572de348ca09b1abcd30b1d4f98592d30bde\": read tcp 10.0.132.4:35134->185.199.110.154:443: read: connection reset by peer" image="ghcr.io/llm-d/llm-d-inference-sim:v0.8.2"
Nov 03 20:14:09.735663 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:14:09.735616    5570 kuberuntime_manager.go:1707] "Unhandled Error" err="container main start failed in pod e2e-distinct-simulated-kserve-7b998b79d9-cpfzl_llm(30b21f09-fd9e-4836-9fc3-c19f818c9022): ErrImagePull: unable to pull image: copying system image from manifest list: parsing image configuration: Get \"https://pkg-containers.githubusercontent.com/ghcr1/blobs/sha256:547b446ce89bf3eb80418748bfc4d1a35b41191c0f4241b1585e587893c1a87e?se=2026-11-03T20%3A20%3A00Z&sig=fIf%2Fd%2Bs3gZX9Piy8NkcAj44M1BV0RnzwcfEvVUpehrg%3D&ske=2026-11-04T12%3A04%3A44Z&skoid=fb3d2a07-ec6c-4fe4-aced-9efe0fd2fe1a&sks=b&skt=2026-11-03T12%3A04%3A44Z&sktid=398a6654-997b-47e9-b12b-9515b896b4de&skv=2025-01-05&sp=r&spr=https&sr=b&sv=2025-01-05&hmac=fa0450ceb02e4eafed18f21e16cf572de348ca09b1abcd30b1d4f98592d30bde\": read tcp 10.0.132.4:35134->185.199.110.154:443: read: connection reset by peer" logger="UnhandledError"
Nov 03 20:14:09.735721 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:14:09.735661    5570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with ErrImagePull: \"unable to pull image: copying system image from manifest list: parsing image configuration: Get \\\"https://pkg-containers.githubusercontent.com/ghcr1/blobs/sha256:547b446ce89bf3eb80418748bfc4d1a35b41191c0f4241b1585e587893c1a87e?se=2026-11-03T20%3A20%3A00Z&sig=fIf%2Fd%2Bs3gZX9Piy8NkcAj44M1BV0RnzwcfEvVUpehrg%3D&ske=2026-11-04T12%3A04%3A44Z&skoid=fb3d2a07-ec6c-4fe4-aced-9efe0fd2fe1a&sks=b&skt=2026-11-03T12%3A04%3A44Z&sktid=398a6654-997b-47e9-b12b-9515b896b4de&skv=2025-01-05&sp=r&spr=https&sr=b&sv=2025-01-05&hmac=fa0450ceb02e4eafed18f21e16cf572de348ca09b1abcd30b1d4f98592d30bde\\\": read tcp 10.0.132.4:35134->185.199.110.154:443: read: connection reset by peer\"" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-cpfzl" podUID="30b21f09-fd9e-4836-9fc3-c19f818c9022"
Nov 03 20:14:10.311597 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:10.311548    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-cpfzl" event={"ID":"30b21f09-fd9e-4836-9fc3-c19f818c9022","Type":"ContainerStarted","Data":"5df9ec6dcfcb55b1a7b90a6207ea3f0b77dca9c57b5c4ed9e4dfc85dbd3ce818"}
Nov 03 20:14:10.313072 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:14:10.313029    5570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with ImagePullBackOff: \"Back-off pulling image \\\"ghcr.io/llm-d/llm-d-inference-sim:v0.8.2\\\": ErrImagePull: unable to pull image: copying system image from manifest list: parsing image configuration: Get \\\"https://pkg-containers.githubusercontent.com/ghcr1/blobs/sha256:547b446ce89bf3eb80418748bfc4d1a35b41191c0f4241b1585e587893c1a87e?se=2026-11-03T20%3A20%3A00Z&sig=fIf%2Fd%2Bs3gZX9Piy8NkcAj44M1BV0RnzwcfEvVUpehrg%3D&ske=2026-11-04T12%3A04%3A44Z&skoid=fb3d2a07-ec6c-4fe4-aced-9efe0fd2fe1a&sks=b&skt=2026-11-03T12%3A04%3A44Z&sktid=398a6654-997b-47e9-b12b-9515b896b4de&skv=2025-01-05&sp=r&spr=https&sr=b&sv=2025-01-05&hmac=fa0450ceb02e4eafed18f21e16cf572de348ca09b1abcd30b1d4f98592d30bde\\\": read tcp 10.0.132.4:35134->185.199.110.154:443: read: connection reset by peer\"" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-cpfzl" podUID="30b21f09-fd9e-4836-9fc3-c19f818c9022"
Nov 03 20:14:11.317166 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:14:11.317126    5570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with ImagePullBackOff: \"Back-off pulling image \\\"ghcr.io/llm-d/llm-d-inference-sim:v0.8.2\\\": ErrImagePull: unable to pull image: copying system image from manifest list: parsing image configuration: Get \\\"https://pkg-containers.githubusercontent.com/ghcr1/blobs/sha256:547b446ce89bf3eb80418748bfc4d1a35b41191c0f4241b1585e587893c1a87e?se=2026-11-03T20%3A20%3A00Z&sig=fIf%2Fd%2Bs3gZX9Piy8NkcAj44M1BV0RnzwcfEvVUpehrg%3D&ske=2026-11-04T12%3A04%3A44Z&skoid=fb3d2a07-ec6c-4fe4-aced-9efe0fd2fe1a&sks=b&skt=2026-11-03T12%3A04%3A44Z&sktid=398a6654-997b-47e9-b12b-9515b896b4de&skv=2025-01-05&sp=r&spr=https&sr=b&sv=2025-01-05&hmac=fa0450ceb02e4eafed18f21e16cf572de348ca09b1abcd30b1d4f98592d30bde\\\": read tcp 10.0.132.4:35134->185.199.110.154:443: read: connection reset by peer\"" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-cpfzl" podUID="30b21f09-fd9e-4836-9fc3-c19f818c9022"
Nov 03 20:14:12.323220 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:12.323189    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g69f8"]
Nov 03 20:14:12.359967 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:12.359927    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-6nh8m"
Nov 03 20:14:12.377658 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:12.377625    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-6nh8m"
Nov 03 20:14:13.337471 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:13.337443    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-6nh8m"
Nov 03 20:14:14.434845 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:14.434811    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz"
Nov 03 20:14:14.447169 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:14.447132    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz"
Nov 03 20:14:15.019088 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:15.019054    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87s5mf"]
Nov 03 20:14:15.023233 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:15.023204    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87s5mf"]
Nov 03 20:14:15.024182 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:15.023404    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87s5mf"
Nov 03 20:14:15.040076 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:15.040049    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Nov 03 20:14:15.119786 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:15.119733    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7306cb9-eeae-447b-b834-870c8d49a5aa-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87s5mf\" (UID: \"b7306cb9-eeae-447b-b834-870c8d49a5aa\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87s5mf"
Nov 03 20:14:15.119786 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:15.119785    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7306cb9-eeae-447b-b834-870c8d49a5aa-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87s5mf\" (UID: \"b7306cb9-eeae-447b-b834-870c8d49a5aa\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87s5mf"
Nov 03 20:14:15.120074 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:15.119932    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7306cb9-eeae-447b-b834-870c8d49a5aa-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87s5mf\" (UID: \"b7306cb9-eeae-447b-b834-870c8d49a5aa\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87s5mf"
Nov 03 20:14:15.120074 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:15.119989    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7306cb9-eeae-447b-b834-870c8d49a5aa-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87s5mf\" (UID: \"b7306cb9-eeae-447b-b834-870c8d49a5aa\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87s5mf"
Nov 03 20:14:15.120074 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:15.120020    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kbkr\" (UniqueName: \"kubernetes.io/projected/b7306cb9-eeae-447b-b834-870c8d49a5aa-kube-api-access-4kbkr\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87s5mf\" (UID: \"b7306cb9-eeae-447b-b834-870c8d49a5aa\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87s5mf"
Nov 03 20:14:15.120074 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:15.120057    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7306cb9-eeae-447b-b834-870c8d49a5aa-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87s5mf\" (UID: \"b7306cb9-eeae-447b-b834-870c8d49a5aa\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87s5mf"
Nov 03 20:14:15.220638 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:15.220597    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7306cb9-eeae-447b-b834-870c8d49a5aa-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87s5mf\" (UID: \"b7306cb9-eeae-447b-b834-870c8d49a5aa\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87s5mf"
Nov 03 20:14:15.220638 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:15.220637    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7306cb9-eeae-447b-b834-870c8d49a5aa-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87s5mf\" (UID: \"b7306cb9-eeae-447b-b834-870c8d49a5aa\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87s5mf"
Nov 03 20:14:15.220940 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:15.220667    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7306cb9-eeae-447b-b834-870c8d49a5aa-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87s5mf\" (UID: \"b7306cb9-eeae-447b-b834-870c8d49a5aa\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87s5mf"
Nov 03 20:14:15.220940 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:15.220694    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7306cb9-eeae-447b-b834-870c8d49a5aa-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87s5mf\" (UID: \"b7306cb9-eeae-447b-b834-870c8d49a5aa\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87s5mf"
Nov 03 20:14:15.220940 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:15.220715    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kbkr\" (UniqueName: \"kubernetes.io/projected/b7306cb9-eeae-447b-b834-870c8d49a5aa-kube-api-access-4kbkr\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87s5mf\" (UID: \"b7306cb9-eeae-447b-b834-870c8d49a5aa\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87s5mf"
Nov 03 20:14:15.220940 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:15.220761    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7306cb9-eeae-447b-b834-870c8d49a5aa-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87s5mf\" (UID: \"b7306cb9-eeae-447b-b834-870c8d49a5aa\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87s5mf"
Nov 03 20:14:15.221453 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:15.221424    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7306cb9-eeae-447b-b834-870c8d49a5aa-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87s5mf\" (UID: \"b7306cb9-eeae-447b-b834-870c8d49a5aa\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87s5mf"
Nov 03 20:14:15.221564 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:15.221498    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7306cb9-eeae-447b-b834-870c8d49a5aa-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87s5mf\" (UID: \"b7306cb9-eeae-447b-b834-870c8d49a5aa\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87s5mf"
Nov 03 20:14:15.221564 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:15.221515    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7306cb9-eeae-447b-b834-870c8d49a5aa-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87s5mf\" (UID: \"b7306cb9-eeae-447b-b834-870c8d49a5aa\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87s5mf"
Nov 03 20:14:15.224287 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:15.223509    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7306cb9-eeae-447b-b834-870c8d49a5aa-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87s5mf\" (UID: \"b7306cb9-eeae-447b-b834-870c8d49a5aa\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87s5mf"
Nov 03 20:14:15.224287 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:15.223735    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7306cb9-eeae-447b-b834-870c8d49a5aa-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87s5mf\" (UID: \"b7306cb9-eeae-447b-b834-870c8d49a5aa\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87s5mf"
Nov 03 20:14:15.240150 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:15.240119    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kbkr\" (UniqueName: \"kubernetes.io/projected/b7306cb9-eeae-447b-b834-870c8d49a5aa-kube-api-access-4kbkr\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87s5mf\" (UID: \"b7306cb9-eeae-447b-b834-870c8d49a5aa\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87s5mf"
Nov 03 20:14:15.342190 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:15.342101    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87s5mf"
Nov 03 20:14:15.348648 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:15.348621    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz"
Nov 03 20:14:15.637499 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:15.637386    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87s5mf"]
Nov 03 20:14:15.642505 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:14:15.642472    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7306cb9_eeae_447b_b834_870c8d49a5aa.slice/crio-1b990e8950796ab89ac01c3be95fcf99c08641b0b8426c24dfbd38107f5cca66 WatchSource:0}: Error finding container 1b990e8950796ab89ac01c3be95fcf99c08641b0b8426c24dfbd38107f5cca66: Status 404 returned error can't find the container with id 1b990e8950796ab89ac01c3be95fcf99c08641b0b8426c24dfbd38107f5cca66
Nov 03 20:14:16.343167 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:16.343127    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87s5mf" event={"ID":"b7306cb9-eeae-447b-b834-870c8d49a5aa","Type":"ContainerStarted","Data":"33470128faab3989634f230c72413557d9a49c0ddef10757873e6b9e6b72e607"}
Nov 03 20:14:16.343345 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:16.343195    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87s5mf" event={"ID":"b7306cb9-eeae-447b-b834-870c8d49a5aa","Type":"ContainerStarted","Data":"1b990e8950796ab89ac01c3be95fcf99c08641b0b8426c24dfbd38107f5cca66"}
Nov 03 20:14:16.381288 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:16.381237    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87s5mf" podStartSLOduration=2.143908751 podStartE2EDuration="2.38122488s" totalImagesPullingTime="237.316129ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:14:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:14:16.380637761 +0000 UTC m=+590.533299648" watchObservedRunningTime="2026-11-03 20:14:16.38122488 +0000 UTC m=+590.533886819"
Nov 03 20:14:17.168561 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:17.168528    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g69f8"]
Nov 03 20:14:17.342978 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:17.342933    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87s5mf"
Nov 03 20:14:22.371500 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:22.371463    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-cpfzl" event={"ID":"30b21f09-fd9e-4836-9fc3-c19f818c9022","Type":"ContainerStarted","Data":"146146306c1b2e92de6a7d24a835fa58e88e558a99b3e4978861372aced3a648"}
Nov 03 20:14:22.411561 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:22.411509    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-cpfzl" podStartSLOduration=1.755246878 podStartE2EDuration="14.411494762s" totalImagesPullingTime="12.656247884s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:14:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=1 observedRunningTime="2026-11-03 20:14:22.410421506 +0000 UTC m=+596.563083404" watchObservedRunningTime="2026-11-03 20:14:22.411494762 +0000 UTC m=+596.564156647"
Nov 03 20:14:23.220189 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:23.220150    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-cpfzl"
Nov 03 20:14:25.343122 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:25.343085    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87s5mf"
Nov 03 20:14:25.355395 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:25.355364    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87s5mf"
Nov 03 20:14:25.400691 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:25.400659    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87s5mf"
Nov 03 20:14:27.012250 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:27.012212    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-crn7x"]
Nov 03 20:14:27.016889 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:27.016862    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-crn7x"
Nov 03 20:14:27.032158 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:27.032135    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-tzlpr\""
Nov 03 20:14:27.045430 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:27.045399    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-crn7x"]
Nov 03 20:14:27.136358 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:27.136315    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ae62bda-375f-4252-8606-82bae17d680d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-crn7x\" (UID: \"4ae62bda-375f-4252-8606-82bae17d680d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-crn7x"
Nov 03 20:14:27.136358 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:27.136365    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ae62bda-375f-4252-8606-82bae17d680d-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-crn7x\" (UID: \"4ae62bda-375f-4252-8606-82bae17d680d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-crn7x"
Nov 03 20:14:27.136592 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:27.136387    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ae62bda-375f-4252-8606-82bae17d680d-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-crn7x\" (UID: \"4ae62bda-375f-4252-8606-82bae17d680d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-crn7x"
Nov 03 20:14:27.136592 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:27.136561    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ae62bda-375f-4252-8606-82bae17d680d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-crn7x\" (UID: \"4ae62bda-375f-4252-8606-82bae17d680d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-crn7x"
Nov 03 20:14:27.136592 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:27.136584    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2vn4\" (UniqueName: \"kubernetes.io/projected/4ae62bda-375f-4252-8606-82bae17d680d-kube-api-access-s2vn4\") pod \"maas-default-gateway-openshift-default-694db55c87-crn7x\" (UID: \"4ae62bda-375f-4252-8606-82bae17d680d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-crn7x"
Nov 03 20:14:27.136694 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:27.136609    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ae62bda-375f-4252-8606-82bae17d680d-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-crn7x\" (UID: \"4ae62bda-375f-4252-8606-82bae17d680d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-crn7x"
Nov 03 20:14:27.136728 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:27.136690    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ae62bda-375f-4252-8606-82bae17d680d-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-crn7x\" (UID: \"4ae62bda-375f-4252-8606-82bae17d680d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-crn7x"
Nov 03 20:14:27.136728 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:27.136708    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ae62bda-375f-4252-8606-82bae17d680d-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-crn7x\" (UID: \"4ae62bda-375f-4252-8606-82bae17d680d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-crn7x"
Nov 03 20:14:27.136728 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:27.136724    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ae62bda-375f-4252-8606-82bae17d680d-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-crn7x\" (UID: \"4ae62bda-375f-4252-8606-82bae17d680d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-crn7x"
Nov 03 20:14:27.226937 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:14:27.226898    5570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8bwhz" containerName="extractor"
Nov 03 20:14:27.237425 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:27.237399    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ae62bda-375f-4252-8606-82bae17d680d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-crn7x\" (UID: \"4ae62bda-375f-4252-8606-82bae17d680d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-crn7x"
Nov 03 20:14:27.237578 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:27.237437    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ae62bda-375f-4252-8606-82bae17d680d-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-crn7x\" (UID: \"4ae62bda-375f-4252-8606-82bae17d680d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-crn7x"
Nov 03 20:14:27.237578 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:27.237456    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ae62bda-375f-4252-8606-82bae17d680d-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-crn7x\" (UID: \"4ae62bda-375f-4252-8606-82bae17d680d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-crn7x"
Nov 03 20:14:27.237578 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:27.237504    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ae62bda-375f-4252-8606-82bae17d680d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-crn7x\" (UID: \"4ae62bda-375f-4252-8606-82bae17d680d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-crn7x"
Nov 03 20:14:27.237578 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:27.237521    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2vn4\" (UniqueName: \"kubernetes.io/projected/4ae62bda-375f-4252-8606-82bae17d680d-kube-api-access-s2vn4\") pod \"maas-default-gateway-openshift-default-694db55c87-crn7x\" (UID: \"4ae62bda-375f-4252-8606-82bae17d680d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-crn7x"
Nov 03 20:14:27.237578 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:27.237555    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ae62bda-375f-4252-8606-82bae17d680d-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-crn7x\" (UID: \"4ae62bda-375f-4252-8606-82bae17d680d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-crn7x"
Nov 03 20:14:27.237866 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:27.237594    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ae62bda-375f-4252-8606-82bae17d680d-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-crn7x\" (UID: \"4ae62bda-375f-4252-8606-82bae17d680d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-crn7x"
Nov 03 20:14:27.237866 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:27.237618    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ae62bda-375f-4252-8606-82bae17d680d-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-crn7x\" (UID: \"4ae62bda-375f-4252-8606-82bae17d680d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-crn7x"
Nov 03 20:14:27.237866 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:27.237640    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ae62bda-375f-4252-8606-82bae17d680d-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-crn7x\" (UID: \"4ae62bda-375f-4252-8606-82bae17d680d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-crn7x"
Nov 03 20:14:27.238005 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:27.237896    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ae62bda-375f-4252-8606-82bae17d680d-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-crn7x\" (UID: \"4ae62bda-375f-4252-8606-82bae17d680d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-crn7x"
Nov 03 20:14:27.238005 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:27.237964    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ae62bda-375f-4252-8606-82bae17d680d-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-crn7x\" (UID: \"4ae62bda-375f-4252-8606-82bae17d680d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-crn7x"
Nov 03 20:14:27.238147 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:27.238123    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ae62bda-375f-4252-8606-82bae17d680d-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-crn7x\" (UID: \"4ae62bda-375f-4252-8606-82bae17d680d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-crn7x"
Nov 03 20:14:27.238208 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:27.238184    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ae62bda-375f-4252-8606-82bae17d680d-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-crn7x\" (UID: \"4ae62bda-375f-4252-8606-82bae17d680d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-crn7x"
Nov 03 20:14:27.238290 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:27.238272    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ae62bda-375f-4252-8606-82bae17d680d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-crn7x\" (UID: \"4ae62bda-375f-4252-8606-82bae17d680d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-crn7x"
Nov 03 20:14:27.239992 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:27.239968    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ae62bda-375f-4252-8606-82bae17d680d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-crn7x\" (UID: \"4ae62bda-375f-4252-8606-82bae17d680d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-crn7x"
Nov 03 20:14:27.240381 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:27.240364    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ae62bda-375f-4252-8606-82bae17d680d-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-crn7x\" (UID: \"4ae62bda-375f-4252-8606-82bae17d680d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-crn7x"
Nov 03 20:14:27.256463 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:27.256431    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2vn4\" (UniqueName: \"kubernetes.io/projected/4ae62bda-375f-4252-8606-82bae17d680d-kube-api-access-s2vn4\") pod \"maas-default-gateway-openshift-default-694db55c87-crn7x\" (UID: \"4ae62bda-375f-4252-8606-82bae17d680d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-crn7x"
Nov 03 20:14:27.256560 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:27.256492    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ae62bda-375f-4252-8606-82bae17d680d-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-crn7x\" (UID: \"4ae62bda-375f-4252-8606-82bae17d680d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-crn7x"
Nov 03 20:14:27.329400 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:27.329306    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-crn7x"
Nov 03 20:14:27.620591 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:27.620507    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-855865dcb4-dx4qk"]
Nov 03 20:14:27.625062 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:27.625039    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-855865dcb4-dx4qk"
Nov 03 20:14:27.642665 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:27.642632    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-855865dcb4-dx4qk"]
Nov 03 20:14:27.645963 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:27.645940    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-crn7x"]
Nov 03 20:14:27.651711 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:14:27.651681    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ae62bda_375f_4252_8606_82bae17d680d.slice/crio-3a83f201afce537b6663d5a6804d57aaaa7fd4cde8ab154c3f9e0a65ead2725c WatchSource:0}: Error finding container 3a83f201afce537b6663d5a6804d57aaaa7fd4cde8ab154c3f9e0a65ead2725c: Status 404 returned error can't find the container with id 3a83f201afce537b6663d5a6804d57aaaa7fd4cde8ab154c3f9e0a65ead2725c
Nov 03 20:14:27.742114 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:27.742076    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gcjh\" (UniqueName: \"kubernetes.io/projected/240accf1-8b59-4221-98b8-ea9589c6e4ba-kube-api-access-4gcjh\") pod \"authorino-855865dcb4-dx4qk\" (UID: \"240accf1-8b59-4221-98b8-ea9589c6e4ba\") " pod="kuadrant-system/authorino-855865dcb4-dx4qk"
Nov 03 20:14:27.742286 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:27.742124    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/240accf1-8b59-4221-98b8-ea9589c6e4ba-tls-cert\") pod \"authorino-855865dcb4-dx4qk\" (UID: \"240accf1-8b59-4221-98b8-ea9589c6e4ba\") " pod="kuadrant-system/authorino-855865dcb4-dx4qk"
Nov 03 20:14:27.843111 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:27.843071    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gcjh\" (UniqueName: \"kubernetes.io/projected/240accf1-8b59-4221-98b8-ea9589c6e4ba-kube-api-access-4gcjh\") pod \"authorino-855865dcb4-dx4qk\" (UID: \"240accf1-8b59-4221-98b8-ea9589c6e4ba\") " pod="kuadrant-system/authorino-855865dcb4-dx4qk"
Nov 03 20:14:27.843292 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:27.843130    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/240accf1-8b59-4221-98b8-ea9589c6e4ba-tls-cert\") pod \"authorino-855865dcb4-dx4qk\" (UID: \"240accf1-8b59-4221-98b8-ea9589c6e4ba\") " pod="kuadrant-system/authorino-855865dcb4-dx4qk"
Nov 03 20:14:27.845572 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:27.845552    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/240accf1-8b59-4221-98b8-ea9589c6e4ba-tls-cert\") pod \"authorino-855865dcb4-dx4qk\" (UID: \"240accf1-8b59-4221-98b8-ea9589c6e4ba\") " pod="kuadrant-system/authorino-855865dcb4-dx4qk"
Nov 03 20:14:27.861738 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:27.861708    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gcjh\" (UniqueName: \"kubernetes.io/projected/240accf1-8b59-4221-98b8-ea9589c6e4ba-kube-api-access-4gcjh\") pod \"authorino-855865dcb4-dx4qk\" (UID: \"240accf1-8b59-4221-98b8-ea9589c6e4ba\") " pod="kuadrant-system/authorino-855865dcb4-dx4qk"
Nov 03 20:14:27.937106 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:27.937018    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-855865dcb4-dx4qk"
Nov 03 20:14:28.232713 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:28.232675    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-855865dcb4-dx4qk"]
Nov 03 20:14:28.236573 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:14:28.236547    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod240accf1_8b59_4221_98b8_ea9589c6e4ba.slice/crio-c40f2955eb82cb31c9d57d72b4835eed05763504c1954cd5f98130c5a9fde264 WatchSource:0}: Error finding container c40f2955eb82cb31c9d57d72b4835eed05763504c1954cd5f98130c5a9fde264: Status 404 returned error can't find the container with id c40f2955eb82cb31c9d57d72b4835eed05763504c1954cd5f98130c5a9fde264
Nov 03 20:14:28.405482 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:28.405446    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-crn7x" event={"ID":"4ae62bda-375f-4252-8606-82bae17d680d","Type":"ContainerStarted","Data":"3a83f201afce537b6663d5a6804d57aaaa7fd4cde8ab154c3f9e0a65ead2725c"}
Nov 03 20:14:28.406966 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:28.406937    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-855865dcb4-dx4qk" event={"ID":"240accf1-8b59-4221-98b8-ea9589c6e4ba","Type":"ContainerStarted","Data":"c40f2955eb82cb31c9d57d72b4835eed05763504c1954cd5f98130c5a9fde264"}
Nov 03 20:14:29.220467 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:29.220427    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-cpfzl"
Nov 03 20:14:29.238515 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:29.238484    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-cpfzl"
Nov 03 20:14:29.416702 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:29.416649    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-855865dcb4-dx4qk" event={"ID":"240accf1-8b59-4221-98b8-ea9589c6e4ba","Type":"ContainerStarted","Data":"f31d64146dbcf865366165dff111d3734118e9d27255e462ce43d99ac9a075a2"}
Nov 03 20:14:29.434644 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:29.432527    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-cpfzl"
Nov 03 20:14:29.456884 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:29.456828    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-855865dcb4-dx4qk" podStartSLOduration=1.781824826 podStartE2EDuration="2.456811154s" totalImagesPullingTime="674.986328ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:14:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:14:29.454913244 +0000 UTC m=+603.607575130" watchObservedRunningTime="2026-11-03 20:14:29.456811154 +0000 UTC m=+603.609473042"
Nov 03 20:14:29.490468 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:29.489797    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d9f5db59-vggm4"]
Nov 03 20:14:29.490468 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:29.490046    5570 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-75d9f5db59-vggm4" podUID="ca3c6734-bb2a-487b-9c6d-c2836a8eea39" containerName="authorino" containerID="cri-o://cca0c77fdc4cb76a64e6919ba4c6f79723ee7df9c075424e4f07fa6b4ced2d51" gracePeriod=30
Nov 03 20:14:29.815727 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:29.815696    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d9f5db59-vggm4"
Nov 03 20:14:29.863961 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:29.863194    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ca3c6734-bb2a-487b-9c6d-c2836a8eea39-tls-cert\" (UniqueName: \"kubernetes.io/secret/ca3c6734-bb2a-487b-9c6d-c2836a8eea39-tls-cert\") pod \"ca3c6734-bb2a-487b-9c6d-c2836a8eea39\" (UID: \"ca3c6734-bb2a-487b-9c6d-c2836a8eea39\") "
Nov 03 20:14:29.863961 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:29.863339    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca3c6734-bb2a-487b-9c6d-c2836a8eea39-kube-api-access-cqw9c\" (UniqueName: \"kubernetes.io/projected/ca3c6734-bb2a-487b-9c6d-c2836a8eea39-kube-api-access-cqw9c\") pod \"ca3c6734-bb2a-487b-9c6d-c2836a8eea39\" (UID: \"ca3c6734-bb2a-487b-9c6d-c2836a8eea39\") "
Nov 03 20:14:29.866206 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:29.866179    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca3c6734-bb2a-487b-9c6d-c2836a8eea39-kube-api-access-cqw9c" pod "ca3c6734-bb2a-487b-9c6d-c2836a8eea39" (UID: "ca3c6734-bb2a-487b-9c6d-c2836a8eea39"). InnerVolumeSpecName "kube-api-access-cqw9c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:14:29.884434 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:29.884400    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca3c6734-bb2a-487b-9c6d-c2836a8eea39-tls-cert" pod "ca3c6734-bb2a-487b-9c6d-c2836a8eea39" (UID: "ca3c6734-bb2a-487b-9c6d-c2836a8eea39"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:14:29.964249 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:29.964218    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqw9c\" (UniqueName: \"kubernetes.io/projected/ca3c6734-bb2a-487b-9c6d-c2836a8eea39-kube-api-access-cqw9c\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:14:29.964249 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:29.964244    5570 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ca3c6734-bb2a-487b-9c6d-c2836a8eea39-tls-cert\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:14:30.423395 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:30.423353    5570 generic.go:356] "Generic (PLEG): container finished" podID="ca3c6734-bb2a-487b-9c6d-c2836a8eea39" containerID="cca0c77fdc4cb76a64e6919ba4c6f79723ee7df9c075424e4f07fa6b4ced2d51" exitCode=0
Nov 03 20:14:30.423918 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:30.423416    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d9f5db59-vggm4"
Nov 03 20:14:30.423918 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:30.423439    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d9f5db59-vggm4" event={"ID":"ca3c6734-bb2a-487b-9c6d-c2836a8eea39","Type":"ContainerDied","Data":"cca0c77fdc4cb76a64e6919ba4c6f79723ee7df9c075424e4f07fa6b4ced2d51"}
Nov 03 20:14:30.423918 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:30.423487    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d9f5db59-vggm4" event={"ID":"ca3c6734-bb2a-487b-9c6d-c2836a8eea39","Type":"ContainerDied","Data":"6913e6afb4b60a03b7bb8a9dbbb5ee6562504d4eb1e5171006058d6985d92a49"}
Nov 03 20:14:30.423918 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:30.423512    5570 scope.go:122] "RemoveContainer" containerID="cca0c77fdc4cb76a64e6919ba4c6f79723ee7df9c075424e4f07fa6b4ced2d51"
Nov 03 20:14:30.435697 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:30.435673    5570 scope.go:122] "RemoveContainer" containerID="cca0c77fdc4cb76a64e6919ba4c6f79723ee7df9c075424e4f07fa6b4ced2d51"
Nov 03 20:14:30.436128 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:14:30.436102    5570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cca0c77fdc4cb76a64e6919ba4c6f79723ee7df9c075424e4f07fa6b4ced2d51\": container with ID starting with cca0c77fdc4cb76a64e6919ba4c6f79723ee7df9c075424e4f07fa6b4ced2d51 not found: ID does not exist" containerID="cca0c77fdc4cb76a64e6919ba4c6f79723ee7df9c075424e4f07fa6b4ced2d51"
Nov 03 20:14:30.436247 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:30.436143    5570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cca0c77fdc4cb76a64e6919ba4c6f79723ee7df9c075424e4f07fa6b4ced2d51"} err="failed to get container status \"cca0c77fdc4cb76a64e6919ba4c6f79723ee7df9c075424e4f07fa6b4ced2d51\": rpc error: code = NotFound desc = could not find container \"cca0c77fdc4cb76a64e6919ba4c6f79723ee7df9c075424e4f07fa6b4ced2d51\": container with ID starting with cca0c77fdc4cb76a64e6919ba4c6f79723ee7df9c075424e4f07fa6b4ced2d51 not found: ID does not exist"
Nov 03 20:14:30.476215 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:30.476149    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d9f5db59-vggm4"]
Nov 03 20:14:30.479458 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:30.479429    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d9f5db59-vggm4"]
Nov 03 20:14:30.751609 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:30.751347    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:14:30.751609 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:30.751432    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:14:30.751609 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:30.751478    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:14:31.429894 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:31.429852    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-crn7x" event={"ID":"4ae62bda-375f-4252-8606-82bae17d680d","Type":"ContainerStarted","Data":"9b29a8fc84e904bbb506a3ae0e2651fd977a35f533be777b2ac4126aad45d515"}
Nov 03 20:14:31.485257 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:31.485203    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-crn7x" podStartSLOduration=2.388901484 podStartE2EDuration="5.485189126s" totalImagesPullingTime="3.096287642s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:14:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:14:31.484091566 +0000 UTC m=+605.636753454" watchObservedRunningTime="2026-11-03 20:14:31.485189126 +0000 UTC m=+605.637851010"
Nov 03 20:14:31.968410 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:31.968368    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca3c6734-bb2a-487b-9c6d-c2836a8eea39" path="/var/lib/kubelet/pods/ca3c6734-bb2a-487b-9c6d-c2836a8eea39/volumes"
Nov 03 20:14:32.330424 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:32.330334    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-crn7x"
Nov 03 20:14:32.331672 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:32.331639    5570 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-crn7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Nov 03 20:14:32.331829 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:32.331699    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-crn7x" podUID="4ae62bda-375f-4252-8606-82bae17d680d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Nov 03 20:14:33.329877 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:33.329830    5570 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-crn7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Nov 03 20:14:33.330347 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:33.329904    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-crn7x" podUID="4ae62bda-375f-4252-8606-82bae17d680d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Nov 03 20:14:34.329719 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:34.329677    5570 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-crn7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Nov 03 20:14:34.329966 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:34.329753    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-crn7x" podUID="4ae62bda-375f-4252-8606-82bae17d680d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Nov 03 20:14:35.330354 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:35.330316    5570 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-crn7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Nov 03 20:14:35.330796 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:35.330388    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-crn7x" podUID="4ae62bda-375f-4252-8606-82bae17d680d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Nov 03 20:14:36.330067 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:36.330020    5570 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-crn7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Nov 03 20:14:36.330278 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:36.330097    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-crn7x" podUID="4ae62bda-375f-4252-8606-82bae17d680d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Nov 03 20:14:37.330488 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:37.330442    5570 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-crn7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Nov 03 20:14:37.330879 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:37.330517    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-crn7x" podUID="4ae62bda-375f-4252-8606-82bae17d680d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Nov 03 20:14:39.330439 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:39.330391    5570 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-crn7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:14:39.330949 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:39.330465    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-crn7x" podUID="4ae62bda-375f-4252-8606-82bae17d680d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:14:40.371963 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:40.371898    5570 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-crn7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:14:40.372538 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:40.371997    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-crn7x" podUID="4ae62bda-375f-4252-8606-82bae17d680d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:14:41.076603 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:41.076566    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-crn7x"
Nov 03 20:14:41.077033 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:41.077005    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-crn7x"
Nov 03 20:14:41.077818 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:41.077803    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-crn7x"
Nov 03 20:14:42.057516 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:42.057484    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-2x92x"]
Nov 03 20:14:42.058470 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:42.058103    5570 cpu_manager.go:414] "containerMap: removing container" podUID="ca3c6734-bb2a-487b-9c6d-c2836a8eea39" containerName="authorino"
Nov 03 20:14:42.058470 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:42.058123    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca3c6734-bb2a-487b-9c6d-c2836a8eea39" containerName="authorino"
Nov 03 20:14:42.058470 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:42.058250    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca3c6734-bb2a-487b-9c6d-c2836a8eea39" containerName="authorino"
Nov 03 20:14:42.064108 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:42.064087    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-2x92x"
Nov 03 20:14:42.094352 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:42.094322    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-2x92x"]
Nov 03 20:14:42.235871 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:42.235832    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ac2fc9c-0a3e-4326-a4f8-8856b1a30e36-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-2x92x\" (UID: \"2ac2fc9c-0a3e-4326-a4f8-8856b1a30e36\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-2x92x"
Nov 03 20:14:42.236072 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:42.235885    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ac2fc9c-0a3e-4326-a4f8-8856b1a30e36-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-2x92x\" (UID: \"2ac2fc9c-0a3e-4326-a4f8-8856b1a30e36\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-2x92x"
Nov 03 20:14:42.236072 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:42.235995    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xp4s9\" (UniqueName: \"kubernetes.io/projected/2ac2fc9c-0a3e-4326-a4f8-8856b1a30e36-kube-api-access-xp4s9\") pod \"maas-default-gateway-openshift-default-694db55c87-2x92x\" (UID: \"2ac2fc9c-0a3e-4326-a4f8-8856b1a30e36\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-2x92x"
Nov 03 20:14:42.236072 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:42.236056    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ac2fc9c-0a3e-4326-a4f8-8856b1a30e36-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-2x92x\" (UID: \"2ac2fc9c-0a3e-4326-a4f8-8856b1a30e36\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-2x92x"
Nov 03 20:14:42.236175 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:42.236074    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ac2fc9c-0a3e-4326-a4f8-8856b1a30e36-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-2x92x\" (UID: \"2ac2fc9c-0a3e-4326-a4f8-8856b1a30e36\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-2x92x"
Nov 03 20:14:42.236175 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:42.236162    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ac2fc9c-0a3e-4326-a4f8-8856b1a30e36-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-2x92x\" (UID: \"2ac2fc9c-0a3e-4326-a4f8-8856b1a30e36\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-2x92x"
Nov 03 20:14:42.236240 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:42.236190    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ac2fc9c-0a3e-4326-a4f8-8856b1a30e36-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-2x92x\" (UID: \"2ac2fc9c-0a3e-4326-a4f8-8856b1a30e36\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-2x92x"
Nov 03 20:14:42.236240 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:42.236206    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ac2fc9c-0a3e-4326-a4f8-8856b1a30e36-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-2x92x\" (UID: \"2ac2fc9c-0a3e-4326-a4f8-8856b1a30e36\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-2x92x"
Nov 03 20:14:42.236346 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:42.236329    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ac2fc9c-0a3e-4326-a4f8-8856b1a30e36-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-2x92x\" (UID: \"2ac2fc9c-0a3e-4326-a4f8-8856b1a30e36\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-2x92x"
Nov 03 20:14:42.338128 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:42.338042    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ac2fc9c-0a3e-4326-a4f8-8856b1a30e36-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-2x92x\" (UID: \"2ac2fc9c-0a3e-4326-a4f8-8856b1a30e36\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-2x92x"
Nov 03 20:14:42.338128 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:42.338119    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ac2fc9c-0a3e-4326-a4f8-8856b1a30e36-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-2x92x\" (UID: \"2ac2fc9c-0a3e-4326-a4f8-8856b1a30e36\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-2x92x"
Nov 03 20:14:42.338332 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:42.338160    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ac2fc9c-0a3e-4326-a4f8-8856b1a30e36-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-2x92x\" (UID: \"2ac2fc9c-0a3e-4326-a4f8-8856b1a30e36\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-2x92x"
Nov 03 20:14:42.338332 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:42.338188    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xp4s9\" (UniqueName: \"kubernetes.io/projected/2ac2fc9c-0a3e-4326-a4f8-8856b1a30e36-kube-api-access-xp4s9\") pod \"maas-default-gateway-openshift-default-694db55c87-2x92x\" (UID: \"2ac2fc9c-0a3e-4326-a4f8-8856b1a30e36\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-2x92x"
Nov 03 20:14:42.338332 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:42.338238    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ac2fc9c-0a3e-4326-a4f8-8856b1a30e36-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-2x92x\" (UID: \"2ac2fc9c-0a3e-4326-a4f8-8856b1a30e36\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-2x92x"
Nov 03 20:14:42.338332 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:42.338260    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ac2fc9c-0a3e-4326-a4f8-8856b1a30e36-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-2x92x\" (UID: \"2ac2fc9c-0a3e-4326-a4f8-8856b1a30e36\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-2x92x"
Nov 03 20:14:42.338332 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:42.338294    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ac2fc9c-0a3e-4326-a4f8-8856b1a30e36-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-2x92x\" (UID: \"2ac2fc9c-0a3e-4326-a4f8-8856b1a30e36\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-2x92x"
Nov 03 20:14:42.338332 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:42.338327    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ac2fc9c-0a3e-4326-a4f8-8856b1a30e36-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-2x92x\" (UID: \"2ac2fc9c-0a3e-4326-a4f8-8856b1a30e36\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-2x92x"
Nov 03 20:14:42.338628 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:42.338353    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ac2fc9c-0a3e-4326-a4f8-8856b1a30e36-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-2x92x\" (UID: \"2ac2fc9c-0a3e-4326-a4f8-8856b1a30e36\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-2x92x"
Nov 03 20:14:42.338790 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:42.338722    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ac2fc9c-0a3e-4326-a4f8-8856b1a30e36-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-2x92x\" (UID: \"2ac2fc9c-0a3e-4326-a4f8-8856b1a30e36\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-2x92x"
Nov 03 20:14:42.338921 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:42.338805    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ac2fc9c-0a3e-4326-a4f8-8856b1a30e36-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-2x92x\" (UID: \"2ac2fc9c-0a3e-4326-a4f8-8856b1a30e36\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-2x92x"
Nov 03 20:14:42.338921 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:42.338881    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ac2fc9c-0a3e-4326-a4f8-8856b1a30e36-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-2x92x\" (UID: \"2ac2fc9c-0a3e-4326-a4f8-8856b1a30e36\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-2x92x"
Nov 03 20:14:42.338921 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:42.338904    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ac2fc9c-0a3e-4326-a4f8-8856b1a30e36-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-2x92x\" (UID: \"2ac2fc9c-0a3e-4326-a4f8-8856b1a30e36\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-2x92x"
Nov 03 20:14:42.339264 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:42.339241    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ac2fc9c-0a3e-4326-a4f8-8856b1a30e36-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-2x92x\" (UID: \"2ac2fc9c-0a3e-4326-a4f8-8856b1a30e36\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-2x92x"
Nov 03 20:14:42.340581 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:42.340559    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ac2fc9c-0a3e-4326-a4f8-8856b1a30e36-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-2x92x\" (UID: \"2ac2fc9c-0a3e-4326-a4f8-8856b1a30e36\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-2x92x"
Nov 03 20:14:42.340814 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:42.340792    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ac2fc9c-0a3e-4326-a4f8-8856b1a30e36-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-2x92x\" (UID: \"2ac2fc9c-0a3e-4326-a4f8-8856b1a30e36\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-2x92x"
Nov 03 20:14:42.356982 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:42.356952    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ac2fc9c-0a3e-4326-a4f8-8856b1a30e36-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-2x92x\" (UID: \"2ac2fc9c-0a3e-4326-a4f8-8856b1a30e36\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-2x92x"
Nov 03 20:14:42.357202 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:42.357177    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xp4s9\" (UniqueName: \"kubernetes.io/projected/2ac2fc9c-0a3e-4326-a4f8-8856b1a30e36-kube-api-access-xp4s9\") pod \"maas-default-gateway-openshift-default-694db55c87-2x92x\" (UID: \"2ac2fc9c-0a3e-4326-a4f8-8856b1a30e36\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-2x92x"
Nov 03 20:14:42.378584 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:42.378552    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-2x92x"
Nov 03 20:14:42.690435 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:42.690380    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-2x92x"]
Nov 03 20:14:42.696298 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:14:42.696257    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ac2fc9c_0a3e_4326_a4f8_8856b1a30e36.slice/crio-4349d6d8cd982d01734cc03ceb2e0b234d18b47689e6e8c4bdabbbce5bdbd420 WatchSource:0}: Error finding container 4349d6d8cd982d01734cc03ceb2e0b234d18b47689e6e8c4bdabbbce5bdbd420: Status 404 returned error can't find the container with id 4349d6d8cd982d01734cc03ceb2e0b234d18b47689e6e8c4bdabbbce5bdbd420
Nov 03 20:14:42.699487 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:42.699451    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:14:42.699579 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:42.699523    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:14:42.699579 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:42.699550    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:14:43.165702 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:43.165619    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-bf6bcc597-rwjc5"]
Nov 03 20:14:43.169386 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:43.169362    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-bf6bcc597-rwjc5"
Nov 03 20:14:43.184306 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:43.184276    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-bf6bcc597-rwjc5"]
Nov 03 20:14:43.195126 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:43.195086    5570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\""
Nov 03 20:14:43.195126 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:43.195087    5570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Nov 03 20:14:43.195323 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:43.195157    5570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Nov 03 20:14:43.196638 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:43.196616    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\""
Nov 03 20:14:43.196721 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:43.196692    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-xbqq6\""
Nov 03 20:14:43.361374 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:43.361325    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44tsp\" (UniqueName: \"kubernetes.io/projected/77fe3c84-a597-4a9d-a49d-392c266db493-kube-api-access-44tsp\") pod \"ai-gateway-controller-bf6bcc597-rwjc5\" (UID: \"77fe3c84-a597-4a9d-a49d-392c266db493\") " pod="opendatahub/ai-gateway-controller-bf6bcc597-rwjc5"
Nov 03 20:14:43.361715 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:43.361690    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77fe3c84-a597-4a9d-a49d-392c266db493-metrics-certs\") pod \"ai-gateway-controller-bf6bcc597-rwjc5\" (UID: \"77fe3c84-a597-4a9d-a49d-392c266db493\") " pod="opendatahub/ai-gateway-controller-bf6bcc597-rwjc5"
Nov 03 20:14:43.463404 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:43.463352    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77fe3c84-a597-4a9d-a49d-392c266db493-metrics-certs\") pod \"ai-gateway-controller-bf6bcc597-rwjc5\" (UID: \"77fe3c84-a597-4a9d-a49d-392c266db493\") " pod="opendatahub/ai-gateway-controller-bf6bcc597-rwjc5"
Nov 03 20:14:43.463600 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:43.463518    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44tsp\" (UniqueName: \"kubernetes.io/projected/77fe3c84-a597-4a9d-a49d-392c266db493-kube-api-access-44tsp\") pod \"ai-gateway-controller-bf6bcc597-rwjc5\" (UID: \"77fe3c84-a597-4a9d-a49d-392c266db493\") " pod="opendatahub/ai-gateway-controller-bf6bcc597-rwjc5"
Nov 03 20:14:43.465879 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:43.465855    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77fe3c84-a597-4a9d-a49d-392c266db493-metrics-certs\") pod \"ai-gateway-controller-bf6bcc597-rwjc5\" (UID: \"77fe3c84-a597-4a9d-a49d-392c266db493\") " pod="opendatahub/ai-gateway-controller-bf6bcc597-rwjc5"
Nov 03 20:14:43.482272 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:43.482243    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-44tsp\" (UniqueName: \"kubernetes.io/projected/77fe3c84-a597-4a9d-a49d-392c266db493-kube-api-access-44tsp\") pod \"ai-gateway-controller-bf6bcc597-rwjc5\" (UID: \"77fe3c84-a597-4a9d-a49d-392c266db493\") " pod="opendatahub/ai-gateway-controller-bf6bcc597-rwjc5"
Nov 03 20:14:43.513337 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:43.513299    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-2x92x" event={"ID":"2ac2fc9c-0a3e-4326-a4f8-8856b1a30e36","Type":"ContainerStarted","Data":"ba1d33e261ae5b946cd1afd83be4e12a57d46a9a8ace20e5a4394ae6e0f6b53f"}
Nov 03 20:14:43.513520 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:43.513354    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-2x92x" event={"ID":"2ac2fc9c-0a3e-4326-a4f8-8856b1a30e36","Type":"ContainerStarted","Data":"4349d6d8cd982d01734cc03ceb2e0b234d18b47689e6e8c4bdabbbce5bdbd420"}
Nov 03 20:14:43.563473 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:43.563407    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-2x92x" podStartSLOduration=1.563387025 podStartE2EDuration="1.563387025s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:14:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:14:43.562392523 +0000 UTC m=+617.715054410" watchObservedRunningTime="2026-11-03 20:14:43.563387025 +0000 UTC m=+617.716048913"
Nov 03 20:14:43.781804 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:43.781688    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-bf6bcc597-rwjc5"
Nov 03 20:14:44.090785 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:44.090672    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-bf6bcc597-rwjc5"]
Nov 03 20:14:44.095085 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:14:44.095051    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77fe3c84_a597_4a9d_a49d_392c266db493.slice/crio-4a3997b226761ac72561d7e07bfb7733c48cd9b5ca4f6ab8541cd01eb01d20f0 WatchSource:0}: Error finding container 4a3997b226761ac72561d7e07bfb7733c48cd9b5ca4f6ab8541cd01eb01d20f0: Status 404 returned error can't find the container with id 4a3997b226761ac72561d7e07bfb7733c48cd9b5ca4f6ab8541cd01eb01d20f0
Nov 03 20:14:44.379703 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:44.379619    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-2x92x"
Nov 03 20:14:44.380925 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:44.380903    5570 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-2x92x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body=
Nov 03 20:14:44.380983 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:44.380952    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-2x92x" podUID="2ac2fc9c-0a3e-4326-a4f8-8856b1a30e36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused"
Nov 03 20:14:44.520585 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:44.520490    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-bf6bcc597-rwjc5" event={"ID":"77fe3c84-a597-4a9d-a49d-392c266db493","Type":"ContainerStarted","Data":"4a3997b226761ac72561d7e07bfb7733c48cd9b5ca4f6ab8541cd01eb01d20f0"}
Nov 03 20:14:45.379059 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:45.379023    5570 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-2x92x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body=
Nov 03 20:14:45.379259 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:45.379075    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-2x92x" podUID="2ac2fc9c-0a3e-4326-a4f8-8856b1a30e36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused"
Nov 03 20:14:46.379341 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:46.379309    5570 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-2x92x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body=
Nov 03 20:14:46.379732 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:46.379361    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-2x92x" podUID="2ac2fc9c-0a3e-4326-a4f8-8856b1a30e36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused"
Nov 03 20:14:47.379635 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:47.379601    5570 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-2x92x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body=
Nov 03 20:14:47.380098 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:47.379653    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-2x92x" podUID="2ac2fc9c-0a3e-4326-a4f8-8856b1a30e36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused"
Nov 03 20:14:47.562215 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:47.562165    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-bf6bcc597-rwjc5" event={"ID":"77fe3c84-a597-4a9d-a49d-392c266db493","Type":"ContainerStarted","Data":"241dfe9da149c1f70a7838363bef89af028e433c6d7bd1d0d5b64b62457a378e"}
Nov 03 20:14:47.562508 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:47.562480    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-bf6bcc597-rwjc5"
Nov 03 20:14:47.600132 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:47.600074    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-bf6bcc597-rwjc5" podStartSLOduration=1.823143515 podStartE2EDuration="4.600059633s" totalImagesPullingTime="2.776916118s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:14:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:14:47.60001031 +0000 UTC m=+621.752672197" watchObservedRunningTime="2026-11-03 20:14:47.600059633 +0000 UTC m=+621.752721519"
Nov 03 20:14:48.379177 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:48.379131    5570 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-2x92x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body=
Nov 03 20:14:48.379375 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:48.379210    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-2x92x" podUID="2ac2fc9c-0a3e-4326-a4f8-8856b1a30e36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused"
Nov 03 20:14:49.379019 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:49.378975    5570 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-2x92x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body=
Nov 03 20:14:49.379551 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:49.379060    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-2x92x" podUID="2ac2fc9c-0a3e-4326-a4f8-8856b1a30e36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused"
Nov 03 20:14:50.848496 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:50.848460    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-2x92x"
Nov 03 20:14:50.848999 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:50.848976    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-2x92x"
Nov 03 20:14:50.849966 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:50.849947    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-2x92x"
Nov 03 20:14:57.226991 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:14:57.226950    5570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8bwhz" containerName="extractor"
Nov 03 20:14:58.569512 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:14:58.569477    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-bf6bcc597-rwjc5"
Nov 03 20:15:00.144471 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:00.144437    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895615-m4lwt"]
Nov 03 20:15:00.149198 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:00.149172    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895615-m4lwt"
Nov 03 20:15:00.163616 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:00.163565    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895615-m4lwt"]
Nov 03 20:15:00.173661 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:00.173637    5570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 20:15:00.173835 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:00.173709    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l7s5w\""
Nov 03 20:15:00.173835 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:00.173709    5570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 20:15:00.217994 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:00.217962    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qs6j\" (UniqueName: \"kubernetes.io/projected/4a8f1a3b-f507-4401-b1b0-c573675817e9-kube-api-access-9qs6j\") pod \"maas-api-key-cleanup-29895615-m4lwt\" (UID: \"4a8f1a3b-f507-4401-b1b0-c573675817e9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895615-m4lwt"
Nov 03 20:15:00.218147 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:00.217998    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a8f1a3b-f507-4401-b1b0-c573675817e9-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895615-m4lwt\" (UID: \"4a8f1a3b-f507-4401-b1b0-c573675817e9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895615-m4lwt"
Nov 03 20:15:00.319420 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:00.319386    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qs6j\" (UniqueName: \"kubernetes.io/projected/4a8f1a3b-f507-4401-b1b0-c573675817e9-kube-api-access-9qs6j\") pod \"maas-api-key-cleanup-29895615-m4lwt\" (UID: \"4a8f1a3b-f507-4401-b1b0-c573675817e9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895615-m4lwt"
Nov 03 20:15:00.319577 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:00.319428    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a8f1a3b-f507-4401-b1b0-c573675817e9-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895615-m4lwt\" (UID: \"4a8f1a3b-f507-4401-b1b0-c573675817e9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895615-m4lwt"
Nov 03 20:15:00.320173 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:00.320151    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a8f1a3b-f507-4401-b1b0-c573675817e9-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895615-m4lwt\" (UID: \"4a8f1a3b-f507-4401-b1b0-c573675817e9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895615-m4lwt"
Nov 03 20:15:00.338041 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:00.338011    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qs6j\" (UniqueName: \"kubernetes.io/projected/4a8f1a3b-f507-4401-b1b0-c573675817e9-kube-api-access-9qs6j\") pod \"maas-api-key-cleanup-29895615-m4lwt\" (UID: \"4a8f1a3b-f507-4401-b1b0-c573675817e9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895615-m4lwt"
Nov 03 20:15:00.461923 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:00.461888    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895615-m4lwt"
Nov 03 20:15:00.757922 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:00.757840    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895615-m4lwt"]
Nov 03 20:15:00.761415 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:15:00.761379    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a8f1a3b_f507_4401_b1b0_c573675817e9.slice/crio-2c87e3e8aed3e063daeaea1acfc8dad41e1373d1277dd5a9c3d7c42e73549856 WatchSource:0}: Error finding container 2c87e3e8aed3e063daeaea1acfc8dad41e1373d1277dd5a9c3d7c42e73549856: Status 404 returned error can't find the container with id 2c87e3e8aed3e063daeaea1acfc8dad41e1373d1277dd5a9c3d7c42e73549856
Nov 03 20:15:00.763350 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:00.763331    5570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 20:15:01.628676 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:01.628638    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895615-m4lwt" event={"ID":"4a8f1a3b-f507-4401-b1b0-c573675817e9","Type":"ContainerStarted","Data":"2c87e3e8aed3e063daeaea1acfc8dad41e1373d1277dd5a9c3d7c42e73549856"}
Nov 03 20:15:04.644149 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:04.644054    5570 generic.go:356] "Generic (PLEG): container finished" podID="4a8f1a3b-f507-4401-b1b0-c573675817e9" containerID="84e2184a38e3fa0b16de6687626e37d0bea123aa33f66615898b6b762efe176b" exitCode=60
Nov 03 20:15:04.644536 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:04.644181    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895615-m4lwt" event={"ID":"4a8f1a3b-f507-4401-b1b0-c573675817e9","Type":"ContainerDied","Data":"84e2184a38e3fa0b16de6687626e37d0bea123aa33f66615898b6b762efe176b"}
Nov 03 20:15:04.644536 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:04.644382    5570 scope.go:122] "RemoveContainer" containerID="84e2184a38e3fa0b16de6687626e37d0bea123aa33f66615898b6b762efe176b"
Nov 03 20:15:05.650993 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:05.650959    5570 generic.go:356] "Generic (PLEG): container finished" podID="4a8f1a3b-f507-4401-b1b0-c573675817e9" containerID="089021de348b2068186d262ab8602e2e7b866601538c362b16d0aac033b3bd66" exitCode=60
Nov 03 20:15:05.651466 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:05.651047    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895615-m4lwt" event={"ID":"4a8f1a3b-f507-4401-b1b0-c573675817e9","Type":"ContainerDied","Data":"089021de348b2068186d262ab8602e2e7b866601538c362b16d0aac033b3bd66"}
Nov 03 20:15:05.651466 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:05.651119    5570 scope.go:122] "RemoveContainer" containerID="84e2184a38e3fa0b16de6687626e37d0bea123aa33f66615898b6b762efe176b"
Nov 03 20:15:05.651466 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:05.651379    5570 scope.go:122] "RemoveContainer" containerID="089021de348b2068186d262ab8602e2e7b866601538c362b16d0aac033b3bd66"
Nov 03 20:15:05.651636 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:15:05.651618    5570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895615-m4lwt_odh-ai-gateway-infra(4a8f1a3b-f507-4401-b1b0-c573675817e9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895615-m4lwt" podUID="4a8f1a3b-f507-4401-b1b0-c573675817e9"
Nov 03 20:15:06.657719 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:06.657693    5570 scope.go:122] "RemoveContainer" containerID="089021de348b2068186d262ab8602e2e7b866601538c362b16d0aac033b3bd66"
Nov 03 20:15:06.658130 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:15:06.657945    5570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895615-m4lwt_odh-ai-gateway-infra(4a8f1a3b-f507-4401-b1b0-c573675817e9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895615-m4lwt" podUID="4a8f1a3b-f507-4401-b1b0-c573675817e9"
Nov 03 20:15:17.958243 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:17.958201    5570 scope.go:122] "RemoveContainer" containerID="089021de348b2068186d262ab8602e2e7b866601538c362b16d0aac033b3bd66"
Nov 03 20:15:18.737966 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:18.737933    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895615-m4lwt" event={"ID":"4a8f1a3b-f507-4401-b1b0-c573675817e9","Type":"ContainerStarted","Data":"76b9ad0727461bdafdce857eb3e8c9ecf9858dfa2eb7e021249c12c9a04100b7"}
Nov 03 20:15:18.777986 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:18.777923    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895615-m4lwt" podStartSLOduration=16.097733878 podStartE2EDuration="18.777909415s" totalImagesPullingTime="2.680175537s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:15:18.776765789 +0000 UTC m=+652.929427677" watchObservedRunningTime="2026-11-03 20:15:18.777909415 +0000 UTC m=+652.930571301"
Nov 03 20:15:19.745049 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:19.745014    5570 generic.go:356] "Generic (PLEG): container finished" podID="4a8f1a3b-f507-4401-b1b0-c573675817e9" containerID="76b9ad0727461bdafdce857eb3e8c9ecf9858dfa2eb7e021249c12c9a04100b7" exitCode=60
Nov 03 20:15:19.745561 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:19.745056    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895615-m4lwt" event={"ID":"4a8f1a3b-f507-4401-b1b0-c573675817e9","Type":"ContainerDied","Data":"76b9ad0727461bdafdce857eb3e8c9ecf9858dfa2eb7e021249c12c9a04100b7"}
Nov 03 20:15:19.745561 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:19.745131    5570 scope.go:122] "RemoveContainer" containerID="089021de348b2068186d262ab8602e2e7b866601538c362b16d0aac033b3bd66"
Nov 03 20:15:19.745561 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:19.745503    5570 scope.go:122] "RemoveContainer" containerID="76b9ad0727461bdafdce857eb3e8c9ecf9858dfa2eb7e021249c12c9a04100b7"
Nov 03 20:15:19.745827 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:15:19.745801    5570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29895615-m4lwt_odh-ai-gateway-infra(4a8f1a3b-f507-4401-b1b0-c573675817e9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895615-m4lwt" podUID="4a8f1a3b-f507-4401-b1b0-c573675817e9"
Nov 03 20:15:19.788636 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:19.788600    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895615-m4lwt"]
Nov 03 20:15:20.890134 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:20.890102    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895615-m4lwt"
Nov 03 20:15:21.016972 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:21.016871    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a8f1a3b-f507-4401-b1b0-c573675817e9-kube-api-access-9qs6j\" (UniqueName: \"kubernetes.io/projected/4a8f1a3b-f507-4401-b1b0-c573675817e9-kube-api-access-9qs6j\") pod \"4a8f1a3b-f507-4401-b1b0-c573675817e9\" (UID: \"4a8f1a3b-f507-4401-b1b0-c573675817e9\") "
Nov 03 20:15:21.016972 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:21.016950    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a8f1a3b-f507-4401-b1b0-c573675817e9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a8f1a3b-f507-4401-b1b0-c573675817e9-maas-api-service-ca\") pod \"4a8f1a3b-f507-4401-b1b0-c573675817e9\" (UID: \"4a8f1a3b-f507-4401-b1b0-c573675817e9\") "
Nov 03 20:15:21.017313 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:21.017284    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a8f1a3b-f507-4401-b1b0-c573675817e9-maas-api-service-ca" pod "4a8f1a3b-f507-4401-b1b0-c573675817e9" (UID: "4a8f1a3b-f507-4401-b1b0-c573675817e9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:15:21.019258 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:21.019224    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a8f1a3b-f507-4401-b1b0-c573675817e9-kube-api-access-9qs6j" pod "4a8f1a3b-f507-4401-b1b0-c573675817e9" (UID: "4a8f1a3b-f507-4401-b1b0-c573675817e9"). InnerVolumeSpecName "kube-api-access-9qs6j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:15:21.117715 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:21.117671    5570 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a8f1a3b-f507-4401-b1b0-c573675817e9-maas-api-service-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:15:21.117715 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:21.117707    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qs6j\" (UniqueName: \"kubernetes.io/projected/4a8f1a3b-f507-4401-b1b0-c573675817e9-kube-api-access-9qs6j\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:15:21.758549 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:21.758514    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895615-m4lwt" event={"ID":"4a8f1a3b-f507-4401-b1b0-c573675817e9","Type":"ContainerDied","Data":"2c87e3e8aed3e063daeaea1acfc8dad41e1373d1277dd5a9c3d7c42e73549856"}
Nov 03 20:15:21.758720 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:21.758578    5570 scope.go:122] "RemoveContainer" containerID="76b9ad0727461bdafdce857eb3e8c9ecf9858dfa2eb7e021249c12c9a04100b7"
Nov 03 20:15:21.758720 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:21.758524    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895615-m4lwt"
Nov 03 20:15:21.808350 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:21.808310    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895615-m4lwt"]
Nov 03 20:15:21.820266 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:21.820238    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895615-m4lwt"]
Nov 03 20:15:21.963185 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:21.963154    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a8f1a3b-f507-4401-b1b0-c573675817e9" path="/var/lib/kubelet/pods/4a8f1a3b-f507-4401-b1b0-c573675817e9/volumes"
Nov 03 20:15:25.064891 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.064840    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-c5nf4"]
Nov 03 20:15:25.065433 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.065384    5570 cpu_manager.go:414] "containerMap: removing container" podUID="4a8f1a3b-f507-4401-b1b0-c573675817e9" containerName="cleanup"
Nov 03 20:15:25.065433 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.065401    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a8f1a3b-f507-4401-b1b0-c573675817e9" containerName="cleanup"
Nov 03 20:15:25.065433 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.065434    5570 cpu_manager.go:414] "containerMap: removing container" podUID="4a8f1a3b-f507-4401-b1b0-c573675817e9" containerName="cleanup"
Nov 03 20:15:25.065541 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.065439    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a8f1a3b-f507-4401-b1b0-c573675817e9" containerName="cleanup"
Nov 03 20:15:25.065541 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.065502    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a8f1a3b-f507-4401-b1b0-c573675817e9" containerName="cleanup"
Nov 03 20:15:25.065541 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.065510    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a8f1a3b-f507-4401-b1b0-c573675817e9" containerName="cleanup"
Nov 03 20:15:25.065541 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.065519    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a8f1a3b-f507-4401-b1b0-c573675817e9" containerName="cleanup"
Nov 03 20:15:25.070191 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.070170    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-c5nf4"
Nov 03 20:15:25.091725 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.091693    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-c5nf4"]
Nov 03 20:15:25.103255 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.101682    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-fd01ad-openshift-default-dockercfg-lbghm\""
Nov 03 20:15:25.155671 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.155630    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a255596-5346-460f-b437-7c48d3d33d54-workload-certs\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-c5nf4\" (UID: \"8a255596-5346-460f-b437-7c48d3d33d54\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-c5nf4"
Nov 03 20:15:25.155917 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.155683    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a255596-5346-460f-b437-7c48d3d33d54-workload-socket\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-c5nf4\" (UID: \"8a255596-5346-460f-b437-7c48d3d33d54\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-c5nf4"
Nov 03 20:15:25.155917 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.155761    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a255596-5346-460f-b437-7c48d3d33d54-istio-envoy\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-c5nf4\" (UID: \"8a255596-5346-460f-b437-7c48d3d33d54\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-c5nf4"
Nov 03 20:15:25.155917 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.155858    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a255596-5346-460f-b437-7c48d3d33d54-istio-token\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-c5nf4\" (UID: \"8a255596-5346-460f-b437-7c48d3d33d54\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-c5nf4"
Nov 03 20:15:25.156165 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.155956    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56ng4\" (UniqueName: \"kubernetes.io/projected/8a255596-5346-460f-b437-7c48d3d33d54-kube-api-access-56ng4\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-c5nf4\" (UID: \"8a255596-5346-460f-b437-7c48d3d33d54\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-c5nf4"
Nov 03 20:15:25.156165 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.156013    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a255596-5346-460f-b437-7c48d3d33d54-istio-podinfo\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-c5nf4\" (UID: \"8a255596-5346-460f-b437-7c48d3d33d54\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-c5nf4"
Nov 03 20:15:25.156165 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.156101    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a255596-5346-460f-b437-7c48d3d33d54-istiod-ca-cert\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-c5nf4\" (UID: \"8a255596-5346-460f-b437-7c48d3d33d54\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-c5nf4"
Nov 03 20:15:25.156336 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.156166    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a255596-5346-460f-b437-7c48d3d33d54-credential-socket\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-c5nf4\" (UID: \"8a255596-5346-460f-b437-7c48d3d33d54\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-c5nf4"
Nov 03 20:15:25.156336 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.156193    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a255596-5346-460f-b437-7c48d3d33d54-istio-data\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-c5nf4\" (UID: \"8a255596-5346-460f-b437-7c48d3d33d54\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-c5nf4"
Nov 03 20:15:25.202575 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.202534    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-vzfw9"]
Nov 03 20:15:25.203202 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.203174    5570 cpu_manager.go:414] "containerMap: removing container" podUID="4a8f1a3b-f507-4401-b1b0-c573675817e9" containerName="cleanup"
Nov 03 20:15:25.203202 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.203204    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a8f1a3b-f507-4401-b1b0-c573675817e9" containerName="cleanup"
Nov 03 20:15:25.206551 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.206530    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-vzfw9"
Nov 03 20:15:25.224524 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.224494    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-9e1bcb-openshift-default-dockercfg-sjgmc\""
Nov 03 20:15:25.226092 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.226055    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-vzfw9"]
Nov 03 20:15:25.257541 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.257505    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a255596-5346-460f-b437-7c48d3d33d54-istiod-ca-cert\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-c5nf4\" (UID: \"8a255596-5346-460f-b437-7c48d3d33d54\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-c5nf4"
Nov 03 20:15:25.257727 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.257563    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a255596-5346-460f-b437-7c48d3d33d54-credential-socket\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-c5nf4\" (UID: \"8a255596-5346-460f-b437-7c48d3d33d54\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-c5nf4"
Nov 03 20:15:25.257727 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.257581    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a255596-5346-460f-b437-7c48d3d33d54-istio-data\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-c5nf4\" (UID: \"8a255596-5346-460f-b437-7c48d3d33d54\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-c5nf4"
Nov 03 20:15:25.257727 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.257620    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a255596-5346-460f-b437-7c48d3d33d54-workload-certs\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-c5nf4\" (UID: \"8a255596-5346-460f-b437-7c48d3d33d54\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-c5nf4"
Nov 03 20:15:25.257727 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.257639    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a255596-5346-460f-b437-7c48d3d33d54-workload-socket\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-c5nf4\" (UID: \"8a255596-5346-460f-b437-7c48d3d33d54\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-c5nf4"
Nov 03 20:15:25.257727 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.257663    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a255596-5346-460f-b437-7c48d3d33d54-istio-envoy\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-c5nf4\" (UID: \"8a255596-5346-460f-b437-7c48d3d33d54\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-c5nf4"
Nov 03 20:15:25.257727 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.257680    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a255596-5346-460f-b437-7c48d3d33d54-istio-token\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-c5nf4\" (UID: \"8a255596-5346-460f-b437-7c48d3d33d54\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-c5nf4"
Nov 03 20:15:25.257727 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.257709    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56ng4\" (UniqueName: \"kubernetes.io/projected/8a255596-5346-460f-b437-7c48d3d33d54-kube-api-access-56ng4\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-c5nf4\" (UID: \"8a255596-5346-460f-b437-7c48d3d33d54\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-c5nf4"
Nov 03 20:15:25.258193 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.257758    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a255596-5346-460f-b437-7c48d3d33d54-istio-podinfo\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-c5nf4\" (UID: \"8a255596-5346-460f-b437-7c48d3d33d54\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-c5nf4"
Nov 03 20:15:25.258193 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.258090    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a255596-5346-460f-b437-7c48d3d33d54-workload-socket\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-c5nf4\" (UID: \"8a255596-5346-460f-b437-7c48d3d33d54\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-c5nf4"
Nov 03 20:15:25.258193 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.258150    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a255596-5346-460f-b437-7c48d3d33d54-istio-data\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-c5nf4\" (UID: \"8a255596-5346-460f-b437-7c48d3d33d54\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-c5nf4"
Nov 03 20:15:25.258301 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.258240    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a255596-5346-460f-b437-7c48d3d33d54-credential-socket\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-c5nf4\" (UID: \"8a255596-5346-460f-b437-7c48d3d33d54\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-c5nf4"
Nov 03 20:15:25.258380 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.258363    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a255596-5346-460f-b437-7c48d3d33d54-workload-certs\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-c5nf4\" (UID: \"8a255596-5346-460f-b437-7c48d3d33d54\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-c5nf4"
Nov 03 20:15:25.258574 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.258553    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a255596-5346-460f-b437-7c48d3d33d54-istiod-ca-cert\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-c5nf4\" (UID: \"8a255596-5346-460f-b437-7c48d3d33d54\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-c5nf4"
Nov 03 20:15:25.260181 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.260163    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a255596-5346-460f-b437-7c48d3d33d54-istio-envoy\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-c5nf4\" (UID: \"8a255596-5346-460f-b437-7c48d3d33d54\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-c5nf4"
Nov 03 20:15:25.260324 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.260306    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a255596-5346-460f-b437-7c48d3d33d54-istio-podinfo\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-c5nf4\" (UID: \"8a255596-5346-460f-b437-7c48d3d33d54\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-c5nf4"
Nov 03 20:15:25.277114 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.277076    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a255596-5346-460f-b437-7c48d3d33d54-istio-token\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-c5nf4\" (UID: \"8a255596-5346-460f-b437-7c48d3d33d54\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-c5nf4"
Nov 03 20:15:25.277266 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.277243    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56ng4\" (UniqueName: \"kubernetes.io/projected/8a255596-5346-460f-b437-7c48d3d33d54-kube-api-access-56ng4\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-c5nf4\" (UID: \"8a255596-5346-460f-b437-7c48d3d33d54\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-c5nf4"
Nov 03 20:15:25.324421 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.324324    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-nz52s"]
Nov 03 20:15:25.330006 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.329976    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-nz52s"
Nov 03 20:15:25.350134 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.350099    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-nz52s"]
Nov 03 20:15:25.359209 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.359178    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a739707b-9c5a-4c9b-90df-501c5dd0c98d-workload-socket\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-vzfw9\" (UID: \"a739707b-9c5a-4c9b-90df-501c5dd0c98d\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-vzfw9"
Nov 03 20:15:25.359405 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.359219    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a739707b-9c5a-4c9b-90df-501c5dd0c98d-istiod-ca-cert\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-vzfw9\" (UID: \"a739707b-9c5a-4c9b-90df-501c5dd0c98d\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-vzfw9"
Nov 03 20:15:25.359405 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.359251    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-879mv\" (UniqueName: \"kubernetes.io/projected/a739707b-9c5a-4c9b-90df-501c5dd0c98d-kube-api-access-879mv\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-vzfw9\" (UID: \"a739707b-9c5a-4c9b-90df-501c5dd0c98d\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-vzfw9"
Nov 03 20:15:25.359405 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.359355    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a739707b-9c5a-4c9b-90df-501c5dd0c98d-credential-socket\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-vzfw9\" (UID: \"a739707b-9c5a-4c9b-90df-501c5dd0c98d\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-vzfw9"
Nov 03 20:15:25.359564 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.359448    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a739707b-9c5a-4c9b-90df-501c5dd0c98d-workload-certs\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-vzfw9\" (UID: \"a739707b-9c5a-4c9b-90df-501c5dd0c98d\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-vzfw9"
Nov 03 20:15:25.359564 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.359476    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a739707b-9c5a-4c9b-90df-501c5dd0c98d-istio-token\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-vzfw9\" (UID: \"a739707b-9c5a-4c9b-90df-501c5dd0c98d\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-vzfw9"
Nov 03 20:15:25.359564 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.359510    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a739707b-9c5a-4c9b-90df-501c5dd0c98d-istio-podinfo\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-vzfw9\" (UID: \"a739707b-9c5a-4c9b-90df-501c5dd0c98d\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-vzfw9"
Nov 03 20:15:25.359661 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.359644    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a739707b-9c5a-4c9b-90df-501c5dd0c98d-istio-data\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-vzfw9\" (UID: \"a739707b-9c5a-4c9b-90df-501c5dd0c98d\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-vzfw9"
Nov 03 20:15:25.359695 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.359680    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a739707b-9c5a-4c9b-90df-501c5dd0c98d-istio-envoy\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-vzfw9\" (UID: \"a739707b-9c5a-4c9b-90df-501c5dd0c98d\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-vzfw9"
Nov 03 20:15:25.383175 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.383138    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-c5nf4"
Nov 03 20:15:25.394860 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.394837    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-9973d9-openshift-default-dockercfg-br8jj\""
Nov 03 20:15:25.455344 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.455310    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-wg864"]
Nov 03 20:15:25.461077 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.461032    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/974badc1-41ad-461c-8719-e1f3f836ccbc-credential-socket\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-nz52s\" (UID: \"974badc1-41ad-461c-8719-e1f3f836ccbc\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-nz52s"
Nov 03 20:15:25.461269 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.461100    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a739707b-9c5a-4c9b-90df-501c5dd0c98d-istio-data\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-vzfw9\" (UID: \"a739707b-9c5a-4c9b-90df-501c5dd0c98d\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-vzfw9"
Nov 03 20:15:25.461269 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.461160    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/974badc1-41ad-461c-8719-e1f3f836ccbc-istio-envoy\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-nz52s\" (UID: \"974badc1-41ad-461c-8719-e1f3f836ccbc\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-nz52s"
Nov 03 20:15:25.461269 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.461201    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a739707b-9c5a-4c9b-90df-501c5dd0c98d-istio-envoy\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-vzfw9\" (UID: \"a739707b-9c5a-4c9b-90df-501c5dd0c98d\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-vzfw9"
Nov 03 20:15:25.461269 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.461217    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/974badc1-41ad-461c-8719-e1f3f836ccbc-istio-token\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-nz52s\" (UID: \"974badc1-41ad-461c-8719-e1f3f836ccbc\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-nz52s"
Nov 03 20:15:25.461269 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.461244    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-wg864"
Nov 03 20:15:25.461269 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.461262    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/974badc1-41ad-461c-8719-e1f3f836ccbc-workload-certs\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-nz52s\" (UID: \"974badc1-41ad-461c-8719-e1f3f836ccbc\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-nz52s"
Nov 03 20:15:25.461575 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.461282    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/974badc1-41ad-461c-8719-e1f3f836ccbc-workload-socket\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-nz52s\" (UID: \"974badc1-41ad-461c-8719-e1f3f836ccbc\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-nz52s"
Nov 03 20:15:25.461575 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.461300    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a739707b-9c5a-4c9b-90df-501c5dd0c98d-workload-socket\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-vzfw9\" (UID: \"a739707b-9c5a-4c9b-90df-501c5dd0c98d\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-vzfw9"
Nov 03 20:15:25.461575 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.461324    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a739707b-9c5a-4c9b-90df-501c5dd0c98d-istiod-ca-cert\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-vzfw9\" (UID: \"a739707b-9c5a-4c9b-90df-501c5dd0c98d\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-vzfw9"
Nov 03 20:15:25.461575 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.461346    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/974badc1-41ad-461c-8719-e1f3f836ccbc-istio-data\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-nz52s\" (UID: \"974badc1-41ad-461c-8719-e1f3f836ccbc\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-nz52s"
Nov 03 20:15:25.461575 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.461369    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-879mv\" (UniqueName: \"kubernetes.io/projected/a739707b-9c5a-4c9b-90df-501c5dd0c98d-kube-api-access-879mv\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-vzfw9\" (UID: \"a739707b-9c5a-4c9b-90df-501c5dd0c98d\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-vzfw9"
Nov 03 20:15:25.461575 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.461389    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a739707b-9c5a-4c9b-90df-501c5dd0c98d-credential-socket\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-vzfw9\" (UID: \"a739707b-9c5a-4c9b-90df-501c5dd0c98d\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-vzfw9"
Nov 03 20:15:25.461575 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.461414    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/974badc1-41ad-461c-8719-e1f3f836ccbc-istio-podinfo\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-nz52s\" (UID: \"974badc1-41ad-461c-8719-e1f3f836ccbc\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-nz52s"
Nov 03 20:15:25.461575 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.461455    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a739707b-9c5a-4c9b-90df-501c5dd0c98d-workload-certs\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-vzfw9\" (UID: \"a739707b-9c5a-4c9b-90df-501c5dd0c98d\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-vzfw9"
Nov 03 20:15:25.461575 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.461473    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a739707b-9c5a-4c9b-90df-501c5dd0c98d-istio-token\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-vzfw9\" (UID: \"a739707b-9c5a-4c9b-90df-501c5dd0c98d\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-vzfw9"
Nov 03 20:15:25.461575 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.461492    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a739707b-9c5a-4c9b-90df-501c5dd0c98d-istio-podinfo\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-vzfw9\" (UID: \"a739707b-9c5a-4c9b-90df-501c5dd0c98d\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-vzfw9"
Nov 03 20:15:25.461575 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.461515    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmnc2\" (UniqueName: \"kubernetes.io/projected/974badc1-41ad-461c-8719-e1f3f836ccbc-kube-api-access-tmnc2\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-nz52s\" (UID: \"974badc1-41ad-461c-8719-e1f3f836ccbc\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-nz52s"
Nov 03 20:15:25.461575 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.461532    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/974badc1-41ad-461c-8719-e1f3f836ccbc-istiod-ca-cert\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-nz52s\" (UID: \"974badc1-41ad-461c-8719-e1f3f836ccbc\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-nz52s"
Nov 03 20:15:25.461575 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.461537    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a739707b-9c5a-4c9b-90df-501c5dd0c98d-istio-data\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-vzfw9\" (UID: \"a739707b-9c5a-4c9b-90df-501c5dd0c98d\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-vzfw9"
Nov 03 20:15:25.462144 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.462122    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a739707b-9c5a-4c9b-90df-501c5dd0c98d-workload-certs\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-vzfw9\" (UID: \"a739707b-9c5a-4c9b-90df-501c5dd0c98d\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-vzfw9"
Nov 03 20:15:25.462378 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.462359    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a739707b-9c5a-4c9b-90df-501c5dd0c98d-workload-socket\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-vzfw9\" (UID: \"a739707b-9c5a-4c9b-90df-501c5dd0c98d\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-vzfw9"
Nov 03 20:15:25.462646 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.462626    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a739707b-9c5a-4c9b-90df-501c5dd0c98d-credential-socket\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-vzfw9\" (UID: \"a739707b-9c5a-4c9b-90df-501c5dd0c98d\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-vzfw9"
Nov 03 20:15:25.462936 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.462918    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a739707b-9c5a-4c9b-90df-501c5dd0c98d-istiod-ca-cert\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-vzfw9\" (UID: \"a739707b-9c5a-4c9b-90df-501c5dd0c98d\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-vzfw9"
Nov 03 20:15:25.464866 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.464826    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a739707b-9c5a-4c9b-90df-501c5dd0c98d-istio-envoy\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-vzfw9\" (UID: \"a739707b-9c5a-4c9b-90df-501c5dd0c98d\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-vzfw9"
Nov 03 20:15:25.465252 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.465234    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a739707b-9c5a-4c9b-90df-501c5dd0c98d-istio-podinfo\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-vzfw9\" (UID: \"a739707b-9c5a-4c9b-90df-501c5dd0c98d\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-vzfw9"
Nov 03 20:15:25.480327 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.479320    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-wg864"]
Nov 03 20:15:25.483715 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.483682    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-879mv\" (UniqueName: \"kubernetes.io/projected/a739707b-9c5a-4c9b-90df-501c5dd0c98d-kube-api-access-879mv\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-vzfw9\" (UID: \"a739707b-9c5a-4c9b-90df-501c5dd0c98d\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-vzfw9"
Nov 03 20:15:25.486789 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.486747    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a739707b-9c5a-4c9b-90df-501c5dd0c98d-istio-token\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-vzfw9\" (UID: \"a739707b-9c5a-4c9b-90df-501c5dd0c98d\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-vzfw9"
Nov 03 20:15:25.521993 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.521960    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-vzfw9"
Nov 03 20:15:25.525133 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.524945    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-6fc748-openshift-default-dockercfg-h5hq6\""
Nov 03 20:15:25.562357 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.562319    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a1b3d29-b647-46e1-9005-e540e937f85f-workload-certs\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-wg864\" (UID: \"6a1b3d29-b647-46e1-9005-e540e937f85f\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-wg864"
Nov 03 20:15:25.562357 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.562370    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmnc2\" (UniqueName: \"kubernetes.io/projected/974badc1-41ad-461c-8719-e1f3f836ccbc-kube-api-access-tmnc2\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-nz52s\" (UID: \"974badc1-41ad-461c-8719-e1f3f836ccbc\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-nz52s"
Nov 03 20:15:25.562708 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.562465    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/974badc1-41ad-461c-8719-e1f3f836ccbc-istiod-ca-cert\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-nz52s\" (UID: \"974badc1-41ad-461c-8719-e1f3f836ccbc\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-nz52s"
Nov 03 20:15:25.562708 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.562618    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a1b3d29-b647-46e1-9005-e540e937f85f-istio-envoy\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-wg864\" (UID: \"6a1b3d29-b647-46e1-9005-e540e937f85f\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-wg864"
Nov 03 20:15:25.562708 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.562656    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/974badc1-41ad-461c-8719-e1f3f836ccbc-credential-socket\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-nz52s\" (UID: \"974badc1-41ad-461c-8719-e1f3f836ccbc\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-nz52s"
Nov 03 20:15:25.562708 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.562685    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a1b3d29-b647-46e1-9005-e540e937f85f-istio-data\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-wg864\" (UID: \"6a1b3d29-b647-46e1-9005-e540e937f85f\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-wg864"
Nov 03 20:15:25.562901 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.562714    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a1b3d29-b647-46e1-9005-e540e937f85f-workload-socket\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-wg864\" (UID: \"6a1b3d29-b647-46e1-9005-e540e937f85f\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-wg864"
Nov 03 20:15:25.562901 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.562803    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a1b3d29-b647-46e1-9005-e540e937f85f-istio-podinfo\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-wg864\" (UID: \"6a1b3d29-b647-46e1-9005-e540e937f85f\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-wg864"
Nov 03 20:15:25.562901 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.562854    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/974badc1-41ad-461c-8719-e1f3f836ccbc-istio-envoy\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-nz52s\" (UID: \"974badc1-41ad-461c-8719-e1f3f836ccbc\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-nz52s"
Nov 03 20:15:25.562901 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.562883    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/974badc1-41ad-461c-8719-e1f3f836ccbc-istio-token\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-nz52s\" (UID: \"974badc1-41ad-461c-8719-e1f3f836ccbc\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-nz52s"
Nov 03 20:15:25.563039 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.562908    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stxjs\" (UniqueName: \"kubernetes.io/projected/6a1b3d29-b647-46e1-9005-e540e937f85f-kube-api-access-stxjs\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-wg864\" (UID: \"6a1b3d29-b647-46e1-9005-e540e937f85f\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-wg864"
Nov 03 20:15:25.563039 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.562927    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a1b3d29-b647-46e1-9005-e540e937f85f-istio-token\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-wg864\" (UID: \"6a1b3d29-b647-46e1-9005-e540e937f85f\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-wg864"
Nov 03 20:15:25.563039 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.562967    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a1b3d29-b647-46e1-9005-e540e937f85f-istiod-ca-cert\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-wg864\" (UID: \"6a1b3d29-b647-46e1-9005-e540e937f85f\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-wg864"
Nov 03 20:15:25.563039 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.562990    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/974badc1-41ad-461c-8719-e1f3f836ccbc-workload-certs\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-nz52s\" (UID: \"974badc1-41ad-461c-8719-e1f3f836ccbc\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-nz52s"
Nov 03 20:15:25.563039 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.563013    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/974badc1-41ad-461c-8719-e1f3f836ccbc-workload-socket\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-nz52s\" (UID: \"974badc1-41ad-461c-8719-e1f3f836ccbc\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-nz52s"
Nov 03 20:15:25.563209 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.563043    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/974badc1-41ad-461c-8719-e1f3f836ccbc-istio-data\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-nz52s\" (UID: \"974badc1-41ad-461c-8719-e1f3f836ccbc\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-nz52s"
Nov 03 20:15:25.563209 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.563080    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/974badc1-41ad-461c-8719-e1f3f836ccbc-istio-podinfo\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-nz52s\" (UID: \"974badc1-41ad-461c-8719-e1f3f836ccbc\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-nz52s"
Nov 03 20:15:25.563209 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.563085    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/974badc1-41ad-461c-8719-e1f3f836ccbc-credential-socket\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-nz52s\" (UID: \"974badc1-41ad-461c-8719-e1f3f836ccbc\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-nz52s"
Nov 03 20:15:25.563355 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.563334    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/974badc1-41ad-461c-8719-e1f3f836ccbc-workload-certs\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-nz52s\" (UID: \"974badc1-41ad-461c-8719-e1f3f836ccbc\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-nz52s"
Nov 03 20:15:25.563415 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.563391    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a1b3d29-b647-46e1-9005-e540e937f85f-credential-socket\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-wg864\" (UID: \"6a1b3d29-b647-46e1-9005-e540e937f85f\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-wg864"
Nov 03 20:15:25.563484 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.563461    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/974badc1-41ad-461c-8719-e1f3f836ccbc-istiod-ca-cert\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-nz52s\" (UID: \"974badc1-41ad-461c-8719-e1f3f836ccbc\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-nz52s"
Nov 03 20:15:25.563553 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.563531    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/974badc1-41ad-461c-8719-e1f3f836ccbc-workload-socket\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-nz52s\" (UID: \"974badc1-41ad-461c-8719-e1f3f836ccbc\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-nz52s"
Nov 03 20:15:25.563762 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.563725    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/974badc1-41ad-461c-8719-e1f3f836ccbc-istio-data\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-nz52s\" (UID: \"974badc1-41ad-461c-8719-e1f3f836ccbc\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-nz52s"
Nov 03 20:15:25.566035 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.566012    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/974badc1-41ad-461c-8719-e1f3f836ccbc-istio-envoy\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-nz52s\" (UID: \"974badc1-41ad-461c-8719-e1f3f836ccbc\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-nz52s"
Nov 03 20:15:25.566439 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.566423    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/974badc1-41ad-461c-8719-e1f3f836ccbc-istio-podinfo\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-nz52s\" (UID: \"974badc1-41ad-461c-8719-e1f3f836ccbc\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-nz52s"
Nov 03 20:15:25.583880 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.582767    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmnc2\" (UniqueName: \"kubernetes.io/projected/974badc1-41ad-461c-8719-e1f3f836ccbc-kube-api-access-tmnc2\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-nz52s\" (UID: \"974badc1-41ad-461c-8719-e1f3f836ccbc\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-nz52s"
Nov 03 20:15:25.585438 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.585400    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/974badc1-41ad-461c-8719-e1f3f836ccbc-istio-token\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-nz52s\" (UID: \"974badc1-41ad-461c-8719-e1f3f836ccbc\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-nz52s"
Nov 03 20:15:25.598951 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.598918    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-jjjr7"]
Nov 03 20:15:25.605068 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.605038    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-jjjr7"
Nov 03 20:15:25.626447 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.626387    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-e40816-openshift-default-dockercfg-rbqqp\""
Nov 03 20:15:25.626679 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.626661    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-jjjr7"]
Nov 03 20:15:25.646804 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.646621    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-nz52s"
Nov 03 20:15:25.664569 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.664535    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a1b3d29-b647-46e1-9005-e540e937f85f-credential-socket\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-wg864\" (UID: \"6a1b3d29-b647-46e1-9005-e540e937f85f\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-wg864"
Nov 03 20:15:25.664746 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.664588    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a1b3d29-b647-46e1-9005-e540e937f85f-workload-certs\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-wg864\" (UID: \"6a1b3d29-b647-46e1-9005-e540e937f85f\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-wg864"
Nov 03 20:15:25.664746 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.664650    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a1b3d29-b647-46e1-9005-e540e937f85f-istio-envoy\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-wg864\" (UID: \"6a1b3d29-b647-46e1-9005-e540e937f85f\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-wg864"
Nov 03 20:15:25.664746 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.664695    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a1b3d29-b647-46e1-9005-e540e937f85f-istio-data\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-wg864\" (UID: \"6a1b3d29-b647-46e1-9005-e540e937f85f\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-wg864"
Nov 03 20:15:25.664746 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.664717    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a1b3d29-b647-46e1-9005-e540e937f85f-workload-socket\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-wg864\" (UID: \"6a1b3d29-b647-46e1-9005-e540e937f85f\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-wg864"
Nov 03 20:15:25.664746 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.664738    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a1b3d29-b647-46e1-9005-e540e937f85f-istio-podinfo\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-wg864\" (UID: \"6a1b3d29-b647-46e1-9005-e540e937f85f\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-wg864"
Nov 03 20:15:25.665067 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.664791    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stxjs\" (UniqueName: \"kubernetes.io/projected/6a1b3d29-b647-46e1-9005-e540e937f85f-kube-api-access-stxjs\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-wg864\" (UID: \"6a1b3d29-b647-46e1-9005-e540e937f85f\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-wg864"
Nov 03 20:15:25.665067 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.664812    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a1b3d29-b647-46e1-9005-e540e937f85f-istio-token\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-wg864\" (UID: \"6a1b3d29-b647-46e1-9005-e540e937f85f\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-wg864"
Nov 03 20:15:25.665067 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.664848    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a1b3d29-b647-46e1-9005-e540e937f85f-istiod-ca-cert\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-wg864\" (UID: \"6a1b3d29-b647-46e1-9005-e540e937f85f\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-wg864"
Nov 03 20:15:25.665067 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.665034    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a1b3d29-b647-46e1-9005-e540e937f85f-workload-certs\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-wg864\" (UID: \"6a1b3d29-b647-46e1-9005-e540e937f85f\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-wg864"
Nov 03 20:15:25.665267 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.665182    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a1b3d29-b647-46e1-9005-e540e937f85f-workload-socket\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-wg864\" (UID: \"6a1b3d29-b647-46e1-9005-e540e937f85f\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-wg864"
Nov 03 20:15:25.665485 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.665461    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a1b3d29-b647-46e1-9005-e540e937f85f-istio-data\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-wg864\" (UID: \"6a1b3d29-b647-46e1-9005-e540e937f85f\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-wg864"
Nov 03 20:15:25.665556 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.665510    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a1b3d29-b647-46e1-9005-e540e937f85f-credential-socket\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-wg864\" (UID: \"6a1b3d29-b647-46e1-9005-e540e937f85f\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-wg864"
Nov 03 20:15:25.665877 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.665855    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a1b3d29-b647-46e1-9005-e540e937f85f-istiod-ca-cert\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-wg864\" (UID: \"6a1b3d29-b647-46e1-9005-e540e937f85f\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-wg864"
Nov 03 20:15:25.667409 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.667383    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a1b3d29-b647-46e1-9005-e540e937f85f-istio-envoy\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-wg864\" (UID: \"6a1b3d29-b647-46e1-9005-e540e937f85f\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-wg864"
Nov 03 20:15:25.667662 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.667638    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a1b3d29-b647-46e1-9005-e540e937f85f-istio-podinfo\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-wg864\" (UID: \"6a1b3d29-b647-46e1-9005-e540e937f85f\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-wg864"
Nov 03 20:15:25.684162 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.684125    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a1b3d29-b647-46e1-9005-e540e937f85f-istio-token\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-wg864\" (UID: \"6a1b3d29-b647-46e1-9005-e540e937f85f\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-wg864"
Nov 03 20:15:25.684347 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.684248    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-stxjs\" (UniqueName: \"kubernetes.io/projected/6a1b3d29-b647-46e1-9005-e540e937f85f-kube-api-access-stxjs\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-wg864\" (UID: \"6a1b3d29-b647-46e1-9005-e540e937f85f\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-wg864"
Nov 03 20:15:25.706407 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.705948    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-c5nf4"]
Nov 03 20:15:25.723187 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:15:25.723150    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a255596_5346_460f_b437_7c48d3d33d54.slice/crio-489c9b309f06d4e2be07db0840600fd8b4cb7a4376291f92672a24215a6aa049 WatchSource:0}: Error finding container 489c9b309f06d4e2be07db0840600fd8b4cb7a4376291f92672a24215a6aa049: Status 404 returned error can't find the container with id 489c9b309f06d4e2be07db0840600fd8b4cb7a4376291f92672a24215a6aa049
Nov 03 20:15:25.726091 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.726052    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:15:25.726205 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.726136    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:15:25.726253 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.726218    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:15:25.773824 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.772093    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ca8412e-c6de-4cb5-944e-ba8afd2fa791-istiod-ca-cert\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-jjjr7\" (UID: \"7ca8412e-c6de-4cb5-944e-ba8afd2fa791\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-jjjr7"
Nov 03 20:15:25.773824 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.772204    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ca8412e-c6de-4cb5-944e-ba8afd2fa791-istio-data\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-jjjr7\" (UID: \"7ca8412e-c6de-4cb5-944e-ba8afd2fa791\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-jjjr7"
Nov 03 20:15:25.773824 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.772283    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hg92x\" (UniqueName: \"kubernetes.io/projected/7ca8412e-c6de-4cb5-944e-ba8afd2fa791-kube-api-access-hg92x\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-jjjr7\" (UID: \"7ca8412e-c6de-4cb5-944e-ba8afd2fa791\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-jjjr7"
Nov 03 20:15:25.773824 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.772313    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ca8412e-c6de-4cb5-944e-ba8afd2fa791-workload-socket\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-jjjr7\" (UID: \"7ca8412e-c6de-4cb5-944e-ba8afd2fa791\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-jjjr7"
Nov 03 20:15:25.773824 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.772351    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ca8412e-c6de-4cb5-944e-ba8afd2fa791-istio-envoy\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-jjjr7\" (UID: \"7ca8412e-c6de-4cb5-944e-ba8afd2fa791\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-jjjr7"
Nov 03 20:15:25.773824 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.772374    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ca8412e-c6de-4cb5-944e-ba8afd2fa791-workload-certs\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-jjjr7\" (UID: \"7ca8412e-c6de-4cb5-944e-ba8afd2fa791\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-jjjr7"
Nov 03 20:15:25.773824 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.772399    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ca8412e-c6de-4cb5-944e-ba8afd2fa791-istio-podinfo\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-jjjr7\" (UID: \"7ca8412e-c6de-4cb5-944e-ba8afd2fa791\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-jjjr7"
Nov 03 20:15:25.773824 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.772424    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ca8412e-c6de-4cb5-944e-ba8afd2fa791-credential-socket\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-jjjr7\" (UID: \"7ca8412e-c6de-4cb5-944e-ba8afd2fa791\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-jjjr7"
Nov 03 20:15:25.773824 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.772465    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ca8412e-c6de-4cb5-944e-ba8afd2fa791-istio-token\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-jjjr7\" (UID: \"7ca8412e-c6de-4cb5-944e-ba8afd2fa791\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-jjjr7"
Nov 03 20:15:25.785490 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.785444    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-c5nf4" event={"ID":"8a255596-5346-460f-b437-7c48d3d33d54","Type":"ContainerStarted","Data":"489c9b309f06d4e2be07db0840600fd8b4cb7a4376291f92672a24215a6aa049"}
Nov 03 20:15:25.789649 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.789626    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-wg864"
Nov 03 20:15:25.841229 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.840878    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-vzfw9"]
Nov 03 20:15:25.842404 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:15:25.842268    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda739707b_9c5a_4c9b_90df_501c5dd0c98d.slice/crio-89b406441abf91875aceffa1f69a33f268137f5fcf043e76144e49fe17fb3ac1 WatchSource:0}: Error finding container 89b406441abf91875aceffa1f69a33f268137f5fcf043e76144e49fe17fb3ac1: Status 404 returned error can't find the container with id 89b406441abf91875aceffa1f69a33f268137f5fcf043e76144e49fe17fb3ac1
Nov 03 20:15:25.846631 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.845942    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:15:25.846631 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.846030    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:15:25.846631 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.846070    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:15:25.888696 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.876102    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ca8412e-c6de-4cb5-944e-ba8afd2fa791-istiod-ca-cert\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-jjjr7\" (UID: \"7ca8412e-c6de-4cb5-944e-ba8afd2fa791\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-jjjr7"
Nov 03 20:15:25.888696 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.876168    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ca8412e-c6de-4cb5-944e-ba8afd2fa791-istio-data\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-jjjr7\" (UID: \"7ca8412e-c6de-4cb5-944e-ba8afd2fa791\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-jjjr7"
Nov 03 20:15:25.888696 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.876247    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hg92x\" (UniqueName: \"kubernetes.io/projected/7ca8412e-c6de-4cb5-944e-ba8afd2fa791-kube-api-access-hg92x\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-jjjr7\" (UID: \"7ca8412e-c6de-4cb5-944e-ba8afd2fa791\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-jjjr7"
Nov 03 20:15:25.888696 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.876286    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ca8412e-c6de-4cb5-944e-ba8afd2fa791-workload-socket\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-jjjr7\" (UID: \"7ca8412e-c6de-4cb5-944e-ba8afd2fa791\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-jjjr7"
Nov 03 20:15:25.888696 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.876336    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ca8412e-c6de-4cb5-944e-ba8afd2fa791-istio-envoy\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-jjjr7\" (UID: \"7ca8412e-c6de-4cb5-944e-ba8afd2fa791\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-jjjr7"
Nov 03 20:15:25.888696 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.876366    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ca8412e-c6de-4cb5-944e-ba8afd2fa791-workload-certs\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-jjjr7\" (UID: \"7ca8412e-c6de-4cb5-944e-ba8afd2fa791\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-jjjr7"
Nov 03 20:15:25.888696 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.876403    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ca8412e-c6de-4cb5-944e-ba8afd2fa791-istio-podinfo\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-jjjr7\" (UID: \"7ca8412e-c6de-4cb5-944e-ba8afd2fa791\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-jjjr7"
Nov 03 20:15:25.888696 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.876436    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ca8412e-c6de-4cb5-944e-ba8afd2fa791-credential-socket\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-jjjr7\" (UID: \"7ca8412e-c6de-4cb5-944e-ba8afd2fa791\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-jjjr7"
Nov 03 20:15:25.888696 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.876499    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ca8412e-c6de-4cb5-944e-ba8afd2fa791-istio-token\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-jjjr7\" (UID: \"7ca8412e-c6de-4cb5-944e-ba8afd2fa791\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-jjjr7"
Nov 03 20:15:25.888696 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.877588    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ca8412e-c6de-4cb5-944e-ba8afd2fa791-istiod-ca-cert\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-jjjr7\" (UID: \"7ca8412e-c6de-4cb5-944e-ba8afd2fa791\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-jjjr7"
Nov 03 20:15:25.888696 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.888124    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ca8412e-c6de-4cb5-944e-ba8afd2fa791-workload-socket\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-jjjr7\" (UID: \"7ca8412e-c6de-4cb5-944e-ba8afd2fa791\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-jjjr7"
Nov 03 20:15:25.888696 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.888391    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ca8412e-c6de-4cb5-944e-ba8afd2fa791-credential-socket\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-jjjr7\" (UID: \"7ca8412e-c6de-4cb5-944e-ba8afd2fa791\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-jjjr7"
Nov 03 20:15:25.888696 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.888637    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ca8412e-c6de-4cb5-944e-ba8afd2fa791-workload-certs\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-jjjr7\" (UID: \"7ca8412e-c6de-4cb5-944e-ba8afd2fa791\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-jjjr7"
Nov 03 20:15:25.889487 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.888933    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ca8412e-c6de-4cb5-944e-ba8afd2fa791-istio-envoy\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-jjjr7\" (UID: \"7ca8412e-c6de-4cb5-944e-ba8afd2fa791\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-jjjr7"
Nov 03 20:15:25.900883 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.890380    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ca8412e-c6de-4cb5-944e-ba8afd2fa791-istio-podinfo\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-jjjr7\" (UID: \"7ca8412e-c6de-4cb5-944e-ba8afd2fa791\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-jjjr7"
Nov 03 20:15:25.900883 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.896866    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmqs6"]
Nov 03 20:15:25.903045 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.902981    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ca8412e-c6de-4cb5-944e-ba8afd2fa791-istio-data\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-jjjr7\" (UID: \"7ca8412e-c6de-4cb5-944e-ba8afd2fa791\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-jjjr7"
Nov 03 20:15:25.909826 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.908698    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmqs6"]
Nov 03 20:15:25.909826 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.908830    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmqs6"
Nov 03 20:15:25.909826 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.908854    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hg92x\" (UniqueName: \"kubernetes.io/projected/7ca8412e-c6de-4cb5-944e-ba8afd2fa791-kube-api-access-hg92x\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-jjjr7\" (UID: \"7ca8412e-c6de-4cb5-944e-ba8afd2fa791\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-jjjr7"
Nov 03 20:15:25.910111 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.910038    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ca8412e-c6de-4cb5-944e-ba8afd2fa791-istio-token\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-jjjr7\" (UID: \"7ca8412e-c6de-4cb5-944e-ba8afd2fa791\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-jjjr7"
Nov 03 20:15:25.927757 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.927581    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-e80203-openshift-default-dockercfg-6kpjl\""
Nov 03 20:15:25.935404 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.935380    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-e40816-openshift-default-dockercfg-rbqqp\""
Nov 03 20:15:25.941494 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.941292    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-jjjr7"
Nov 03 20:15:25.970845 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.970081    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-nz52s"]
Nov 03 20:15:25.976690 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:15:25.976657    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod974badc1_41ad_461c_8719_e1f3f836ccbc.slice/crio-a84b94fa7ae49c10475785130e3bd1d9c18b45c504b01c0d1035a5901ce6f050 WatchSource:0}: Error finding container a84b94fa7ae49c10475785130e3bd1d9c18b45c504b01c0d1035a5901ce6f050: Status 404 returned error can't find the container with id a84b94fa7ae49c10475785130e3bd1d9c18b45c504b01c0d1035a5901ce6f050
Nov 03 20:15:25.979206 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.979162    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:15:25.979295 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.979266    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:15:25.979335 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:25.979315    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:15:26.098854 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:26.081460    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/516f80b2-26e4-44d2-b97a-6e740d0b53e6-istio-podinfo\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmqs6\" (UID: \"516f80b2-26e4-44d2-b97a-6e740d0b53e6\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmqs6"
Nov 03 20:15:26.115275 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:26.115243    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/516f80b2-26e4-44d2-b97a-6e740d0b53e6-istio-envoy\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmqs6\" (UID: \"516f80b2-26e4-44d2-b97a-6e740d0b53e6\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmqs6"
Nov 03 20:15:26.115425 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:26.115300    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/516f80b2-26e4-44d2-b97a-6e740d0b53e6-istio-data\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmqs6\" (UID: \"516f80b2-26e4-44d2-b97a-6e740d0b53e6\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmqs6"
Nov 03 20:15:26.115425 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:26.115359    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/516f80b2-26e4-44d2-b97a-6e740d0b53e6-workload-socket\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmqs6\" (UID: \"516f80b2-26e4-44d2-b97a-6e740d0b53e6\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmqs6"
Nov 03 20:15:26.115506 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:26.115425    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/516f80b2-26e4-44d2-b97a-6e740d0b53e6-istiod-ca-cert\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmqs6\" (UID: \"516f80b2-26e4-44d2-b97a-6e740d0b53e6\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmqs6"
Nov 03 20:15:26.115506 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:26.115484    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/516f80b2-26e4-44d2-b97a-6e740d0b53e6-istio-token\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmqs6\" (UID: \"516f80b2-26e4-44d2-b97a-6e740d0b53e6\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmqs6"
Nov 03 20:15:26.115577 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:26.115527    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/516f80b2-26e4-44d2-b97a-6e740d0b53e6-credential-socket\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmqs6\" (UID: \"516f80b2-26e4-44d2-b97a-6e740d0b53e6\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmqs6"
Nov 03 20:15:26.115577 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:26.115554    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/516f80b2-26e4-44d2-b97a-6e740d0b53e6-workload-certs\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmqs6\" (UID: \"516f80b2-26e4-44d2-b97a-6e740d0b53e6\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmqs6"
Nov 03 20:15:26.115636 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:26.115578    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhdbd\" (UniqueName: \"kubernetes.io/projected/516f80b2-26e4-44d2-b97a-6e740d0b53e6-kube-api-access-nhdbd\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmqs6\" (UID: \"516f80b2-26e4-44d2-b97a-6e740d0b53e6\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmqs6"
Nov 03 20:15:26.115678 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:26.113154    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-wg864"]
Nov 03 20:15:26.147234 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:26.147199    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:15:26.147353 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:26.147288    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:15:26.147426 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:26.147411    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:15:26.216511 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:26.216483    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/516f80b2-26e4-44d2-b97a-6e740d0b53e6-istio-podinfo\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmqs6\" (UID: \"516f80b2-26e4-44d2-b97a-6e740d0b53e6\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmqs6"
Nov 03 20:15:26.216755 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:26.216735    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/516f80b2-26e4-44d2-b97a-6e740d0b53e6-istio-envoy\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmqs6\" (UID: \"516f80b2-26e4-44d2-b97a-6e740d0b53e6\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmqs6"
Nov 03 20:15:26.216928 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:26.216907    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/516f80b2-26e4-44d2-b97a-6e740d0b53e6-istio-data\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmqs6\" (UID: \"516f80b2-26e4-44d2-b97a-6e740d0b53e6\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmqs6"
Nov 03 20:15:26.217899 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:26.217877    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/516f80b2-26e4-44d2-b97a-6e740d0b53e6-workload-socket\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmqs6\" (UID: \"516f80b2-26e4-44d2-b97a-6e740d0b53e6\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmqs6"
Nov 03 20:15:26.218054 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:26.217488    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/516f80b2-26e4-44d2-b97a-6e740d0b53e6-workload-socket\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmqs6\" (UID: \"516f80b2-26e4-44d2-b97a-6e740d0b53e6\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmqs6"
Nov 03 20:15:26.218229 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:26.218210    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/516f80b2-26e4-44d2-b97a-6e740d0b53e6-istiod-ca-cert\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmqs6\" (UID: \"516f80b2-26e4-44d2-b97a-6e740d0b53e6\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmqs6"
Nov 03 20:15:26.218364 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:26.218351    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/516f80b2-26e4-44d2-b97a-6e740d0b53e6-istio-token\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmqs6\" (UID: \"516f80b2-26e4-44d2-b97a-6e740d0b53e6\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmqs6"
Nov 03 20:15:26.218458 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:26.218448    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/516f80b2-26e4-44d2-b97a-6e740d0b53e6-credential-socket\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmqs6\" (UID: \"516f80b2-26e4-44d2-b97a-6e740d0b53e6\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmqs6"
Nov 03 20:15:26.218533 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:26.218524    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/516f80b2-26e4-44d2-b97a-6e740d0b53e6-workload-certs\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmqs6\" (UID: \"516f80b2-26e4-44d2-b97a-6e740d0b53e6\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmqs6"
Nov 03 20:15:26.218610 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:26.218598    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhdbd\" (UniqueName: \"kubernetes.io/projected/516f80b2-26e4-44d2-b97a-6e740d0b53e6-kube-api-access-nhdbd\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmqs6\" (UID: \"516f80b2-26e4-44d2-b97a-6e740d0b53e6\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmqs6"
Nov 03 20:15:26.219179 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:26.219164    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/516f80b2-26e4-44d2-b97a-6e740d0b53e6-credential-socket\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmqs6\" (UID: \"516f80b2-26e4-44d2-b97a-6e740d0b53e6\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmqs6"
Nov 03 20:15:26.219503 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:26.219489    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/516f80b2-26e4-44d2-b97a-6e740d0b53e6-workload-certs\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmqs6\" (UID: \"516f80b2-26e4-44d2-b97a-6e740d0b53e6\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmqs6"
Nov 03 20:15:26.219703 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:26.219681    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/516f80b2-26e4-44d2-b97a-6e740d0b53e6-istiod-ca-cert\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmqs6\" (UID: \"516f80b2-26e4-44d2-b97a-6e740d0b53e6\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmqs6"
Nov 03 20:15:26.220156 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:26.220140    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/516f80b2-26e4-44d2-b97a-6e740d0b53e6-istio-podinfo\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmqs6\" (UID: \"516f80b2-26e4-44d2-b97a-6e740d0b53e6\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmqs6"
Nov 03 20:15:26.220360 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:26.220346    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/516f80b2-26e4-44d2-b97a-6e740d0b53e6-istio-data\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmqs6\" (UID: \"516f80b2-26e4-44d2-b97a-6e740d0b53e6\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmqs6"
Nov 03 20:15:26.225258 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:26.225234    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/516f80b2-26e4-44d2-b97a-6e740d0b53e6-istio-envoy\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmqs6\" (UID: \"516f80b2-26e4-44d2-b97a-6e740d0b53e6\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmqs6"
Nov 03 20:15:26.238989 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:26.238928    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/516f80b2-26e4-44d2-b97a-6e740d0b53e6-istio-token\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmqs6\" (UID: \"516f80b2-26e4-44d2-b97a-6e740d0b53e6\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmqs6"
Nov 03 20:15:26.241468 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:26.241441    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhdbd\" (UniqueName: \"kubernetes.io/projected/516f80b2-26e4-44d2-b97a-6e740d0b53e6-kube-api-access-nhdbd\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmqs6\" (UID: \"516f80b2-26e4-44d2-b97a-6e740d0b53e6\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmqs6"
Nov 03 20:15:26.243434 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:26.243413    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmqs6"
Nov 03 20:15:26.260941 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:26.260850    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-jjjr7"]
Nov 03 20:15:26.267271 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:15:26.267238    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ca8412e_c6de_4cb5_944e_ba8afd2fa791.slice/crio-4ab71c224982bd60dcd2bbc0ab9f10ef89374f3563e8b3718ebca4210186952a WatchSource:0}: Error finding container 4ab71c224982bd60dcd2bbc0ab9f10ef89374f3563e8b3718ebca4210186952a: Status 404 returned error can't find the container with id 4ab71c224982bd60dcd2bbc0ab9f10ef89374f3563e8b3718ebca4210186952a
Nov 03 20:15:26.272495 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:26.271174    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:15:26.272495 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:26.271264    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:15:26.272495 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:26.271306    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:15:26.586075 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:26.586037    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmqs6"]
Nov 03 20:15:26.591343 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:15:26.591306    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod516f80b2_26e4_44d2_b97a_6e740d0b53e6.slice/crio-2729a9a0f959f575cd063fa1673a90dd9f08b2da502aa55e8876146174afa3ab WatchSource:0}: Error finding container 2729a9a0f959f575cd063fa1673a90dd9f08b2da502aa55e8876146174afa3ab: Status 404 returned error can't find the container with id 2729a9a0f959f575cd063fa1673a90dd9f08b2da502aa55e8876146174afa3ab
Nov 03 20:15:26.594855 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:26.594811    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:15:26.594963 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:26.594911    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:15:26.594963 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:26.594956    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:15:26.794041 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:26.794004    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-jjjr7" event={"ID":"7ca8412e-c6de-4cb5-944e-ba8afd2fa791","Type":"ContainerStarted","Data":"2ea2028d5fcd730c4d6c7fb86bd31a166bf09e340eba28a808236e578dcc3ffa"}
Nov 03 20:15:26.794250 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:26.794067    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-jjjr7" event={"ID":"7ca8412e-c6de-4cb5-944e-ba8afd2fa791","Type":"ContainerStarted","Data":"4ab71c224982bd60dcd2bbc0ab9f10ef89374f3563e8b3718ebca4210186952a"}
Nov 03 20:15:26.795836 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:26.795808    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-wg864" event={"ID":"6a1b3d29-b647-46e1-9005-e540e937f85f","Type":"ContainerStarted","Data":"1e34129bff2aa0dc0d4d3ecc12b6cc9633c75b5005270c89221a9df5b61b304d"}
Nov 03 20:15:26.795973 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:26.795860    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-wg864" event={"ID":"6a1b3d29-b647-46e1-9005-e540e937f85f","Type":"ContainerStarted","Data":"10f3d7ff68442da2d65b5be8d4a9ecdf55afe10edfb109b2e2c967653062a859"}
Nov 03 20:15:26.797652 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:26.797627    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-vzfw9" event={"ID":"a739707b-9c5a-4c9b-90df-501c5dd0c98d","Type":"ContainerStarted","Data":"016c483e8a6ffb6d372c0a550482b3dc7344bd52f597fb28b1466c5c5565141a"}
Nov 03 20:15:26.797812 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:26.797683    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-vzfw9" event={"ID":"a739707b-9c5a-4c9b-90df-501c5dd0c98d","Type":"ContainerStarted","Data":"89b406441abf91875aceffa1f69a33f268137f5fcf043e76144e49fe17fb3ac1"}
Nov 03 20:15:26.800182 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:26.800152    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-c5nf4" event={"ID":"8a255596-5346-460f-b437-7c48d3d33d54","Type":"ContainerStarted","Data":"425dede10b05299e0b57616be19f61783c0772db42f2d9151301ed07b22f477f"}
Nov 03 20:15:26.801754 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:26.801728    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-nz52s" event={"ID":"974badc1-41ad-461c-8719-e1f3f836ccbc","Type":"ContainerStarted","Data":"55d50dbccd68644d558fe9e9ce5c19730dc0d11c8b11b77a7e28d9c0dfce32ee"}
Nov 03 20:15:26.801868 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:26.801792    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-nz52s" event={"ID":"974badc1-41ad-461c-8719-e1f3f836ccbc","Type":"ContainerStarted","Data":"a84b94fa7ae49c10475785130e3bd1d9c18b45c504b01c0d1035a5901ce6f050"}
Nov 03 20:15:26.803413 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:26.803387    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmqs6" event={"ID":"516f80b2-26e4-44d2-b97a-6e740d0b53e6","Type":"ContainerStarted","Data":"cee1cab22ca2c6810bad67da56bb4911028e06424d912aed8fcdd1742ea8c9ae"}
Nov 03 20:15:26.803537 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:26.803425    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmqs6" event={"ID":"516f80b2-26e4-44d2-b97a-6e740d0b53e6","Type":"ContainerStarted","Data":"2729a9a0f959f575cd063fa1673a90dd9f08b2da502aa55e8876146174afa3ab"}
Nov 03 20:15:26.835496 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:26.835447    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-jjjr7" podStartSLOduration=1.835434945 podStartE2EDuration="1.835434945s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:15:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:15:26.83480972 +0000 UTC m=+660.987471607" watchObservedRunningTime="2026-11-03 20:15:26.835434945 +0000 UTC m=+660.988096825"
Nov 03 20:15:26.875383 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:26.875328    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-c5nf4" podStartSLOduration=1.875309279 podStartE2EDuration="1.875309279s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:15:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:15:26.874391322 +0000 UTC m=+661.027053213" watchObservedRunningTime="2026-11-03 20:15:26.875309279 +0000 UTC m=+661.027971159"
Nov 03 20:15:26.919196 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:26.919066    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-wg864" podStartSLOduration=1.9190502569999999 podStartE2EDuration="1.919050257s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:15:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:15:26.918082978 +0000 UTC m=+661.070744866" watchObservedRunningTime="2026-11-03 20:15:26.919050257 +0000 UTC m=+661.071712147"
Nov 03 20:15:26.942165 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:26.942126    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-jjjr7"
Nov 03 20:15:26.960558 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:26.960497    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-vzfw9" podStartSLOduration=1.960477399 podStartE2EDuration="1.960477399s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:15:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:15:26.959153322 +0000 UTC m=+661.111815223" watchObservedRunningTime="2026-11-03 20:15:26.960477399 +0000 UTC m=+661.113139280"
Nov 03 20:15:27.001509 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:27.001439    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-nz52s" podStartSLOduration=2.001419774 podStartE2EDuration="2.001419774s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:15:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:15:27.00122723 +0000 UTC m=+661.153889121" watchObservedRunningTime="2026-11-03 20:15:27.001419774 +0000 UTC m=+661.154081663"
Nov 03 20:15:27.043349 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:27.043282    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmqs6" podStartSLOduration=2.043267643 podStartE2EDuration="2.043267643s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:15:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:15:27.042207146 +0000 UTC m=+661.194869033" watchObservedRunningTime="2026-11-03 20:15:27.043267643 +0000 UTC m=+661.195929528"
Nov 03 20:15:27.234327 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:15:27.234280    5570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8bwhz" containerName="extractor"
Nov 03 20:15:27.244599 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:27.244568    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmqs6"
Nov 03 20:15:27.250092 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:27.250064    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmqs6"
Nov 03 20:15:27.383506 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:27.383465    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-c5nf4"
Nov 03 20:15:27.388983 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:27.388951    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-c5nf4"
Nov 03 20:15:27.522688 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:27.522605    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-vzfw9"
Nov 03 20:15:27.528152 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:27.528119    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-vzfw9"
Nov 03 20:15:27.647375 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:27.647334    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-nz52s"
Nov 03 20:15:27.652729 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:27.652698    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-nz52s"
Nov 03 20:15:27.790650 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:27.790549    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-wg864"
Nov 03 20:15:27.795965 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:27.795906    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-wg864"
Nov 03 20:15:27.810972 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:27.810938    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-vzfw9"
Nov 03 20:15:27.811208 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:27.811184    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-nz52s"
Nov 03 20:15:27.811208 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:27.811223    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-wg864"
Nov 03 20:15:27.811419 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:27.811251    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmqs6"
Nov 03 20:15:27.812128 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:27.812108    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-c5nf4"
Nov 03 20:15:27.812764 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:27.812745    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-wg864"
Nov 03 20:15:27.812874 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:27.812825    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-nz52s"
Nov 03 20:15:27.812874 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:27.812859    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-vzfw9"
Nov 03 20:15:27.812976 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:27.812920    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-c5nf4"
Nov 03 20:15:27.813261 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:27.813246    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmqs6"
Nov 03 20:15:27.947086 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:27.947056    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-jjjr7"
Nov 03 20:15:28.816066 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:28.816038    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-jjjr7"
Nov 03 20:15:28.816572 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:28.816550    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-jjjr7"
Nov 03 20:15:36.678561 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:36.678525    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g69f8"]
Nov 03 20:15:37.995115 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:37.995075    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-6fc748-6b8d84dfb-4tvlr"]
Nov 03 20:15:38.003663 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:38.003635    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-6fc748-6b8d84dfb-4tvlr"
Nov 03 20:15:38.016423 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:38.016394    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-6fc748-6b8d84dfb-4tvlr"]
Nov 03 20:15:38.032996 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:38.026971    5570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-6fc748\""
Nov 03 20:15:38.032996 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:38.027106    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-6fc748-dockercfg-86btt\""
Nov 03 20:15:38.060423 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:38.060376    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lg2h\" (UniqueName: \"kubernetes.io/projected/75db83cf-ba54-414a-9041-094aa630a60a-kube-api-access-7lg2h\") pod \"payload-processing-e2e-worker-w1-6fc748-6b8d84dfb-4tvlr\" (UID: \"75db83cf-ba54-414a-9041-094aa630a60a\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-6fc748-6b8d84dfb-4tvlr"
Nov 03 20:15:38.060628 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:38.060569    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75db83cf-ba54-414a-9041-094aa630a60a-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-6fc748-6b8d84dfb-4tvlr\" (UID: \"75db83cf-ba54-414a-9041-094aa630a60a\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-6fc748-6b8d84dfb-4tvlr"
Nov 03 20:15:38.166483 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:38.166432    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lg2h\" (UniqueName: \"kubernetes.io/projected/75db83cf-ba54-414a-9041-094aa630a60a-kube-api-access-7lg2h\") pod \"payload-processing-e2e-worker-w1-6fc748-6b8d84dfb-4tvlr\" (UID: \"75db83cf-ba54-414a-9041-094aa630a60a\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-6fc748-6b8d84dfb-4tvlr"
Nov 03 20:15:38.166674 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:38.166576    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75db83cf-ba54-414a-9041-094aa630a60a-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-6fc748-6b8d84dfb-4tvlr\" (UID: \"75db83cf-ba54-414a-9041-094aa630a60a\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-6fc748-6b8d84dfb-4tvlr"
Nov 03 20:15:38.167750 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:38.167552    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75db83cf-ba54-414a-9041-094aa630a60a-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-6fc748-6b8d84dfb-4tvlr\" (UID: \"75db83cf-ba54-414a-9041-094aa630a60a\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-6fc748-6b8d84dfb-4tvlr"
Nov 03 20:15:38.191424 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:38.191388    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lg2h\" (UniqueName: \"kubernetes.io/projected/75db83cf-ba54-414a-9041-094aa630a60a-kube-api-access-7lg2h\") pod \"payload-processing-e2e-worker-w1-6fc748-6b8d84dfb-4tvlr\" (UID: \"75db83cf-ba54-414a-9041-094aa630a60a\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-6fc748-6b8d84dfb-4tvlr"
Nov 03 20:15:38.317335 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:38.317244    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-6fc748-6b8d84dfb-4tvlr"
Nov 03 20:15:38.626891 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:38.626391    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-6fc748-6b8d84dfb-4tvlr"]
Nov 03 20:15:38.879148 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:38.879056    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-6fc748-6b8d84dfb-4tvlr" event={"ID":"75db83cf-ba54-414a-9041-094aa630a60a","Type":"ContainerStarted","Data":"a04523c80e655466a5e50ecf8433a76315e6c506f131466c9dd0befbe1207282"}
Nov 03 20:15:39.017349 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:39.017306    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-e40816-548465ddf5-mzdj7"]
Nov 03 20:15:39.022324 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:39.022290    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e40816-548465ddf5-mzdj7"
Nov 03 20:15:39.038750 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:39.038716    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-e40816-548465ddf5-mzdj7"]
Nov 03 20:15:39.077416 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:39.077375    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94a03b52-b293-4475-8b23-fbbbae06fb26-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-e40816-548465ddf5-mzdj7\" (UID: \"94a03b52-b293-4475-8b23-fbbbae06fb26\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e40816-548465ddf5-mzdj7"
Nov 03 20:15:39.077620 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:39.077576    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nx8k9\" (UniqueName: \"kubernetes.io/projected/94a03b52-b293-4475-8b23-fbbbae06fb26-kube-api-access-nx8k9\") pod \"payload-pre-processing-e2e-worker-w0-e40816-548465ddf5-mzdj7\" (UID: \"94a03b52-b293-4475-8b23-fbbbae06fb26\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e40816-548465ddf5-mzdj7"
Nov 03 20:15:39.078544 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:39.078522    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-e40816-dockercfg-zbq6q\""
Nov 03 20:15:39.078702 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:39.078630    5570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-e40816\""
Nov 03 20:15:39.179142 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:39.179048    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nx8k9\" (UniqueName: \"kubernetes.io/projected/94a03b52-b293-4475-8b23-fbbbae06fb26-kube-api-access-nx8k9\") pod \"payload-pre-processing-e2e-worker-w0-e40816-548465ddf5-mzdj7\" (UID: \"94a03b52-b293-4475-8b23-fbbbae06fb26\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e40816-548465ddf5-mzdj7"
Nov 03 20:15:39.179331 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:39.179156    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94a03b52-b293-4475-8b23-fbbbae06fb26-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-e40816-548465ddf5-mzdj7\" (UID: \"94a03b52-b293-4475-8b23-fbbbae06fb26\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e40816-548465ddf5-mzdj7"
Nov 03 20:15:39.180173 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:39.180143    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94a03b52-b293-4475-8b23-fbbbae06fb26-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-e40816-548465ddf5-mzdj7\" (UID: \"94a03b52-b293-4475-8b23-fbbbae06fb26\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e40816-548465ddf5-mzdj7"
Nov 03 20:15:39.202482 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:39.202439    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nx8k9\" (UniqueName: \"kubernetes.io/projected/94a03b52-b293-4475-8b23-fbbbae06fb26-kube-api-access-nx8k9\") pod \"payload-pre-processing-e2e-worker-w0-e40816-548465ddf5-mzdj7\" (UID: \"94a03b52-b293-4475-8b23-fbbbae06fb26\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e40816-548465ddf5-mzdj7"
Nov 03 20:15:39.348517 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:39.348465    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e40816-548465ddf5-mzdj7"
Nov 03 20:15:39.685739 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:39.685708    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-e40816-548465ddf5-mzdj7"]
Nov 03 20:15:39.689035 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:15:39.689003    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94a03b52_b293_4475_8b23_fbbbae06fb26.slice/crio-9b39545510de8f1b362e4a18192b5b839e958609ffb23ae806f52080b7a9f6b4 WatchSource:0}: Error finding container 9b39545510de8f1b362e4a18192b5b839e958609ffb23ae806f52080b7a9f6b4: Status 404 returned error can't find the container with id 9b39545510de8f1b362e4a18192b5b839e958609ffb23ae806f52080b7a9f6b4
Nov 03 20:15:39.889286 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:39.889240    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e40816-548465ddf5-mzdj7" event={"ID":"94a03b52-b293-4475-8b23-fbbbae06fb26","Type":"ContainerStarted","Data":"9b39545510de8f1b362e4a18192b5b839e958609ffb23ae806f52080b7a9f6b4"}
Nov 03 20:15:41.323324 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:41.323074    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g69f8"]
Nov 03 20:15:41.906698 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:41.906660    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-6fc748-6b8d84dfb-4tvlr" event={"ID":"75db83cf-ba54-414a-9041-094aa630a60a","Type":"ContainerStarted","Data":"a9afc22fe0d84c757e6fa956bb7652781c88cdd767dc124612294b26216b2019"}
Nov 03 20:15:41.906945 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:41.906729    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-6fc748-6b8d84dfb-4tvlr"
Nov 03 20:15:41.908611 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:41.908573    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e40816-548465ddf5-mzdj7" event={"ID":"94a03b52-b293-4475-8b23-fbbbae06fb26","Type":"ContainerStarted","Data":"e8cc79d0834cf7d7b1ed1f4463ba238eecea9b687962905be6718222cd5de000"}
Nov 03 20:15:41.908739 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:41.908641    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e40816-548465ddf5-mzdj7"
Nov 03 20:15:41.960009 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:41.956081    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-6fc748-6b8d84dfb-4tvlr" podStartSLOduration=2.561676411 podStartE2EDuration="4.956059935s" totalImagesPullingTime="2.394383524s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:15:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:15:41.954303226 +0000 UTC m=+676.106965116" watchObservedRunningTime="2026-11-03 20:15:41.956059935 +0000 UTC m=+676.108721823"
Nov 03 20:15:42.002449 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:42.002381    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e40816-548465ddf5-mzdj7" podStartSLOduration=2.572468139 podStartE2EDuration="4.002365176s" totalImagesPullingTime="1.429897037s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:15:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:15:42.000958708 +0000 UTC m=+676.153620586" watchObservedRunningTime="2026-11-03 20:15:42.002365176 +0000 UTC m=+676.155027055"
Nov 03 20:15:45.515802 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:45.510905    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-9e1bcb-c7dccc559-tbldl"]
Nov 03 20:15:45.523511 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:45.523485    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-9e1bcb-c7dccc559-tbldl"
Nov 03 20:15:45.534351 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:45.534319    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-9e1bcb-c7dccc559-tbldl"]
Nov 03 20:15:45.545380 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:45.545347    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 03 20:15:45.545571 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:45.545437    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w6-9e1bcb-serving-cert\""
Nov 03 20:15:45.545571 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:45.545560    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-49bxb\""
Nov 03 20:15:45.545704 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:45.545670    5570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 20:15:45.570797 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:45.568976    5570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 20:15:45.717942 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:45.705091    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvm78\" (UniqueName: \"kubernetes.io/projected/cd3d374d-4721-4ccc-8283-dccefe85be90-kube-api-access-jvm78\") pod \"maas-api-e2e-worker-w6-9e1bcb-c7dccc559-tbldl\" (UID: \"cd3d374d-4721-4ccc-8283-dccefe85be90\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-9e1bcb-c7dccc559-tbldl"
Nov 03 20:15:45.717942 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:45.717936    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd3d374d-4721-4ccc-8283-dccefe85be90-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-9e1bcb-c7dccc559-tbldl\" (UID: \"cd3d374d-4721-4ccc-8283-dccefe85be90\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-9e1bcb-c7dccc559-tbldl"
Nov 03 20:15:45.718234 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:45.718008    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd3d374d-4721-4ccc-8283-dccefe85be90-metrics-certs\") pod \"maas-api-e2e-worker-w6-9e1bcb-c7dccc559-tbldl\" (UID: \"cd3d374d-4721-4ccc-8283-dccefe85be90\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-9e1bcb-c7dccc559-tbldl"
Nov 03 20:15:45.718234 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:45.718058    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd3d374d-4721-4ccc-8283-dccefe85be90-maas-api-tls\") pod \"maas-api-e2e-worker-w6-9e1bcb-c7dccc559-tbldl\" (UID: \"cd3d374d-4721-4ccc-8283-dccefe85be90\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-9e1bcb-c7dccc559-tbldl"
Nov 03 20:15:45.832620 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:45.832508    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvm78\" (UniqueName: \"kubernetes.io/projected/cd3d374d-4721-4ccc-8283-dccefe85be90-kube-api-access-jvm78\") pod \"maas-api-e2e-worker-w6-9e1bcb-c7dccc559-tbldl\" (UID: \"cd3d374d-4721-4ccc-8283-dccefe85be90\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-9e1bcb-c7dccc559-tbldl"
Nov 03 20:15:45.832960 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:45.832934    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd3d374d-4721-4ccc-8283-dccefe85be90-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-9e1bcb-c7dccc559-tbldl\" (UID: \"cd3d374d-4721-4ccc-8283-dccefe85be90\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-9e1bcb-c7dccc559-tbldl"
Nov 03 20:15:45.833143 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:45.833126    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd3d374d-4721-4ccc-8283-dccefe85be90-metrics-certs\") pod \"maas-api-e2e-worker-w6-9e1bcb-c7dccc559-tbldl\" (UID: \"cd3d374d-4721-4ccc-8283-dccefe85be90\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-9e1bcb-c7dccc559-tbldl"
Nov 03 20:15:45.833284 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:45.833269    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd3d374d-4721-4ccc-8283-dccefe85be90-maas-api-tls\") pod \"maas-api-e2e-worker-w6-9e1bcb-c7dccc559-tbldl\" (UID: \"cd3d374d-4721-4ccc-8283-dccefe85be90\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-9e1bcb-c7dccc559-tbldl"
Nov 03 20:15:45.835124 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:45.835098    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd3d374d-4721-4ccc-8283-dccefe85be90-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-9e1bcb-c7dccc559-tbldl\" (UID: \"cd3d374d-4721-4ccc-8283-dccefe85be90\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-9e1bcb-c7dccc559-tbldl"
Nov 03 20:15:45.857583 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:45.857552    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvm78\" (UniqueName: \"kubernetes.io/projected/cd3d374d-4721-4ccc-8283-dccefe85be90-kube-api-access-jvm78\") pod \"maas-api-e2e-worker-w6-9e1bcb-c7dccc559-tbldl\" (UID: \"cd3d374d-4721-4ccc-8283-dccefe85be90\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-9e1bcb-c7dccc559-tbldl"
Nov 03 20:15:45.874388 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:45.874354    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd3d374d-4721-4ccc-8283-dccefe85be90-metrics-certs\") pod \"maas-api-e2e-worker-w6-9e1bcb-c7dccc559-tbldl\" (UID: \"cd3d374d-4721-4ccc-8283-dccefe85be90\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-9e1bcb-c7dccc559-tbldl"
Nov 03 20:15:45.886431 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:45.886082    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd3d374d-4721-4ccc-8283-dccefe85be90-maas-api-tls\") pod \"maas-api-e2e-worker-w6-9e1bcb-c7dccc559-tbldl\" (UID: \"cd3d374d-4721-4ccc-8283-dccefe85be90\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-9e1bcb-c7dccc559-tbldl"
Nov 03 20:15:46.062796 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:46.058834    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g69f8"]
Nov 03 20:15:46.148292 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:46.147965    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-9e1bcb-c7dccc559-tbldl"
Nov 03 20:15:46.811861 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:46.810079    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-9e1bcb-c7dccc559-tbldl"]
Nov 03 20:15:47.033681 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:47.033641    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-9e1bcb-c7dccc559-tbldl" event={"ID":"cd3d374d-4721-4ccc-8283-dccefe85be90","Type":"ContainerStarted","Data":"2c08e6b58cedb3b90322891e3bb17564d3c85a561c90615151bed415a8d6ed31"}
Nov 03 20:15:51.852042 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:51.851196    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g69f8"]
Nov 03 20:15:52.992131 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:52.991037    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e40816-548465ddf5-mzdj7"
Nov 03 20:15:52.993501 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:52.992817    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-6fc748-6b8d84dfb-4tvlr"
Nov 03 20:15:55.671823 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:55.671121    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-2zvr4"]
Nov 03 20:15:55.729811 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:55.729728    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-2zvr4"]
Nov 03 20:15:55.731706 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:55.730985    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-2zvr4"
Nov 03 20:15:55.815027 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:55.814974    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8edff27-9b0c-40e5-9a72-4a85d7efd3e9-istio-token\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-2zvr4\" (UID: \"c8edff27-9b0c-40e5-9a72-4a85d7efd3e9\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-2zvr4"
Nov 03 20:15:55.815232 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:55.815040    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8edff27-9b0c-40e5-9a72-4a85d7efd3e9-credential-socket\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-2zvr4\" (UID: \"c8edff27-9b0c-40e5-9a72-4a85d7efd3e9\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-2zvr4"
Nov 03 20:15:55.815232 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:55.815111    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8edff27-9b0c-40e5-9a72-4a85d7efd3e9-istio-envoy\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-2zvr4\" (UID: \"c8edff27-9b0c-40e5-9a72-4a85d7efd3e9\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-2zvr4"
Nov 03 20:15:55.815232 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:55.815146    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8edff27-9b0c-40e5-9a72-4a85d7efd3e9-workload-certs\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-2zvr4\" (UID: \"c8edff27-9b0c-40e5-9a72-4a85d7efd3e9\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-2zvr4"
Nov 03 20:15:55.815232 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:55.815186    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8edff27-9b0c-40e5-9a72-4a85d7efd3e9-istio-data\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-2zvr4\" (UID: \"c8edff27-9b0c-40e5-9a72-4a85d7efd3e9\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-2zvr4"
Nov 03 20:15:55.815232 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:55.815219    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfv7d\" (UniqueName: \"kubernetes.io/projected/c8edff27-9b0c-40e5-9a72-4a85d7efd3e9-kube-api-access-dfv7d\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-2zvr4\" (UID: \"c8edff27-9b0c-40e5-9a72-4a85d7efd3e9\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-2zvr4"
Nov 03 20:15:55.815534 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:55.815259    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8edff27-9b0c-40e5-9a72-4a85d7efd3e9-istio-podinfo\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-2zvr4\" (UID: \"c8edff27-9b0c-40e5-9a72-4a85d7efd3e9\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-2zvr4"
Nov 03 20:15:55.815534 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:55.815285    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8edff27-9b0c-40e5-9a72-4a85d7efd3e9-workload-socket\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-2zvr4\" (UID: \"c8edff27-9b0c-40e5-9a72-4a85d7efd3e9\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-2zvr4"
Nov 03 20:15:55.815534 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:55.815385    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8edff27-9b0c-40e5-9a72-4a85d7efd3e9-istiod-ca-cert\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-2zvr4\" (UID: \"c8edff27-9b0c-40e5-9a72-4a85d7efd3e9\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-2zvr4"
Nov 03 20:15:55.919034 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:55.916407    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8edff27-9b0c-40e5-9a72-4a85d7efd3e9-istiod-ca-cert\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-2zvr4\" (UID: \"c8edff27-9b0c-40e5-9a72-4a85d7efd3e9\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-2zvr4"
Nov 03 20:15:55.919034 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:55.916506    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8edff27-9b0c-40e5-9a72-4a85d7efd3e9-istio-token\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-2zvr4\" (UID: \"c8edff27-9b0c-40e5-9a72-4a85d7efd3e9\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-2zvr4"
Nov 03 20:15:55.919034 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:55.916544    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8edff27-9b0c-40e5-9a72-4a85d7efd3e9-credential-socket\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-2zvr4\" (UID: \"c8edff27-9b0c-40e5-9a72-4a85d7efd3e9\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-2zvr4"
Nov 03 20:15:55.919034 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:55.916614    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8edff27-9b0c-40e5-9a72-4a85d7efd3e9-istio-envoy\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-2zvr4\" (UID: \"c8edff27-9b0c-40e5-9a72-4a85d7efd3e9\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-2zvr4"
Nov 03 20:15:55.919034 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:55.916652    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8edff27-9b0c-40e5-9a72-4a85d7efd3e9-workload-certs\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-2zvr4\" (UID: \"c8edff27-9b0c-40e5-9a72-4a85d7efd3e9\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-2zvr4"
Nov 03 20:15:55.919034 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:55.916687    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8edff27-9b0c-40e5-9a72-4a85d7efd3e9-istio-data\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-2zvr4\" (UID: \"c8edff27-9b0c-40e5-9a72-4a85d7efd3e9\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-2zvr4"
Nov 03 20:15:55.919034 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:55.916722    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfv7d\" (UniqueName: \"kubernetes.io/projected/c8edff27-9b0c-40e5-9a72-4a85d7efd3e9-kube-api-access-dfv7d\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-2zvr4\" (UID: \"c8edff27-9b0c-40e5-9a72-4a85d7efd3e9\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-2zvr4"
Nov 03 20:15:55.919034 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:55.916762    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8edff27-9b0c-40e5-9a72-4a85d7efd3e9-istio-podinfo\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-2zvr4\" (UID: \"c8edff27-9b0c-40e5-9a72-4a85d7efd3e9\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-2zvr4"
Nov 03 20:15:55.919034 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:55.916804    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8edff27-9b0c-40e5-9a72-4a85d7efd3e9-workload-socket\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-2zvr4\" (UID: \"c8edff27-9b0c-40e5-9a72-4a85d7efd3e9\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-2zvr4"
Nov 03 20:15:55.919034 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:55.917389    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8edff27-9b0c-40e5-9a72-4a85d7efd3e9-workload-socket\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-2zvr4\" (UID: \"c8edff27-9b0c-40e5-9a72-4a85d7efd3e9\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-2zvr4"
Nov 03 20:15:55.919034 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:55.918103    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8edff27-9b0c-40e5-9a72-4a85d7efd3e9-istiod-ca-cert\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-2zvr4\" (UID: \"c8edff27-9b0c-40e5-9a72-4a85d7efd3e9\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-2zvr4"
Nov 03 20:15:55.919034 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:55.918691    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8edff27-9b0c-40e5-9a72-4a85d7efd3e9-credential-socket\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-2zvr4\" (UID: \"c8edff27-9b0c-40e5-9a72-4a85d7efd3e9\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-2zvr4"
Nov 03 20:15:55.919899 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:55.919070    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8edff27-9b0c-40e5-9a72-4a85d7efd3e9-istio-data\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-2zvr4\" (UID: \"c8edff27-9b0c-40e5-9a72-4a85d7efd3e9\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-2zvr4"
Nov 03 20:15:55.919899 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:55.919563    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8edff27-9b0c-40e5-9a72-4a85d7efd3e9-workload-certs\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-2zvr4\" (UID: \"c8edff27-9b0c-40e5-9a72-4a85d7efd3e9\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-2zvr4"
Nov 03 20:15:55.921709 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:55.921656    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8edff27-9b0c-40e5-9a72-4a85d7efd3e9-istio-envoy\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-2zvr4\" (UID: \"c8edff27-9b0c-40e5-9a72-4a85d7efd3e9\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-2zvr4"
Nov 03 20:15:55.953832 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:55.948993    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8edff27-9b0c-40e5-9a72-4a85d7efd3e9-istio-podinfo\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-2zvr4\" (UID: \"c8edff27-9b0c-40e5-9a72-4a85d7efd3e9\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-2zvr4"
Nov 03 20:15:55.988866 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:55.988792    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8edff27-9b0c-40e5-9a72-4a85d7efd3e9-istio-token\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-2zvr4\" (UID: \"c8edff27-9b0c-40e5-9a72-4a85d7efd3e9\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-2zvr4"
Nov 03 20:15:55.996201 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:55.996134    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfv7d\" (UniqueName: \"kubernetes.io/projected/c8edff27-9b0c-40e5-9a72-4a85d7efd3e9-kube-api-access-dfv7d\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-2zvr4\" (UID: \"c8edff27-9b0c-40e5-9a72-4a85d7efd3e9\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-2zvr4"
Nov 03 20:15:56.142364 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:56.142069    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-2zvr4"
Nov 03 20:15:56.632746 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:56.632710    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-2zvr4"]
Nov 03 20:15:56.772118 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:15:56.772069    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8edff27_9b0c_40e5_9a72_4a85d7efd3e9.slice/crio-7359d0787ecdb6855a786aea51b75b85f1d2ce53fdaa9a4428988b88c0095fe5 WatchSource:0}: Error finding container 7359d0787ecdb6855a786aea51b75b85f1d2ce53fdaa9a4428988b88c0095fe5: Status 404 returned error can't find the container with id 7359d0787ecdb6855a786aea51b75b85f1d2ce53fdaa9a4428988b88c0095fe5
Nov 03 20:15:56.778037 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:56.777867    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:15:56.778037 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:56.777961    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:15:56.778037 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:56.777994    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:15:57.269224 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:15:57.268989    5570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8bwhz" containerName="extractor"
Nov 03 20:15:57.514751 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:57.494934    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-2zvr4" event={"ID":"c8edff27-9b0c-40e5-9a72-4a85d7efd3e9","Type":"ContainerStarted","Data":"511ac9d6b28ec82c5a67a5a17fb5cfd8ca9042b566e6369939735547e1f06b1f"}
Nov 03 20:15:57.514751 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:57.494996    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-2zvr4" event={"ID":"c8edff27-9b0c-40e5-9a72-4a85d7efd3e9","Type":"ContainerStarted","Data":"7359d0787ecdb6855a786aea51b75b85f1d2ce53fdaa9a4428988b88c0095fe5"}
Nov 03 20:15:59.269078 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:58.146456    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-2zvr4"
Nov 03 20:15:59.269078 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:58.148482    5570 patch_prober.go:29] interesting pod/e2e-worker-w1-6fc748-openshift-default-665546959f-2zvr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body=
Nov 03 20:15:59.269078 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:58.148531    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-2zvr4" podUID="c8edff27-9b0c-40e5-9a72-4a85d7efd3e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused"
Nov 03 20:15:59.269078 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:58.503711    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-9e1bcb-c7dccc559-tbldl" event={"ID":"cd3d374d-4721-4ccc-8283-dccefe85be90","Type":"ContainerStarted","Data":"1d8effea7f0c0f28c9dc1aa5c2f7c7b5d37e0207f9f40937e2246c1697181af4"}
Nov 03 20:15:59.269078 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:58.503841    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-9e1bcb-c7dccc559-tbldl"
Nov 03 20:15:59.269078 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:58.546423    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-2zvr4" podStartSLOduration=3.546401494 podStartE2EDuration="3.546401494s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:15:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:15:57.570797847 +0000 UTC m=+691.723459735" watchObservedRunningTime="2026-11-03 20:15:58.546401494 +0000 UTC m=+692.699063382"
Nov 03 20:15:59.269078 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:58.548483    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-9e1bcb-c7dccc559-tbldl" podStartSLOduration=2.509154505 podStartE2EDuration="13.548465472s" totalImagesPullingTime="11.039310967s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:15:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:15:58.546222196 +0000 UTC m=+692.698884084" watchObservedRunningTime="2026-11-03 20:15:58.548465472 +0000 UTC m=+692.701127362"
Nov 03 20:15:59.269078 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:58.767924    5570 patch_prober.go:29] interesting pod/e2e-worker-w1-6fc748-openshift-default-665546959f-wg864 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": read tcp 10.134.0.2:42244->10.134.0.41:15021: read: connection reset by peer" start-of-body=
Nov 03 20:15:59.269078 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:58.767988    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-wg864" podUID="6a1b3d29-b647-46e1-9005-e540e937f85f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": read tcp 10.134.0.2:42244->10.134.0.41:15021: read: connection reset by peer"
Nov 03 20:15:59.269078 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:58.854935    5570 patch_prober.go:29] interesting pod/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmqs6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:15:59.269078 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:58.855001    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmqs6" podUID="516f80b2-26e4-44d2-b97a-6e740d0b53e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:15:59.269078 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:58.855875    5570 patch_prober.go:29] interesting pod/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-c5nf4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:15:59.269078 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:58.855928    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-c5nf4" podUID="8a255596-5346-460f-b437-7c48d3d33d54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:15:59.269078 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:58.856060    5570 patch_prober.go:29] interesting pod/e2e-worker-w2-9973d9-openshift-default-ffb6475-nz52s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:15:59.269078 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:58.856086    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-nz52s" podUID="974badc1-41ad-461c-8719-e1f3f836ccbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:15:59.269078 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:59.144183    5570 patch_prober.go:29] interesting pod/e2e-worker-w1-6fc748-openshift-default-665546959f-2zvr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body=
Nov 03 20:15:59.269078 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:59.144246    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-2zvr4" podUID="c8edff27-9b0c-40e5-9a72-4a85d7efd3e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused"
Nov 03 20:15:59.606330 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:59.511198    5570 generic.go:356] "Generic (PLEG): container finished" podID="6a1b3d29-b647-46e1-9005-e540e937f85f" containerID="1e34129bff2aa0dc0d4d3ecc12b6cc9633c75b5005270c89221a9df5b61b304d" exitCode=0
Nov 03 20:15:59.606330 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:59.511266    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-wg864" event={"ID":"6a1b3d29-b647-46e1-9005-e540e937f85f","Type":"ContainerDied","Data":"1e34129bff2aa0dc0d4d3ecc12b6cc9633c75b5005270c89221a9df5b61b304d"}
Nov 03 20:15:59.606330 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:59.511754    5570 scope.go:122] "RemoveContainer" containerID="1e34129bff2aa0dc0d4d3ecc12b6cc9633c75b5005270c89221a9df5b61b304d"
Nov 03 20:15:59.606330 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:59.513238    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:15:59.606330 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:59.513305    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:15:59.606330 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:59.513337    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:15:59.606330 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:59.513348    5570 generic.go:356] "Generic (PLEG): container finished" podID="516f80b2-26e4-44d2-b97a-6e740d0b53e6" containerID="cee1cab22ca2c6810bad67da56bb4911028e06424d912aed8fcdd1742ea8c9ae" exitCode=0
Nov 03 20:15:59.606330 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:59.513543    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmqs6" event={"ID":"516f80b2-26e4-44d2-b97a-6e740d0b53e6","Type":"ContainerDied","Data":"cee1cab22ca2c6810bad67da56bb4911028e06424d912aed8fcdd1742ea8c9ae"}
Nov 03 20:15:59.606330 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:59.513995    5570 scope.go:122] "RemoveContainer" containerID="cee1cab22ca2c6810bad67da56bb4911028e06424d912aed8fcdd1742ea8c9ae"
Nov 03 20:15:59.606330 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:59.515234    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:15:59.606330 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:59.515311    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:15:59.606330 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:59.515360    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:15:59.859401 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:59.858916    5570 patch_prober.go:29] interesting pod/e2e-worker-w0-e40816-openshift-default-744c57967c-jjjr7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:15:59.859401 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:15:59.858988    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-jjjr7" podUID="7ca8412e-c6de-4cb5-944e-ba8afd2fa791" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:16:00.143290 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:00.143193    5570 patch_prober.go:29] interesting pod/e2e-worker-w1-6fc748-openshift-default-665546959f-2zvr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body=
Nov 03 20:16:00.143290 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:00.143265    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-2zvr4" podUID="c8edff27-9b0c-40e5-9a72-4a85d7efd3e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused"
Nov 03 20:16:00.568831 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:00.559994    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-wg864" event={"ID":"6a1b3d29-b647-46e1-9005-e540e937f85f","Type":"ContainerStarted","Data":"c112fdc1a8a3d21c59453de2ff24628f1b064fd91b46b2f5e53085450bd216c0"}
Nov 03 20:16:00.582636 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:00.581530    5570 generic.go:356] "Generic (PLEG): container finished" podID="974badc1-41ad-461c-8719-e1f3f836ccbc" containerID="55d50dbccd68644d558fe9e9ce5c19730dc0d11c8b11b77a7e28d9c0dfce32ee" exitCode=0
Nov 03 20:16:00.582636 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:00.581637    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-nz52s" event={"ID":"974badc1-41ad-461c-8719-e1f3f836ccbc","Type":"ContainerDied","Data":"55d50dbccd68644d558fe9e9ce5c19730dc0d11c8b11b77a7e28d9c0dfce32ee"}
Nov 03 20:16:00.582636 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:00.582143    5570 scope.go:122] "RemoveContainer" containerID="55d50dbccd68644d558fe9e9ce5c19730dc0d11c8b11b77a7e28d9c0dfce32ee"
Nov 03 20:16:00.601340 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:00.600906    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:16:00.601340 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:00.601000    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:16:00.601340 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:00.601050    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:16:00.733549 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:00.733279    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmqs6" event={"ID":"516f80b2-26e4-44d2-b97a-6e740d0b53e6","Type":"ContainerStarted","Data":"5ade2867118dee88a4e2a26316583dc95b4296cf59fd952962d9b69fc346409c"}
Nov 03 20:16:00.790614 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:00.790527    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-wg864"
Nov 03 20:16:00.801744 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:00.790900    5570 patch_prober.go:29] interesting pod/e2e-worker-w1-6fc748-openshift-default-665546959f-wg864 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Nov 03 20:16:00.801744 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:00.790954    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-wg864" podUID="6a1b3d29-b647-46e1-9005-e540e937f85f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Nov 03 20:16:01.200860 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:01.195440    5570 patch_prober.go:29] interesting pod/e2e-worker-w1-6fc748-openshift-default-665546959f-2zvr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body=
Nov 03 20:16:01.200860 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:01.195497    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-2zvr4" podUID="c8edff27-9b0c-40e5-9a72-4a85d7efd3e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused"
Nov 03 20:16:01.252634 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:01.252548    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmqs6"
Nov 03 20:16:01.253283 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:01.252948    5570 patch_prober.go:29] interesting pod/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmqs6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body=
Nov 03 20:16:01.253283 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:01.253002    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmqs6" podUID="516f80b2-26e4-44d2-b97a-6e740d0b53e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused"
Nov 03 20:16:01.785800 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:01.777464    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-nz52s" event={"ID":"974badc1-41ad-461c-8719-e1f3f836ccbc","Type":"ContainerStarted","Data":"f907ae94459b2d9e4d50bb2b9a2f8edfa7b72411d9525c0dd6685e0bad46decd"}
Nov 03 20:16:01.797801 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:01.790542    5570 patch_prober.go:29] interesting pod/e2e-worker-w1-6fc748-openshift-default-665546959f-wg864 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Nov 03 20:16:01.797801 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:01.790602    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-wg864" podUID="6a1b3d29-b647-46e1-9005-e540e937f85f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Nov 03 20:16:02.145595 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:02.145067    5570 patch_prober.go:29] interesting pod/e2e-worker-w1-6fc748-openshift-default-665546959f-2zvr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body=
Nov 03 20:16:02.145595 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:02.145131    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-2zvr4" podUID="c8edff27-9b0c-40e5-9a72-4a85d7efd3e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused"
Nov 03 20:16:02.247801 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:02.244179    5570 patch_prober.go:29] interesting pod/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmqs6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body=
Nov 03 20:16:02.247801 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:02.244246    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmqs6" podUID="516f80b2-26e4-44d2-b97a-6e740d0b53e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused"
Nov 03 20:16:02.480814 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:02.476189    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g69f8"]
Nov 03 20:16:02.653909 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:02.650147    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-nz52s"
Nov 03 20:16:02.653909 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:02.650274    5570 patch_prober.go:29] interesting pod/e2e-worker-w2-9973d9-openshift-default-ffb6475-nz52s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Nov 03 20:16:02.653909 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:02.650320    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-nz52s" podUID="974badc1-41ad-461c-8719-e1f3f836ccbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Nov 03 20:16:02.845863 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:02.820843    5570 patch_prober.go:29] interesting pod/e2e-worker-w1-6fc748-openshift-default-665546959f-wg864 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Nov 03 20:16:02.845863 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:02.820926    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-wg864" podUID="6a1b3d29-b647-46e1-9005-e540e937f85f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Nov 03 20:16:03.186152 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:03.156136    5570 patch_prober.go:29] interesting pod/e2e-worker-w1-6fc748-openshift-default-665546959f-2zvr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body=
Nov 03 20:16:03.186152 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:03.156208    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-2zvr4" podUID="c8edff27-9b0c-40e5-9a72-4a85d7efd3e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused"
Nov 03 20:16:03.321799 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:03.317619    5570 patch_prober.go:29] interesting pod/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmqs6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body=
Nov 03 20:16:03.321799 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:03.317685    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmqs6" podUID="516f80b2-26e4-44d2-b97a-6e740d0b53e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused"
Nov 03 20:16:03.673450 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:03.672197    5570 patch_prober.go:29] interesting pod/e2e-worker-w2-9973d9-openshift-default-ffb6475-nz52s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Nov 03 20:16:03.673450 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:03.672268    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-nz52s" podUID="974badc1-41ad-461c-8719-e1f3f836ccbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Nov 03 20:16:03.803803 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:03.803525    5570 patch_prober.go:29] interesting pod/e2e-worker-w1-6fc748-openshift-default-665546959f-wg864 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Nov 03 20:16:03.803803 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:03.803586    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-wg864" podUID="6a1b3d29-b647-46e1-9005-e540e937f85f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Nov 03 20:16:04.155008 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:04.153387    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-2zvr4"
Nov 03 20:16:04.155008 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:04.154838    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-2zvr4"
Nov 03 20:16:04.158019 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:04.157943    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-2zvr4"
Nov 03 20:16:04.248406 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:04.247931    5570 patch_prober.go:29] interesting pod/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmqs6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body=
Nov 03 20:16:04.248406 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:04.247995    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmqs6" podUID="516f80b2-26e4-44d2-b97a-6e740d0b53e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused"
Nov 03 20:16:04.648680 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:04.648211    5570 patch_prober.go:29] interesting pod/e2e-worker-w2-9973d9-openshift-default-ffb6475-nz52s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Nov 03 20:16:04.648680 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:04.648281    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-nz52s" podUID="974badc1-41ad-461c-8719-e1f3f836ccbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Nov 03 20:16:04.791633 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:04.791445    5570 patch_prober.go:29] interesting pod/e2e-worker-w1-6fc748-openshift-default-665546959f-wg864 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Nov 03 20:16:04.792207 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:04.792166    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-wg864" podUID="6a1b3d29-b647-46e1-9005-e540e937f85f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Nov 03 20:16:05.255801 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:05.245225    5570 patch_prober.go:29] interesting pod/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmqs6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body=
Nov 03 20:16:05.255801 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:05.245291    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmqs6" podUID="516f80b2-26e4-44d2-b97a-6e740d0b53e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused"
Nov 03 20:16:05.650804 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:05.650234    5570 patch_prober.go:29] interesting pod/e2e-worker-w2-9973d9-openshift-default-ffb6475-nz52s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Nov 03 20:16:05.650804 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:05.650298    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-nz52s" podUID="974badc1-41ad-461c-8719-e1f3f836ccbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Nov 03 20:16:05.809070 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:05.791584    5570 patch_prober.go:29] interesting pod/e2e-worker-w1-6fc748-openshift-default-665546959f-wg864 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Nov 03 20:16:05.809070 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:05.791647    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-wg864" podUID="6a1b3d29-b647-46e1-9005-e540e937f85f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Nov 03 20:16:05.809070 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:05.796974    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-9e1bcb-c7dccc559-tbldl"
Nov 03 20:16:06.222447 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:06.220457    5570 generic.go:356] "Generic (PLEG): container finished" podID="8a255596-5346-460f-b437-7c48d3d33d54" containerID="425dede10b05299e0b57616be19f61783c0772db42f2d9151301ed07b22f477f" exitCode=0
Nov 03 20:16:06.222447 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:06.221409    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-c5nf4" event={"ID":"8a255596-5346-460f-b437-7c48d3d33d54","Type":"ContainerDied","Data":"425dede10b05299e0b57616be19f61783c0772db42f2d9151301ed07b22f477f"}
Nov 03 20:16:06.222447 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:06.221823    5570 scope.go:122] "RemoveContainer" containerID="425dede10b05299e0b57616be19f61783c0772db42f2d9151301ed07b22f477f"
Nov 03 20:16:06.224264 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:06.224003    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:16:06.224264 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:06.224099    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:16:06.224264 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:06.224150    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:16:06.261282 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:06.260882    5570 patch_prober.go:29] interesting pod/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmqs6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body=
Nov 03 20:16:06.261282 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:06.260938    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmqs6" podUID="516f80b2-26e4-44d2-b97a-6e740d0b53e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused"
Nov 03 20:16:06.661799 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:06.658192    5570 patch_prober.go:29] interesting pod/e2e-worker-w2-9973d9-openshift-default-ffb6475-nz52s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Nov 03 20:16:06.661799 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:06.658254    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-nz52s" podUID="974badc1-41ad-461c-8719-e1f3f836ccbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Nov 03 20:16:06.881352 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:06.881046    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-wg864"
Nov 03 20:16:06.883806 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:06.882461    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-wg864"
Nov 03 20:16:06.884819 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:06.884731    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-wg864"
Nov 03 20:16:07.299433 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:07.299386    5570 generic.go:356] "Generic (PLEG): container finished" podID="7ca8412e-c6de-4cb5-944e-ba8afd2fa791" containerID="2ea2028d5fcd730c4d6c7fb86bd31a166bf09e340eba28a808236e578dcc3ffa" exitCode=0
Nov 03 20:16:07.309820 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:07.300233    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-jjjr7" event={"ID":"7ca8412e-c6de-4cb5-944e-ba8afd2fa791","Type":"ContainerDied","Data":"2ea2028d5fcd730c4d6c7fb86bd31a166bf09e340eba28a808236e578dcc3ffa"}
Nov 03 20:16:07.321733 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:07.320694    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-c5nf4" event={"ID":"8a255596-5346-460f-b437-7c48d3d33d54","Type":"ContainerStarted","Data":"a7bb15afee0634b9ec32b88804532afc13e276d3c21985b7c080bb3aa350a328"}
Nov 03 20:16:07.322191 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:07.322161    5570 scope.go:122] "RemoveContainer" containerID="2ea2028d5fcd730c4d6c7fb86bd31a166bf09e340eba28a808236e578dcc3ffa"
Nov 03 20:16:07.335839 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:07.330312    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:16:07.335839 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:07.330457    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:16:07.335839 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:07.330519    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:16:07.453801 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:07.414212    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-c5nf4"
Nov 03 20:16:07.453801 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:07.414333    5570 patch_prober.go:29] interesting pod/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-c5nf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Nov 03 20:16:07.453801 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:07.414389    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-c5nf4" podUID="8a255596-5346-460f-b437-7c48d3d33d54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Nov 03 20:16:07.518925 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:07.516114    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmqs6"
Nov 03 20:16:07.518925 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:07.517046    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmqs6"
Nov 03 20:16:07.519713 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:07.519656    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmqs6"
Nov 03 20:16:07.691799 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:07.674930    5570 patch_prober.go:29] interesting pod/e2e-worker-w2-9973d9-openshift-default-ffb6475-nz52s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Nov 03 20:16:07.691799 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:07.675002    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-nz52s" podUID="974badc1-41ad-461c-8719-e1f3f836ccbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Nov 03 20:16:08.365801 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:08.364197    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-jjjr7" event={"ID":"7ca8412e-c6de-4cb5-944e-ba8afd2fa791","Type":"ContainerStarted","Data":"0e87fdc31bf499c3143805dba02ed2b9d49780fdf96d7d02f51272999a98282d"}
Nov 03 20:16:08.387797 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:08.386309    5570 patch_prober.go:29] interesting pod/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-c5nf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Nov 03 20:16:08.387797 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:08.386391    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-c5nf4" podUID="8a255596-5346-460f-b437-7c48d3d33d54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Nov 03 20:16:08.945292 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:08.945201    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-jjjr7"
Nov 03 20:16:08.945615 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:08.945593    5570 patch_prober.go:29] interesting pod/e2e-worker-w0-e40816-openshift-default-744c57967c-jjjr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Nov 03 20:16:08.945764 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:08.945742    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-jjjr7" podUID="7ca8412e-c6de-4cb5-944e-ba8afd2fa791" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Nov 03 20:16:09.314238 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:09.313883    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-nz52s"
Nov 03 20:16:09.315256 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:09.315236    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-nz52s"
Nov 03 20:16:09.317077 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:09.317055    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-nz52s"
Nov 03 20:16:09.390462 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:09.390426    5570 patch_prober.go:29] interesting pod/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-c5nf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Nov 03 20:16:09.391072 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:09.391041    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-c5nf4" podUID="8a255596-5346-460f-b437-7c48d3d33d54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Nov 03 20:16:09.948800 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:09.946244    5570 patch_prober.go:29] interesting pod/e2e-worker-w0-e40816-openshift-default-744c57967c-jjjr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Nov 03 20:16:09.948800 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:09.946325    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-jjjr7" podUID="7ca8412e-c6de-4cb5-944e-ba8afd2fa791" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Nov 03 20:16:10.385862 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:10.385196    5570 patch_prober.go:29] interesting pod/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-c5nf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Nov 03 20:16:10.385862 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:10.385260    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-c5nf4" podUID="8a255596-5346-460f-b437-7c48d3d33d54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Nov 03 20:16:10.405464 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:10.405431    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sf2lg"]
Nov 03 20:16:10.413602 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:10.410314    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sf2lg"
Nov 03 20:16:10.436142 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:10.436069    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sf2lg"]
Nov 03 20:16:10.516550 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:10.516500    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07093a96-d60c-490f-910b-734f41752c17-workload-socket\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sf2lg\" (UID: \"07093a96-d60c-490f-910b-734f41752c17\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sf2lg"
Nov 03 20:16:10.516724 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:10.516614    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtwv6\" (UniqueName: \"kubernetes.io/projected/07093a96-d60c-490f-910b-734f41752c17-kube-api-access-jtwv6\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sf2lg\" (UID: \"07093a96-d60c-490f-910b-734f41752c17\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sf2lg"
Nov 03 20:16:10.516916 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:10.516878    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07093a96-d60c-490f-910b-734f41752c17-istio-data\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sf2lg\" (UID: \"07093a96-d60c-490f-910b-734f41752c17\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sf2lg"
Nov 03 20:16:10.517074 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:10.517049    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07093a96-d60c-490f-910b-734f41752c17-workload-certs\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sf2lg\" (UID: \"07093a96-d60c-490f-910b-734f41752c17\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sf2lg"
Nov 03 20:16:10.517161 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:10.517139    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07093a96-d60c-490f-910b-734f41752c17-istio-token\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sf2lg\" (UID: \"07093a96-d60c-490f-910b-734f41752c17\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sf2lg"
Nov 03 20:16:10.517277 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:10.517254    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07093a96-d60c-490f-910b-734f41752c17-istio-podinfo\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sf2lg\" (UID: \"07093a96-d60c-490f-910b-734f41752c17\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sf2lg"
Nov 03 20:16:10.517326 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:10.517311    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07093a96-d60c-490f-910b-734f41752c17-istio-envoy\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sf2lg\" (UID: \"07093a96-d60c-490f-910b-734f41752c17\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sf2lg"
Nov 03 20:16:10.517401 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:10.517385    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07093a96-d60c-490f-910b-734f41752c17-credential-socket\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sf2lg\" (UID: \"07093a96-d60c-490f-910b-734f41752c17\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sf2lg"
Nov 03 20:16:10.517489 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:10.517476    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07093a96-d60c-490f-910b-734f41752c17-istiod-ca-cert\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sf2lg\" (UID: \"07093a96-d60c-490f-910b-734f41752c17\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sf2lg"
Nov 03 20:16:10.618666 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:10.618628    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07093a96-d60c-490f-910b-734f41752c17-workload-certs\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sf2lg\" (UID: \"07093a96-d60c-490f-910b-734f41752c17\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sf2lg"
Nov 03 20:16:10.618916 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:10.618901    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07093a96-d60c-490f-910b-734f41752c17-istio-token\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sf2lg\" (UID: \"07093a96-d60c-490f-910b-734f41752c17\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sf2lg"
Nov 03 20:16:10.619028 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:10.619018    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07093a96-d60c-490f-910b-734f41752c17-istio-podinfo\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sf2lg\" (UID: \"07093a96-d60c-490f-910b-734f41752c17\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sf2lg"
Nov 03 20:16:10.619105 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:10.619096    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07093a96-d60c-490f-910b-734f41752c17-istio-envoy\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sf2lg\" (UID: \"07093a96-d60c-490f-910b-734f41752c17\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sf2lg"
Nov 03 20:16:10.619199 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:10.619167    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07093a96-d60c-490f-910b-734f41752c17-workload-certs\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sf2lg\" (UID: \"07093a96-d60c-490f-910b-734f41752c17\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sf2lg"
Nov 03 20:16:10.619249 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:10.619192    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07093a96-d60c-490f-910b-734f41752c17-credential-socket\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sf2lg\" (UID: \"07093a96-d60c-490f-910b-734f41752c17\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sf2lg"
Nov 03 20:16:10.619333 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:10.619324    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07093a96-d60c-490f-910b-734f41752c17-istiod-ca-cert\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sf2lg\" (UID: \"07093a96-d60c-490f-910b-734f41752c17\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sf2lg"
Nov 03 20:16:10.619395 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:10.619383    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07093a96-d60c-490f-910b-734f41752c17-workload-socket\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sf2lg\" (UID: \"07093a96-d60c-490f-910b-734f41752c17\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sf2lg"
Nov 03 20:16:10.619471 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:10.619462    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtwv6\" (UniqueName: \"kubernetes.io/projected/07093a96-d60c-490f-910b-734f41752c17-kube-api-access-jtwv6\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sf2lg\" (UID: \"07093a96-d60c-490f-910b-734f41752c17\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sf2lg"
Nov 03 20:16:10.619539 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:10.619531    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07093a96-d60c-490f-910b-734f41752c17-istio-data\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sf2lg\" (UID: \"07093a96-d60c-490f-910b-734f41752c17\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sf2lg"
Nov 03 20:16:10.619985 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:10.619959    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07093a96-d60c-490f-910b-734f41752c17-istio-data\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sf2lg\" (UID: \"07093a96-d60c-490f-910b-734f41752c17\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sf2lg"
Nov 03 20:16:10.620663 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:10.620315    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07093a96-d60c-490f-910b-734f41752c17-workload-socket\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sf2lg\" (UID: \"07093a96-d60c-490f-910b-734f41752c17\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sf2lg"
Nov 03 20:16:10.620663 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:10.620337    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07093a96-d60c-490f-910b-734f41752c17-istiod-ca-cert\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sf2lg\" (UID: \"07093a96-d60c-490f-910b-734f41752c17\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sf2lg"
Nov 03 20:16:10.620663 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:10.620581    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07093a96-d60c-490f-910b-734f41752c17-credential-socket\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sf2lg\" (UID: \"07093a96-d60c-490f-910b-734f41752c17\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sf2lg"
Nov 03 20:16:10.622597 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:10.622568    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07093a96-d60c-490f-910b-734f41752c17-istio-envoy\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sf2lg\" (UID: \"07093a96-d60c-490f-910b-734f41752c17\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sf2lg"
Nov 03 20:16:10.622697 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:10.622679    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07093a96-d60c-490f-910b-734f41752c17-istio-podinfo\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sf2lg\" (UID: \"07093a96-d60c-490f-910b-734f41752c17\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sf2lg"
Nov 03 20:16:10.675863 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:10.670833    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtwv6\" (UniqueName: \"kubernetes.io/projected/07093a96-d60c-490f-910b-734f41752c17-kube-api-access-jtwv6\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sf2lg\" (UID: \"07093a96-d60c-490f-910b-734f41752c17\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sf2lg"
Nov 03 20:16:10.697311 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:10.697262    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07093a96-d60c-490f-910b-734f41752c17-istio-token\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sf2lg\" (UID: \"07093a96-d60c-490f-910b-734f41752c17\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sf2lg"
Nov 03 20:16:10.806254 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:10.806009    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sf2lg"
Nov 03 20:16:10.974646 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:10.945278    5570 patch_prober.go:29] interesting pod/e2e-worker-w0-e40816-openshift-default-744c57967c-jjjr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Nov 03 20:16:10.974646 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:10.945343    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-jjjr7" podUID="7ca8412e-c6de-4cb5-944e-ba8afd2fa791" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Nov 03 20:16:11.189423 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:11.189322    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sf2lg"]
Nov 03 20:16:11.196625 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:16:11.196586    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07093a96_d60c_490f_910b_734f41752c17.slice/crio-2f0a020b0fbcc4d7ff6676b2bbb635d5f5f89249504ac614149acab58ca6525a WatchSource:0}: Error finding container 2f0a020b0fbcc4d7ff6676b2bbb635d5f5f89249504ac614149acab58ca6525a: Status 404 returned error can't find the container with id 2f0a020b0fbcc4d7ff6676b2bbb635d5f5f89249504ac614149acab58ca6525a
Nov 03 20:16:11.199219 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:11.199182    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:16:11.199393 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:11.199280    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:16:11.199393 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:11.199329    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:16:11.388797 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:11.388178    5570 patch_prober.go:29] interesting pod/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-c5nf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Nov 03 20:16:11.388797 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:11.388254    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-c5nf4" podUID="8a255596-5346-460f-b437-7c48d3d33d54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Nov 03 20:16:11.434688 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:11.434650    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sf2lg" event={"ID":"07093a96-d60c-490f-910b-734f41752c17","Type":"ContainerStarted","Data":"52cc5ce0dff70b1613a235dd886d31d1c1e2960a2c37ef9b6eeff114ec29d26a"}
Nov 03 20:16:11.435184 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:11.434706    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sf2lg" event={"ID":"07093a96-d60c-490f-910b-734f41752c17","Type":"ContainerStarted","Data":"2f0a020b0fbcc4d7ff6676b2bbb635d5f5f89249504ac614149acab58ca6525a"}
Nov 03 20:16:11.806875 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:11.806844    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sf2lg"
Nov 03 20:16:11.947807 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:11.944078    5570 patch_prober.go:29] interesting pod/e2e-worker-w0-e40816-openshift-default-744c57967c-jjjr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Nov 03 20:16:11.947807 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:11.944146    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-jjjr7" podUID="7ca8412e-c6de-4cb5-944e-ba8afd2fa791" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Nov 03 20:16:12.385524 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:12.384226    5570 patch_prober.go:29] interesting pod/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-c5nf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Nov 03 20:16:12.385524 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:12.384292    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-c5nf4" podUID="8a255596-5346-460f-b437-7c48d3d33d54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Nov 03 20:16:12.808478 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:12.808428    5570 patch_prober.go:29] interesting pod/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sf2lg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body=
Nov 03 20:16:12.808721 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:12.808484    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sf2lg" podUID="07093a96-d60c-490f-910b-734f41752c17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused"
Nov 03 20:16:12.812029 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:12.811998    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-c5nf4"
Nov 03 20:16:12.899097 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:12.898793    5570 patch_prober.go:29] interesting pod/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-vzfw9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": read tcp 10.134.0.2:33086->10.134.0.39:15021: read: connection reset by peer" start-of-body=
Nov 03 20:16:12.899097 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:12.898857    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-vzfw9" podUID="a739707b-9c5a-4c9b-90df-501c5dd0c98d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": read tcp 10.134.0.2:33086->10.134.0.39:15021: read: connection reset by peer"
Nov 03 20:16:12.946673 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:12.946211    5570 patch_prober.go:29] interesting pod/e2e-worker-w0-e40816-openshift-default-744c57967c-jjjr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Nov 03 20:16:12.946673 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:12.946279    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-jjjr7" podUID="7ca8412e-c6de-4cb5-944e-ba8afd2fa791" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Nov 03 20:16:13.390807 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:13.390707    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-c5nf4"
Nov 03 20:16:13.462188 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:13.462112    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sf2lg" podStartSLOduration=3.46208722 podStartE2EDuration="3.46208722s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:16:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:16:11.491748474 +0000 UTC m=+705.644410364" watchObservedRunningTime="2026-11-03 20:16:13.46208722 +0000 UTC m=+707.614749108"
Nov 03 20:16:13.490851 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:13.490811    5570 generic.go:356] "Generic (PLEG): container finished" podID="a739707b-9c5a-4c9b-90df-501c5dd0c98d" containerID="016c483e8a6ffb6d372c0a550482b3dc7344bd52f597fb28b1466c5c5565141a" exitCode=0
Nov 03 20:16:13.491084 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:13.490874    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-vzfw9" event={"ID":"a739707b-9c5a-4c9b-90df-501c5dd0c98d","Type":"ContainerDied","Data":"016c483e8a6ffb6d372c0a550482b3dc7344bd52f597fb28b1466c5c5565141a"}
Nov 03 20:16:13.491622 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:13.491477    5570 scope.go:122] "RemoveContainer" containerID="016c483e8a6ffb6d372c0a550482b3dc7344bd52f597fb28b1466c5c5565141a"
Nov 03 20:16:13.492569 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:13.492351    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-c5nf4"
Nov 03 20:16:13.492908 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:13.492884    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:16:13.492974 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:13.492961    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:16:13.493017 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:13.493002    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:16:13.807800 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:13.807031    5570 patch_prober.go:29] interesting pod/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sf2lg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body=
Nov 03 20:16:13.807800 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:13.807083    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sf2lg" podUID="07093a96-d60c-490f-910b-734f41752c17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused"
Nov 03 20:16:13.817097 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:13.817042    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-jjjr7"
Nov 03 20:16:13.943799 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:13.943236    5570 patch_prober.go:29] interesting pod/e2e-worker-w0-e40816-openshift-default-744c57967c-jjjr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Nov 03 20:16:13.943799 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:13.943302    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-jjjr7" podUID="7ca8412e-c6de-4cb5-944e-ba8afd2fa791" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Nov 03 20:16:14.499334 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:14.499295    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-vzfw9" event={"ID":"a739707b-9c5a-4c9b-90df-501c5dd0c98d","Type":"ContainerStarted","Data":"c98ca04c2cb8bdbe656136e1cd8d0fcac1daa8fdb11f2f3858d8c7f463fa7537"}
Nov 03 20:16:14.522806 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:14.522756    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-vzfw9"
Nov 03 20:16:14.523030 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:14.523008    5570 patch_prober.go:29] interesting pod/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-vzfw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Nov 03 20:16:14.523084 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:14.523072    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-vzfw9" podUID="a739707b-9c5a-4c9b-90df-501c5dd0c98d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Nov 03 20:16:14.807155 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:14.807064    5570 patch_prober.go:29] interesting pod/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sf2lg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body=
Nov 03 20:16:14.807155 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:14.807120    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sf2lg" podUID="07093a96-d60c-490f-910b-734f41752c17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused"
Nov 03 20:16:14.975797 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:14.975189    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-jjjr7"
Nov 03 20:16:14.982075 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:14.982042    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-jjjr7"
Nov 03 20:16:15.523500 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:15.523466    5570 patch_prober.go:29] interesting pod/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-vzfw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Nov 03 20:16:15.523684 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:15.523521    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-vzfw9" podUID="a739707b-9c5a-4c9b-90df-501c5dd0c98d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Nov 03 20:16:15.807137 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:15.807052    5570 patch_prober.go:29] interesting pod/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sf2lg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body=
Nov 03 20:16:15.807137 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:15.807108    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sf2lg" podUID="07093a96-d60c-490f-910b-734f41752c17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused"
Nov 03 20:16:16.523210 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:16.523172    5570 patch_prober.go:29] interesting pod/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-vzfw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Nov 03 20:16:16.523628 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:16.523229    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-vzfw9" podUID="a739707b-9c5a-4c9b-90df-501c5dd0c98d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Nov 03 20:16:16.806794 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:16.806680    5570 patch_prober.go:29] interesting pod/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sf2lg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body=
Nov 03 20:16:16.806794 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:16.806736    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sf2lg" podUID="07093a96-d60c-490f-910b-734f41752c17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused"
Nov 03 20:16:17.011558 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:17.011525    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-e40816/simulator-w0-e40816-kserve-688f79796f-2tvw5"]
Nov 03 20:16:17.018737 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:17.018714    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e40816/simulator-w0-e40816-kserve-688f79796f-2tvw5"
Nov 03 20:16:17.035157 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:17.035130    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-e40816/simulator-w0-e40816-kserve-688f79796f-2tvw5"]
Nov 03 20:16:17.085421 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:17.085301    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-e40816\"/\"default-dockercfg-2vgst\""
Nov 03 20:16:17.085997 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:17.085951    5570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-e40816\"/\"kube-root-ca.crt\""
Nov 03 20:16:17.086471 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:17.086453    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-e40816\"/\"simulator-w0-e40816-kserve-self-signed-certs\""
Nov 03 20:16:17.086822 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:17.086805    5570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-e40816\"/\"openshift-service-ca.crt\""
Nov 03 20:16:17.195934 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:17.195668    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62-tls-certs\") pod \"simulator-w0-e40816-kserve-688f79796f-2tvw5\" (UID: \"c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62\") " pod="e2e-models-e2e-worker-w0-e40816/simulator-w0-e40816-kserve-688f79796f-2tvw5"
Nov 03 20:16:17.195934 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:17.195709    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62-home\") pod \"simulator-w0-e40816-kserve-688f79796f-2tvw5\" (UID: \"c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62\") " pod="e2e-models-e2e-worker-w0-e40816/simulator-w0-e40816-kserve-688f79796f-2tvw5"
Nov 03 20:16:17.195934 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:17.195734    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7q7j\" (UniqueName: \"kubernetes.io/projected/c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62-kube-api-access-q7q7j\") pod \"simulator-w0-e40816-kserve-688f79796f-2tvw5\" (UID: \"c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62\") " pod="e2e-models-e2e-worker-w0-e40816/simulator-w0-e40816-kserve-688f79796f-2tvw5"
Nov 03 20:16:17.195934 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:17.195763    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62-dshm\") pod \"simulator-w0-e40816-kserve-688f79796f-2tvw5\" (UID: \"c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62\") " pod="e2e-models-e2e-worker-w0-e40816/simulator-w0-e40816-kserve-688f79796f-2tvw5"
Nov 03 20:16:17.195934 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:17.195833    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62-model-cache\") pod \"simulator-w0-e40816-kserve-688f79796f-2tvw5\" (UID: \"c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62\") " pod="e2e-models-e2e-worker-w0-e40816/simulator-w0-e40816-kserve-688f79796f-2tvw5"
Nov 03 20:16:17.196340 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:17.196083    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62-tmp-dir\") pod \"simulator-w0-e40816-kserve-688f79796f-2tvw5\" (UID: \"c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62\") " pod="e2e-models-e2e-worker-w0-e40816/simulator-w0-e40816-kserve-688f79796f-2tvw5"
Nov 03 20:16:17.297445 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:17.297406    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62-tls-certs\") pod \"simulator-w0-e40816-kserve-688f79796f-2tvw5\" (UID: \"c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62\") " pod="e2e-models-e2e-worker-w0-e40816/simulator-w0-e40816-kserve-688f79796f-2tvw5"
Nov 03 20:16:17.297673 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:17.297657    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62-home\") pod \"simulator-w0-e40816-kserve-688f79796f-2tvw5\" (UID: \"c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62\") " pod="e2e-models-e2e-worker-w0-e40816/simulator-w0-e40816-kserve-688f79796f-2tvw5"
Nov 03 20:16:17.297767 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:17.297755    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7q7j\" (UniqueName: \"kubernetes.io/projected/c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62-kube-api-access-q7q7j\") pod \"simulator-w0-e40816-kserve-688f79796f-2tvw5\" (UID: \"c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62\") " pod="e2e-models-e2e-worker-w0-e40816/simulator-w0-e40816-kserve-688f79796f-2tvw5"
Nov 03 20:16:17.297893 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:17.297883    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62-dshm\") pod \"simulator-w0-e40816-kserve-688f79796f-2tvw5\" (UID: \"c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62\") " pod="e2e-models-e2e-worker-w0-e40816/simulator-w0-e40816-kserve-688f79796f-2tvw5"
Nov 03 20:16:17.298008 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:17.297981    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62-model-cache\") pod \"simulator-w0-e40816-kserve-688f79796f-2tvw5\" (UID: \"c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62\") " pod="e2e-models-e2e-worker-w0-e40816/simulator-w0-e40816-kserve-688f79796f-2tvw5"
Nov 03 20:16:17.298111 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:17.298102    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62-tmp-dir\") pod \"simulator-w0-e40816-kserve-688f79796f-2tvw5\" (UID: \"c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62\") " pod="e2e-models-e2e-worker-w0-e40816/simulator-w0-e40816-kserve-688f79796f-2tvw5"
Nov 03 20:16:17.298317 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:17.298173    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62-home\") pod \"simulator-w0-e40816-kserve-688f79796f-2tvw5\" (UID: \"c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62\") " pod="e2e-models-e2e-worker-w0-e40816/simulator-w0-e40816-kserve-688f79796f-2tvw5"
Nov 03 20:16:17.298500 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:17.298473    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62-model-cache\") pod \"simulator-w0-e40816-kserve-688f79796f-2tvw5\" (UID: \"c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62\") " pod="e2e-models-e2e-worker-w0-e40816/simulator-w0-e40816-kserve-688f79796f-2tvw5"
Nov 03 20:16:17.298561 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:17.298503    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62-tmp-dir\") pod \"simulator-w0-e40816-kserve-688f79796f-2tvw5\" (UID: \"c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62\") " pod="e2e-models-e2e-worker-w0-e40816/simulator-w0-e40816-kserve-688f79796f-2tvw5"
Nov 03 20:16:17.301937 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:17.301741    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62-tls-certs\") pod \"simulator-w0-e40816-kserve-688f79796f-2tvw5\" (UID: \"c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62\") " pod="e2e-models-e2e-worker-w0-e40816/simulator-w0-e40816-kserve-688f79796f-2tvw5"
Nov 03 20:16:17.301937 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:17.301748    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62-dshm\") pod \"simulator-w0-e40816-kserve-688f79796f-2tvw5\" (UID: \"c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62\") " pod="e2e-models-e2e-worker-w0-e40816/simulator-w0-e40816-kserve-688f79796f-2tvw5"
Nov 03 20:16:17.340852 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:17.331375    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7q7j\" (UniqueName: \"kubernetes.io/projected/c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62-kube-api-access-q7q7j\") pod \"simulator-w0-e40816-kserve-688f79796f-2tvw5\" (UID: \"c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62\") " pod="e2e-models-e2e-worker-w0-e40816/simulator-w0-e40816-kserve-688f79796f-2tvw5"
Nov 03 20:16:17.351688 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:17.346000    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e40816/simulator-w0-e40816-kserve-688f79796f-2tvw5"
Nov 03 20:16:17.525590 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:17.525134    5570 patch_prober.go:29] interesting pod/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-vzfw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Nov 03 20:16:17.525590 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:17.525206    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-vzfw9" podUID="a739707b-9c5a-4c9b-90df-501c5dd0c98d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Nov 03 20:16:17.806889 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:17.806851    5570 patch_prober.go:29] interesting pod/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sf2lg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body=
Nov 03 20:16:17.807094 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:17.806921    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sf2lg" podUID="07093a96-d60c-490f-910b-734f41752c17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused"
Nov 03 20:16:17.851161 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:17.849411    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-e40816/simulator-w0-e40816-kserve-688f79796f-2tvw5"]
Nov 03 20:16:18.264799 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:18.264507    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g69f8"]
Nov 03 20:16:18.527312 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:18.527215    5570 patch_prober.go:29] interesting pod/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-vzfw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Nov 03 20:16:18.527956 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:18.527912    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-vzfw9" podUID="a739707b-9c5a-4c9b-90df-501c5dd0c98d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Nov 03 20:16:18.554814 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:18.554368    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e40816/simulator-w0-e40816-kserve-688f79796f-2tvw5" event={"ID":"c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62","Type":"ContainerStarted","Data":"a72316ce80e586f5136087ae6bd2eddc4062a714495080e83050487e5e330a81"}
Nov 03 20:16:18.554814 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:18.554427    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e40816/simulator-w0-e40816-kserve-688f79796f-2tvw5" event={"ID":"c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62","Type":"ContainerStarted","Data":"8106f236cc15616f87882da99a0451d4d904a6dab96fedbadd78a1c6bff657dc"}
Nov 03 20:16:18.601798 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:18.600939    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-e40816/simulator-w0-e40816-kserve-688f79796f-2tvw5" podStartSLOduration=2.600917419 podStartE2EDuration="2.600917419s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:16:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:16:18.600249912 +0000 UTC m=+712.752911799" watchObservedRunningTime="2026-11-03 20:16:18.600917419 +0000 UTC m=+712.753579313"
Nov 03 20:16:18.823408 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:18.823314    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sf2lg"
Nov 03 20:16:18.825416 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:18.825372    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sf2lg"
Nov 03 20:16:18.827465 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:18.826885    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sf2lg"
Nov 03 20:16:19.347189 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:19.347131    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e40816/simulator-w0-e40816-kserve-688f79796f-2tvw5"
Nov 03 20:16:19.522825 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:19.522783    5570 patch_prober.go:29] interesting pod/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-vzfw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Nov 03 20:16:19.523015 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:19.522860    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-vzfw9" podUID="a739707b-9c5a-4c9b-90df-501c5dd0c98d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Nov 03 20:16:20.526873 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:20.526844    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-vzfw9"
Nov 03 20:16:20.527306 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:20.527193    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-vzfw9"
Nov 03 20:16:20.527762 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:20.527743    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-vzfw9"
Nov 03 20:16:22.262504 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:22.262468    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-e80203/premium-w4-e80203-kserve-5f9f445456-bms9d"]
Nov 03 20:16:22.270373 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:22.270350    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-e80203/premium-w4-e80203-kserve-5f9f445456-bms9d"
Nov 03 20:16:22.289192 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:22.289161    5570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-e80203\"/\"openshift-service-ca.crt\""
Nov 03 20:16:22.289684 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:22.289658    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-e80203/premium-w4-e80203-kserve-5f9f445456-bms9d"]
Nov 03 20:16:22.300424 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:22.300392    5570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-e80203\"/\"kube-root-ca.crt\""
Nov 03 20:16:22.302613 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:22.302595    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-e80203\"/\"default-dockercfg-zvkt9\""
Nov 03 20:16:22.302685 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:22.302619    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-e80203\"/\"premium-w4-e80203-kserve-self-signed-certs\""
Nov 03 20:16:22.395293 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:22.395260    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ad76efc-20c2-4951-b22a-c5a39cd39ed4-home\") pod \"premium-w4-e80203-kserve-5f9f445456-bms9d\" (UID: \"6ad76efc-20c2-4951-b22a-c5a39cd39ed4\") " pod="e2e-models-e2e-worker-w4-e80203/premium-w4-e80203-kserve-5f9f445456-bms9d"
Nov 03 20:16:22.395461 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:22.395333    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ad76efc-20c2-4951-b22a-c5a39cd39ed4-tmp-dir\") pod \"premium-w4-e80203-kserve-5f9f445456-bms9d\" (UID: \"6ad76efc-20c2-4951-b22a-c5a39cd39ed4\") " pod="e2e-models-e2e-worker-w4-e80203/premium-w4-e80203-kserve-5f9f445456-bms9d"
Nov 03 20:16:22.395590 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:22.395492    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ad76efc-20c2-4951-b22a-c5a39cd39ed4-model-cache\") pod \"premium-w4-e80203-kserve-5f9f445456-bms9d\" (UID: \"6ad76efc-20c2-4951-b22a-c5a39cd39ed4\") " pod="e2e-models-e2e-worker-w4-e80203/premium-w4-e80203-kserve-5f9f445456-bms9d"
Nov 03 20:16:22.395590 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:22.395550    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ad76efc-20c2-4951-b22a-c5a39cd39ed4-dshm\") pod \"premium-w4-e80203-kserve-5f9f445456-bms9d\" (UID: \"6ad76efc-20c2-4951-b22a-c5a39cd39ed4\") " pod="e2e-models-e2e-worker-w4-e80203/premium-w4-e80203-kserve-5f9f445456-bms9d"
Nov 03 20:16:22.395661 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:22.395600    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dprnk\" (UniqueName: \"kubernetes.io/projected/6ad76efc-20c2-4951-b22a-c5a39cd39ed4-kube-api-access-dprnk\") pod \"premium-w4-e80203-kserve-5f9f445456-bms9d\" (UID: \"6ad76efc-20c2-4951-b22a-c5a39cd39ed4\") " pod="e2e-models-e2e-worker-w4-e80203/premium-w4-e80203-kserve-5f9f445456-bms9d"
Nov 03 20:16:22.395708 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:22.395692    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ad76efc-20c2-4951-b22a-c5a39cd39ed4-tls-certs\") pod \"premium-w4-e80203-kserve-5f9f445456-bms9d\" (UID: \"6ad76efc-20c2-4951-b22a-c5a39cd39ed4\") " pod="e2e-models-e2e-worker-w4-e80203/premium-w4-e80203-kserve-5f9f445456-bms9d"
Nov 03 20:16:22.497799 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:22.496758    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ad76efc-20c2-4951-b22a-c5a39cd39ed4-home\") pod \"premium-w4-e80203-kserve-5f9f445456-bms9d\" (UID: \"6ad76efc-20c2-4951-b22a-c5a39cd39ed4\") " pod="e2e-models-e2e-worker-w4-e80203/premium-w4-e80203-kserve-5f9f445456-bms9d"
Nov 03 20:16:22.497799 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:22.496956    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ad76efc-20c2-4951-b22a-c5a39cd39ed4-tmp-dir\") pod \"premium-w4-e80203-kserve-5f9f445456-bms9d\" (UID: \"6ad76efc-20c2-4951-b22a-c5a39cd39ed4\") " pod="e2e-models-e2e-worker-w4-e80203/premium-w4-e80203-kserve-5f9f445456-bms9d"
Nov 03 20:16:22.497799 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:22.497059    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ad76efc-20c2-4951-b22a-c5a39cd39ed4-model-cache\") pod \"premium-w4-e80203-kserve-5f9f445456-bms9d\" (UID: \"6ad76efc-20c2-4951-b22a-c5a39cd39ed4\") " pod="e2e-models-e2e-worker-w4-e80203/premium-w4-e80203-kserve-5f9f445456-bms9d"
Nov 03 20:16:22.497799 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:22.497115    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ad76efc-20c2-4951-b22a-c5a39cd39ed4-dshm\") pod \"premium-w4-e80203-kserve-5f9f445456-bms9d\" (UID: \"6ad76efc-20c2-4951-b22a-c5a39cd39ed4\") " pod="e2e-models-e2e-worker-w4-e80203/premium-w4-e80203-kserve-5f9f445456-bms9d"
Nov 03 20:16:22.497799 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:22.497146    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dprnk\" (UniqueName: \"kubernetes.io/projected/6ad76efc-20c2-4951-b22a-c5a39cd39ed4-kube-api-access-dprnk\") pod \"premium-w4-e80203-kserve-5f9f445456-bms9d\" (UID: \"6ad76efc-20c2-4951-b22a-c5a39cd39ed4\") " pod="e2e-models-e2e-worker-w4-e80203/premium-w4-e80203-kserve-5f9f445456-bms9d"
Nov 03 20:16:22.497799 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:22.497193    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ad76efc-20c2-4951-b22a-c5a39cd39ed4-tls-certs\") pod \"premium-w4-e80203-kserve-5f9f445456-bms9d\" (UID: \"6ad76efc-20c2-4951-b22a-c5a39cd39ed4\") " pod="e2e-models-e2e-worker-w4-e80203/premium-w4-e80203-kserve-5f9f445456-bms9d"
Nov 03 20:16:22.498268 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:22.498112    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ad76efc-20c2-4951-b22a-c5a39cd39ed4-model-cache\") pod \"premium-w4-e80203-kserve-5f9f445456-bms9d\" (UID: \"6ad76efc-20c2-4951-b22a-c5a39cd39ed4\") " pod="e2e-models-e2e-worker-w4-e80203/premium-w4-e80203-kserve-5f9f445456-bms9d"
Nov 03 20:16:22.498332 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:22.498298    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ad76efc-20c2-4951-b22a-c5a39cd39ed4-home\") pod \"premium-w4-e80203-kserve-5f9f445456-bms9d\" (UID: \"6ad76efc-20c2-4951-b22a-c5a39cd39ed4\") " pod="e2e-models-e2e-worker-w4-e80203/premium-w4-e80203-kserve-5f9f445456-bms9d"
Nov 03 20:16:22.499882 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:22.499072    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ad76efc-20c2-4951-b22a-c5a39cd39ed4-tmp-dir\") pod \"premium-w4-e80203-kserve-5f9f445456-bms9d\" (UID: \"6ad76efc-20c2-4951-b22a-c5a39cd39ed4\") " pod="e2e-models-e2e-worker-w4-e80203/premium-w4-e80203-kserve-5f9f445456-bms9d"
Nov 03 20:16:22.502210 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:22.502185    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ad76efc-20c2-4951-b22a-c5a39cd39ed4-tls-certs\") pod \"premium-w4-e80203-kserve-5f9f445456-bms9d\" (UID: \"6ad76efc-20c2-4951-b22a-c5a39cd39ed4\") " pod="e2e-models-e2e-worker-w4-e80203/premium-w4-e80203-kserve-5f9f445456-bms9d"
Nov 03 20:16:22.502735 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:22.502489    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ad76efc-20c2-4951-b22a-c5a39cd39ed4-dshm\") pod \"premium-w4-e80203-kserve-5f9f445456-bms9d\" (UID: \"6ad76efc-20c2-4951-b22a-c5a39cd39ed4\") " pod="e2e-models-e2e-worker-w4-e80203/premium-w4-e80203-kserve-5f9f445456-bms9d"
Nov 03 20:16:22.519162 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:22.519091    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dprnk\" (UniqueName: \"kubernetes.io/projected/6ad76efc-20c2-4951-b22a-c5a39cd39ed4-kube-api-access-dprnk\") pod \"premium-w4-e80203-kserve-5f9f445456-bms9d\" (UID: \"6ad76efc-20c2-4951-b22a-c5a39cd39ed4\") " pod="e2e-models-e2e-worker-w4-e80203/premium-w4-e80203-kserve-5f9f445456-bms9d"
Nov 03 20:16:22.589710 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:22.589677    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-e80203/premium-w4-e80203-kserve-5f9f445456-bms9d"
Nov 03 20:16:22.947267 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:22.947237    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-e80203/premium-w4-e80203-kserve-5f9f445456-bms9d"]
Nov 03 20:16:22.949185 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:16:22.949158    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6ad76efc_20c2_4951_b22a_c5a39cd39ed4.slice/crio-80804640db3c0c3ec17bff5c409a25e8034e0fbf6eb3d7bef4ed3c03a4d241f1 WatchSource:0}: Error finding container 80804640db3c0c3ec17bff5c409a25e8034e0fbf6eb3d7bef4ed3c03a4d241f1: Status 404 returned error can't find the container with id 80804640db3c0c3ec17bff5c409a25e8034e0fbf6eb3d7bef4ed3c03a4d241f1
Nov 03 20:16:23.592660 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:23.592620    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-e80203/premium-w4-e80203-kserve-5f9f445456-bms9d" event={"ID":"6ad76efc-20c2-4951-b22a-c5a39cd39ed4","Type":"ContainerStarted","Data":"c3b73dcf7616cb22723e91b2213cf6b65ff8a87c72b5883c75303046ba91cc20"}
Nov 03 20:16:23.592660 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:23.592673    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-e80203/premium-w4-e80203-kserve-5f9f445456-bms9d" event={"ID":"6ad76efc-20c2-4951-b22a-c5a39cd39ed4","Type":"ContainerStarted","Data":"80804640db3c0c3ec17bff5c409a25e8034e0fbf6eb3d7bef4ed3c03a4d241f1"}
Nov 03 20:16:23.635021 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:23.634965    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-e80203/premium-w4-e80203-kserve-5f9f445456-bms9d" podStartSLOduration=1.6349501069999999 podStartE2EDuration="1.634950107s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:16:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:16:23.634456806 +0000 UTC m=+717.787118690" watchObservedRunningTime="2026-11-03 20:16:23.634950107 +0000 UTC m=+717.787612039"
Nov 03 20:16:24.591798 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:24.590739    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-e80203/premium-w4-e80203-kserve-5f9f445456-bms9d"
Nov 03 20:16:25.780800 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:25.776167    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-9gvtf"]
Nov 03 20:16:25.841257 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:25.832880    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-9gvtf"]
Nov 03 20:16:25.841257 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:25.833022    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-9gvtf"
Nov 03 20:16:25.947794 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:25.944799    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e7f17ab-0f1d-4a59-b415-fe844ae7c1e7-workload-socket\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-9gvtf\" (UID: \"5e7f17ab-0f1d-4a59-b415-fe844ae7c1e7\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-9gvtf"
Nov 03 20:16:25.947794 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:25.944980    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e7f17ab-0f1d-4a59-b415-fe844ae7c1e7-istio-token\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-9gvtf\" (UID: \"5e7f17ab-0f1d-4a59-b415-fe844ae7c1e7\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-9gvtf"
Nov 03 20:16:25.947794 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:25.945023    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e7f17ab-0f1d-4a59-b415-fe844ae7c1e7-credential-socket\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-9gvtf\" (UID: \"5e7f17ab-0f1d-4a59-b415-fe844ae7c1e7\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-9gvtf"
Nov 03 20:16:25.947794 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:25.945095    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e7f17ab-0f1d-4a59-b415-fe844ae7c1e7-istio-data\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-9gvtf\" (UID: \"5e7f17ab-0f1d-4a59-b415-fe844ae7c1e7\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-9gvtf"
Nov 03 20:16:25.947794 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:25.945153    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e7f17ab-0f1d-4a59-b415-fe844ae7c1e7-istio-podinfo\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-9gvtf\" (UID: \"5e7f17ab-0f1d-4a59-b415-fe844ae7c1e7\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-9gvtf"
Nov 03 20:16:25.947794 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:25.945223    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e7f17ab-0f1d-4a59-b415-fe844ae7c1e7-workload-certs\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-9gvtf\" (UID: \"5e7f17ab-0f1d-4a59-b415-fe844ae7c1e7\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-9gvtf"
Nov 03 20:16:25.947794 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:25.945334    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e7f17ab-0f1d-4a59-b415-fe844ae7c1e7-istio-envoy\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-9gvtf\" (UID: \"5e7f17ab-0f1d-4a59-b415-fe844ae7c1e7\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-9gvtf"
Nov 03 20:16:26.046627 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:26.046530    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e7f17ab-0f1d-4a59-b415-fe844ae7c1e7-istio-envoy\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-9gvtf\" (UID: \"5e7f17ab-0f1d-4a59-b415-fe844ae7c1e7\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-9gvtf"
Nov 03 20:16:26.047011 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:26.046984    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e7f17ab-0f1d-4a59-b415-fe844ae7c1e7-workload-socket\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-9gvtf\" (UID: \"5e7f17ab-0f1d-4a59-b415-fe844ae7c1e7\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-9gvtf"
Nov 03 20:16:26.047175 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:26.047157    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msl98\" (UniqueName: \"kubernetes.io/projected/5e7f17ab-0f1d-4a59-b415-fe844ae7c1e7-kube-api-access-msl98\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-9gvtf\" (UID: \"5e7f17ab-0f1d-4a59-b415-fe844ae7c1e7\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-9gvtf"
Nov 03 20:16:26.047296 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:26.047281    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e7f17ab-0f1d-4a59-b415-fe844ae7c1e7-istio-token\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-9gvtf\" (UID: \"5e7f17ab-0f1d-4a59-b415-fe844ae7c1e7\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-9gvtf"
Nov 03 20:16:26.047429 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:26.047412    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e7f17ab-0f1d-4a59-b415-fe844ae7c1e7-credential-socket\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-9gvtf\" (UID: \"5e7f17ab-0f1d-4a59-b415-fe844ae7c1e7\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-9gvtf"
Nov 03 20:16:26.047558 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:26.047543    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e7f17ab-0f1d-4a59-b415-fe844ae7c1e7-istio-data\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-9gvtf\" (UID: \"5e7f17ab-0f1d-4a59-b415-fe844ae7c1e7\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-9gvtf"
Nov 03 20:16:26.047680 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:26.047665    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e7f17ab-0f1d-4a59-b415-fe844ae7c1e7-istio-podinfo\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-9gvtf\" (UID: \"5e7f17ab-0f1d-4a59-b415-fe844ae7c1e7\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-9gvtf"
Nov 03 20:16:26.047863 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:26.047835    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e7f17ab-0f1d-4a59-b415-fe844ae7c1e7-workload-certs\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-9gvtf\" (UID: \"5e7f17ab-0f1d-4a59-b415-fe844ae7c1e7\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-9gvtf"
Nov 03 20:16:26.048138 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:26.048118    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e7f17ab-0f1d-4a59-b415-fe844ae7c1e7-istiod-ca-cert\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-9gvtf\" (UID: \"5e7f17ab-0f1d-4a59-b415-fe844ae7c1e7\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-9gvtf"
Nov 03 20:16:26.049233 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:26.049208    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e7f17ab-0f1d-4a59-b415-fe844ae7c1e7-workload-certs\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-9gvtf\" (UID: \"5e7f17ab-0f1d-4a59-b415-fe844ae7c1e7\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-9gvtf"
Nov 03 20:16:26.049599 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:26.049573    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e7f17ab-0f1d-4a59-b415-fe844ae7c1e7-istio-data\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-9gvtf\" (UID: \"5e7f17ab-0f1d-4a59-b415-fe844ae7c1e7\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-9gvtf"
Nov 03 20:16:26.050016 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:26.049987    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e7f17ab-0f1d-4a59-b415-fe844ae7c1e7-workload-socket\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-9gvtf\" (UID: \"5e7f17ab-0f1d-4a59-b415-fe844ae7c1e7\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-9gvtf"
Nov 03 20:16:26.050790 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:26.050741    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e7f17ab-0f1d-4a59-b415-fe844ae7c1e7-credential-socket\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-9gvtf\" (UID: \"5e7f17ab-0f1d-4a59-b415-fe844ae7c1e7\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-9gvtf"
Nov 03 20:16:26.051612 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:26.051589    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e7f17ab-0f1d-4a59-b415-fe844ae7c1e7-istio-envoy\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-9gvtf\" (UID: \"5e7f17ab-0f1d-4a59-b415-fe844ae7c1e7\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-9gvtf"
Nov 03 20:16:26.056957 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:26.056921    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e7f17ab-0f1d-4a59-b415-fe844ae7c1e7-istio-podinfo\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-9gvtf\" (UID: \"5e7f17ab-0f1d-4a59-b415-fe844ae7c1e7\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-9gvtf"
Nov 03 20:16:26.082567 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:26.082515    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e7f17ab-0f1d-4a59-b415-fe844ae7c1e7-istio-token\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-9gvtf\" (UID: \"5e7f17ab-0f1d-4a59-b415-fe844ae7c1e7\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-9gvtf"
Nov 03 20:16:26.153415 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:26.153378    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e7f17ab-0f1d-4a59-b415-fe844ae7c1e7-istiod-ca-cert\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-9gvtf\" (UID: \"5e7f17ab-0f1d-4a59-b415-fe844ae7c1e7\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-9gvtf"
Nov 03 20:16:26.153874 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:26.153849    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msl98\" (UniqueName: \"kubernetes.io/projected/5e7f17ab-0f1d-4a59-b415-fe844ae7c1e7-kube-api-access-msl98\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-9gvtf\" (UID: \"5e7f17ab-0f1d-4a59-b415-fe844ae7c1e7\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-9gvtf"
Nov 03 20:16:26.155275 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:26.155248    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e7f17ab-0f1d-4a59-b415-fe844ae7c1e7-istiod-ca-cert\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-9gvtf\" (UID: \"5e7f17ab-0f1d-4a59-b415-fe844ae7c1e7\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-9gvtf"
Nov 03 20:16:26.200811 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:26.198010    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-msl98\" (UniqueName: \"kubernetes.io/projected/5e7f17ab-0f1d-4a59-b415-fe844ae7c1e7-kube-api-access-msl98\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-9gvtf\" (UID: \"5e7f17ab-0f1d-4a59-b415-fe844ae7c1e7\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-9gvtf"
Nov 03 20:16:26.203479 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:26.202004    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-9gvtf"
Nov 03 20:16:26.819918 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:26.819877    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-9gvtf"]
Nov 03 20:16:26.823080 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:16:26.823049    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e7f17ab_0f1d_4a59_b415_fe844ae7c1e7.slice/crio-81ea73746ad1a8b753bc3e054f74a6e5546fcc93e882a572e9488bed4c2fc2f6 WatchSource:0}: Error finding container 81ea73746ad1a8b753bc3e054f74a6e5546fcc93e882a572e9488bed4c2fc2f6: Status 404 returned error can't find the container with id 81ea73746ad1a8b753bc3e054f74a6e5546fcc93e882a572e9488bed4c2fc2f6
Nov 03 20:16:26.826061 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:26.826021    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:16:26.826135 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:26.826122    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:16:26.826181 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:26.826168    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:16:27.231494 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:16:27.231455    5570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8bwhz" containerName="extractor"
Nov 03 20:16:27.347554 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:27.347516    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-e40816/simulator-w0-e40816-kserve-688f79796f-2tvw5"
Nov 03 20:16:27.366150 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:27.366121    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-e40816/simulator-w0-e40816-kserve-688f79796f-2tvw5"
Nov 03 20:16:27.633986 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:27.633883    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-9gvtf" event={"ID":"5e7f17ab-0f1d-4a59-b415-fe844ae7c1e7","Type":"ContainerStarted","Data":"3e7c2142d5118a937b9c4c1db78402b22ba93d6da618fb6dea1c3717ae2ab38b"}
Nov 03 20:16:27.633986 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:27.633954    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-9gvtf" event={"ID":"5e7f17ab-0f1d-4a59-b415-fe844ae7c1e7","Type":"ContainerStarted","Data":"81ea73746ad1a8b753bc3e054f74a6e5546fcc93e882a572e9488bed4c2fc2f6"}
Nov 03 20:16:27.666836 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:27.664012    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e40816/simulator-w0-e40816-kserve-688f79796f-2tvw5"
Nov 03 20:16:27.681854 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:27.681761    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-9gvtf" podStartSLOduration=2.681747085 podStartE2EDuration="2.681747085s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:16:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:16:27.680484824 +0000 UTC m=+721.833146713" watchObservedRunningTime="2026-11-03 20:16:27.681747085 +0000 UTC m=+721.834408973"
Nov 03 20:16:28.203570 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:28.203532    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-9gvtf"
Nov 03 20:16:28.205066 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:28.205043    5570 patch_prober.go:29] interesting pod/e2e-worker-w1-6fc748-openshift-default-665546959f-9gvtf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Nov 03 20:16:28.205131 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:28.205111    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-9gvtf" podUID="5e7f17ab-0f1d-4a59-b415-fe844ae7c1e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Nov 03 20:16:29.203186 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:29.203148    5570 patch_prober.go:29] interesting pod/e2e-worker-w1-6fc748-openshift-default-665546959f-9gvtf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Nov 03 20:16:29.203388 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:29.203219    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-9gvtf" podUID="5e7f17ab-0f1d-4a59-b415-fe844ae7c1e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Nov 03 20:16:30.203094 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:30.203055    5570 patch_prober.go:29] interesting pod/e2e-worker-w1-6fc748-openshift-default-665546959f-9gvtf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Nov 03 20:16:30.203615 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:30.203128    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-9gvtf" podUID="5e7f17ab-0f1d-4a59-b415-fe844ae7c1e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Nov 03 20:16:31.203126 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:31.203091    5570 patch_prober.go:29] interesting pod/e2e-worker-w1-6fc748-openshift-default-665546959f-9gvtf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Nov 03 20:16:31.203576 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:31.203145    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-9gvtf" podUID="5e7f17ab-0f1d-4a59-b415-fe844ae7c1e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Nov 03 20:16:32.202943 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:32.202902    5570 patch_prober.go:29] interesting pod/e2e-worker-w1-6fc748-openshift-default-665546959f-9gvtf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Nov 03 20:16:32.203140 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:32.202964    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-9gvtf" podUID="5e7f17ab-0f1d-4a59-b415-fe844ae7c1e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Nov 03 20:16:32.590135 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:32.590038    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-e80203/premium-w4-e80203-kserve-5f9f445456-bms9d"
Nov 03 20:16:32.602587 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:32.602561    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-e80203/premium-w4-e80203-kserve-5f9f445456-bms9d"
Nov 03 20:16:32.698310 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:32.698280    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-e80203/premium-w4-e80203-kserve-5f9f445456-bms9d"
Nov 03 20:16:33.203152 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:33.203115    5570 patch_prober.go:29] interesting pod/e2e-worker-w1-6fc748-openshift-default-665546959f-9gvtf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Nov 03 20:16:33.203347 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:33.203170    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-9gvtf" podUID="5e7f17ab-0f1d-4a59-b415-fe844ae7c1e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Nov 03 20:16:34.208464 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:34.208427    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-9gvtf"
Nov 03 20:16:34.209058 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:34.208979    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-9gvtf"
Nov 03 20:16:34.210060 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:34.210036    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-9gvtf"
Nov 03 20:16:34.225404 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:34.225372    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-482722/premium-w3-482722-kserve-689dc7bd5d-x8rjp"]
Nov 03 20:16:34.229165 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:34.229142    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-482722/premium-w3-482722-kserve-689dc7bd5d-x8rjp"
Nov 03 20:16:34.246385 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:34.246357    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-482722/premium-w3-482722-kserve-689dc7bd5d-x8rjp"]
Nov 03 20:16:34.247249 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:34.247228    5570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-482722\"/\"kube-root-ca.crt\""
Nov 03 20:16:34.251366 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:34.251341    5570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-482722\"/\"openshift-service-ca.crt\""
Nov 03 20:16:34.251366 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:34.251349    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-482722\"/\"default-dockercfg-hcpz2\""
Nov 03 20:16:34.251506 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:34.251373    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-482722\"/\"premium-w3-482722-kserve-self-signed-certs\""
Nov 03 20:16:34.328299 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:34.328260    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/369045c3-4d6a-49a7-9bc6-f2939883bee8-home\") pod \"premium-w3-482722-kserve-689dc7bd5d-x8rjp\" (UID: \"369045c3-4d6a-49a7-9bc6-f2939883bee8\") " pod="e2e-models-e2e-worker-w3-482722/premium-w3-482722-kserve-689dc7bd5d-x8rjp"
Nov 03 20:16:34.328517 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:34.328321    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/369045c3-4d6a-49a7-9bc6-f2939883bee8-tmp-dir\") pod \"premium-w3-482722-kserve-689dc7bd5d-x8rjp\" (UID: \"369045c3-4d6a-49a7-9bc6-f2939883bee8\") " pod="e2e-models-e2e-worker-w3-482722/premium-w3-482722-kserve-689dc7bd5d-x8rjp"
Nov 03 20:16:34.328517 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:34.328422    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/369045c3-4d6a-49a7-9bc6-f2939883bee8-model-cache\") pod \"premium-w3-482722-kserve-689dc7bd5d-x8rjp\" (UID: \"369045c3-4d6a-49a7-9bc6-f2939883bee8\") " pod="e2e-models-e2e-worker-w3-482722/premium-w3-482722-kserve-689dc7bd5d-x8rjp"
Nov 03 20:16:34.328646 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:34.328536    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hmzj\" (UniqueName: \"kubernetes.io/projected/369045c3-4d6a-49a7-9bc6-f2939883bee8-kube-api-access-6hmzj\") pod \"premium-w3-482722-kserve-689dc7bd5d-x8rjp\" (UID: \"369045c3-4d6a-49a7-9bc6-f2939883bee8\") " pod="e2e-models-e2e-worker-w3-482722/premium-w3-482722-kserve-689dc7bd5d-x8rjp"
Nov 03 20:16:34.328707 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:34.328653    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/369045c3-4d6a-49a7-9bc6-f2939883bee8-tls-certs\") pod \"premium-w3-482722-kserve-689dc7bd5d-x8rjp\" (UID: \"369045c3-4d6a-49a7-9bc6-f2939883bee8\") " pod="e2e-models-e2e-worker-w3-482722/premium-w3-482722-kserve-689dc7bd5d-x8rjp"
Nov 03 20:16:34.328903 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:34.328880    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/369045c3-4d6a-49a7-9bc6-f2939883bee8-dshm\") pod \"premium-w3-482722-kserve-689dc7bd5d-x8rjp\" (UID: \"369045c3-4d6a-49a7-9bc6-f2939883bee8\") " pod="e2e-models-e2e-worker-w3-482722/premium-w3-482722-kserve-689dc7bd5d-x8rjp"
Nov 03 20:16:34.430537 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:34.430508    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/369045c3-4d6a-49a7-9bc6-f2939883bee8-tls-certs\") pod \"premium-w3-482722-kserve-689dc7bd5d-x8rjp\" (UID: \"369045c3-4d6a-49a7-9bc6-f2939883bee8\") " pod="e2e-models-e2e-worker-w3-482722/premium-w3-482722-kserve-689dc7bd5d-x8rjp"
Nov 03 20:16:34.430719 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:34.430659    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/369045c3-4d6a-49a7-9bc6-f2939883bee8-dshm\") pod \"premium-w3-482722-kserve-689dc7bd5d-x8rjp\" (UID: \"369045c3-4d6a-49a7-9bc6-f2939883bee8\") " pod="e2e-models-e2e-worker-w3-482722/premium-w3-482722-kserve-689dc7bd5d-x8rjp"
Nov 03 20:16:34.430829 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:34.430742    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/369045c3-4d6a-49a7-9bc6-f2939883bee8-home\") pod \"premium-w3-482722-kserve-689dc7bd5d-x8rjp\" (UID: \"369045c3-4d6a-49a7-9bc6-f2939883bee8\") " pod="e2e-models-e2e-worker-w3-482722/premium-w3-482722-kserve-689dc7bd5d-x8rjp"
Nov 03 20:16:34.430829 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:34.430769    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/369045c3-4d6a-49a7-9bc6-f2939883bee8-tmp-dir\") pod \"premium-w3-482722-kserve-689dc7bd5d-x8rjp\" (UID: \"369045c3-4d6a-49a7-9bc6-f2939883bee8\") " pod="e2e-models-e2e-worker-w3-482722/premium-w3-482722-kserve-689dc7bd5d-x8rjp"
Nov 03 20:16:34.430829 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:34.430809    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/369045c3-4d6a-49a7-9bc6-f2939883bee8-model-cache\") pod \"premium-w3-482722-kserve-689dc7bd5d-x8rjp\" (UID: \"369045c3-4d6a-49a7-9bc6-f2939883bee8\") " pod="e2e-models-e2e-worker-w3-482722/premium-w3-482722-kserve-689dc7bd5d-x8rjp"
Nov 03 20:16:34.430996 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:34.430845    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hmzj\" (UniqueName: \"kubernetes.io/projected/369045c3-4d6a-49a7-9bc6-f2939883bee8-kube-api-access-6hmzj\") pod \"premium-w3-482722-kserve-689dc7bd5d-x8rjp\" (UID: \"369045c3-4d6a-49a7-9bc6-f2939883bee8\") " pod="e2e-models-e2e-worker-w3-482722/premium-w3-482722-kserve-689dc7bd5d-x8rjp"
Nov 03 20:16:34.431199 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:34.431178    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/369045c3-4d6a-49a7-9bc6-f2939883bee8-home\") pod \"premium-w3-482722-kserve-689dc7bd5d-x8rjp\" (UID: \"369045c3-4d6a-49a7-9bc6-f2939883bee8\") " pod="e2e-models-e2e-worker-w3-482722/premium-w3-482722-kserve-689dc7bd5d-x8rjp"
Nov 03 20:16:34.431295 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:34.431221    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/369045c3-4d6a-49a7-9bc6-f2939883bee8-tmp-dir\") pod \"premium-w3-482722-kserve-689dc7bd5d-x8rjp\" (UID: \"369045c3-4d6a-49a7-9bc6-f2939883bee8\") " pod="e2e-models-e2e-worker-w3-482722/premium-w3-482722-kserve-689dc7bd5d-x8rjp"
Nov 03 20:16:34.432947 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:34.432920    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/369045c3-4d6a-49a7-9bc6-f2939883bee8-model-cache\") pod \"premium-w3-482722-kserve-689dc7bd5d-x8rjp\" (UID: \"369045c3-4d6a-49a7-9bc6-f2939883bee8\") " pod="e2e-models-e2e-worker-w3-482722/premium-w3-482722-kserve-689dc7bd5d-x8rjp"
Nov 03 20:16:34.432947 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:34.432939    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/369045c3-4d6a-49a7-9bc6-f2939883bee8-dshm\") pod \"premium-w3-482722-kserve-689dc7bd5d-x8rjp\" (UID: \"369045c3-4d6a-49a7-9bc6-f2939883bee8\") " pod="e2e-models-e2e-worker-w3-482722/premium-w3-482722-kserve-689dc7bd5d-x8rjp"
Nov 03 20:16:34.433329 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:34.433306    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/369045c3-4d6a-49a7-9bc6-f2939883bee8-tls-certs\") pod \"premium-w3-482722-kserve-689dc7bd5d-x8rjp\" (UID: \"369045c3-4d6a-49a7-9bc6-f2939883bee8\") " pod="e2e-models-e2e-worker-w3-482722/premium-w3-482722-kserve-689dc7bd5d-x8rjp"
Nov 03 20:16:34.451504 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:34.451477    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hmzj\" (UniqueName: \"kubernetes.io/projected/369045c3-4d6a-49a7-9bc6-f2939883bee8-kube-api-access-6hmzj\") pod \"premium-w3-482722-kserve-689dc7bd5d-x8rjp\" (UID: \"369045c3-4d6a-49a7-9bc6-f2939883bee8\") " pod="e2e-models-e2e-worker-w3-482722/premium-w3-482722-kserve-689dc7bd5d-x8rjp"
Nov 03 20:16:34.543057 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:34.542974    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-482722/premium-w3-482722-kserve-689dc7bd5d-x8rjp"
Nov 03 20:16:34.844893 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:34.844867    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-482722/premium-w3-482722-kserve-689dc7bd5d-x8rjp"]
Nov 03 20:16:34.846208 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:16:34.846174    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod369045c3_4d6a_49a7_9bc6_f2939883bee8.slice/crio-76c9987e584a0ea357a472e354bbea18668b8836d33e3679c1235c6c19940246 WatchSource:0}: Error finding container 76c9987e584a0ea357a472e354bbea18668b8836d33e3679c1235c6c19940246: Status 404 returned error can't find the container with id 76c9987e584a0ea357a472e354bbea18668b8836d33e3679c1235c6c19940246
Nov 03 20:16:35.078196 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:35.078152    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g69f8"]
Nov 03 20:16:35.705561 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:35.705524    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-482722/premium-w3-482722-kserve-689dc7bd5d-x8rjp" event={"ID":"369045c3-4d6a-49a7-9bc6-f2939883bee8","Type":"ContainerStarted","Data":"ffcbc1a7b9d7c549e41b0ed6f5ea4c2ee7c624ba331bc58642873ce48c298f3b"}
Nov 03 20:16:35.705561 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:35.705567    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-482722/premium-w3-482722-kserve-689dc7bd5d-x8rjp" event={"ID":"369045c3-4d6a-49a7-9bc6-f2939883bee8","Type":"ContainerStarted","Data":"76c9987e584a0ea357a472e354bbea18668b8836d33e3679c1235c6c19940246"}
Nov 03 20:16:35.748826 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:35.748760    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-482722/premium-w3-482722-kserve-689dc7bd5d-x8rjp" podStartSLOduration=1.748745961 podStartE2EDuration="1.748745961s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:16:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:16:35.747425317 +0000 UTC m=+729.900087211" watchObservedRunningTime="2026-11-03 20:16:35.748745961 +0000 UTC m=+729.901407911"
Nov 03 20:16:36.543907 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:36.543867    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-482722/premium-w3-482722-kserve-689dc7bd5d-x8rjp"
Nov 03 20:16:40.865263 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:40.865226    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-jbtqd"]
Nov 03 20:16:40.869265 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:40.869239    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-jbtqd"
Nov 03 20:16:40.902588 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:40.902542    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-jbtqd"]
Nov 03 20:16:41.031732 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:41.031691    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/baacee7f-b912-4964-9855-be2beb0094cb-workload-socket\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-jbtqd\" (UID: \"baacee7f-b912-4964-9855-be2beb0094cb\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-jbtqd"
Nov 03 20:16:41.031910 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:41.031839    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/baacee7f-b912-4964-9855-be2beb0094cb-istiod-ca-cert\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-jbtqd\" (UID: \"baacee7f-b912-4964-9855-be2beb0094cb\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-jbtqd"
Nov 03 20:16:41.031910 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:41.031882    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cw925\" (UniqueName: \"kubernetes.io/projected/baacee7f-b912-4964-9855-be2beb0094cb-kube-api-access-cw925\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-jbtqd\" (UID: \"baacee7f-b912-4964-9855-be2beb0094cb\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-jbtqd"
Nov 03 20:16:41.031910 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:41.031905    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/baacee7f-b912-4964-9855-be2beb0094cb-istio-data\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-jbtqd\" (UID: \"baacee7f-b912-4964-9855-be2beb0094cb\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-jbtqd"
Nov 03 20:16:41.032039 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:41.031921    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/baacee7f-b912-4964-9855-be2beb0094cb-istio-token\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-jbtqd\" (UID: \"baacee7f-b912-4964-9855-be2beb0094cb\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-jbtqd"
Nov 03 20:16:41.032039 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:41.032001    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/baacee7f-b912-4964-9855-be2beb0094cb-istio-envoy\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-jbtqd\" (UID: \"baacee7f-b912-4964-9855-be2beb0094cb\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-jbtqd"
Nov 03 20:16:41.032158 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:41.032136    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/baacee7f-b912-4964-9855-be2beb0094cb-credential-socket\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-jbtqd\" (UID: \"baacee7f-b912-4964-9855-be2beb0094cb\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-jbtqd"
Nov 03 20:16:41.032297 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:41.032281    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/baacee7f-b912-4964-9855-be2beb0094cb-istio-podinfo\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-jbtqd\" (UID: \"baacee7f-b912-4964-9855-be2beb0094cb\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-jbtqd"
Nov 03 20:16:41.032350 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:41.032335    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/baacee7f-b912-4964-9855-be2beb0094cb-workload-certs\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-jbtqd\" (UID: \"baacee7f-b912-4964-9855-be2beb0094cb\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-jbtqd"
Nov 03 20:16:41.133470 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:41.133373    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/baacee7f-b912-4964-9855-be2beb0094cb-istiod-ca-cert\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-jbtqd\" (UID: \"baacee7f-b912-4964-9855-be2beb0094cb\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-jbtqd"
Nov 03 20:16:41.133737 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:41.133702    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cw925\" (UniqueName: \"kubernetes.io/projected/baacee7f-b912-4964-9855-be2beb0094cb-kube-api-access-cw925\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-jbtqd\" (UID: \"baacee7f-b912-4964-9855-be2beb0094cb\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-jbtqd"
Nov 03 20:16:41.133907 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:41.133752    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/baacee7f-b912-4964-9855-be2beb0094cb-istio-data\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-jbtqd\" (UID: \"baacee7f-b912-4964-9855-be2beb0094cb\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-jbtqd"
Nov 03 20:16:41.133907 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:41.133793    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/baacee7f-b912-4964-9855-be2beb0094cb-istio-token\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-jbtqd\" (UID: \"baacee7f-b912-4964-9855-be2beb0094cb\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-jbtqd"
Nov 03 20:16:41.133907 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:41.133836    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/baacee7f-b912-4964-9855-be2beb0094cb-istio-envoy\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-jbtqd\" (UID: \"baacee7f-b912-4964-9855-be2beb0094cb\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-jbtqd"
Nov 03 20:16:41.133907 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:41.133886    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/baacee7f-b912-4964-9855-be2beb0094cb-credential-socket\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-jbtqd\" (UID: \"baacee7f-b912-4964-9855-be2beb0094cb\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-jbtqd"
Nov 03 20:16:41.134138 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:41.133980    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/baacee7f-b912-4964-9855-be2beb0094cb-istio-podinfo\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-jbtqd\" (UID: \"baacee7f-b912-4964-9855-be2beb0094cb\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-jbtqd"
Nov 03 20:16:41.134138 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:41.134027    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/baacee7f-b912-4964-9855-be2beb0094cb-workload-certs\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-jbtqd\" (UID: \"baacee7f-b912-4964-9855-be2beb0094cb\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-jbtqd"
Nov 03 20:16:41.134138 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:41.134080    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/baacee7f-b912-4964-9855-be2beb0094cb-workload-socket\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-jbtqd\" (UID: \"baacee7f-b912-4964-9855-be2beb0094cb\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-jbtqd"
Nov 03 20:16:41.134300 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:41.134276    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/baacee7f-b912-4964-9855-be2beb0094cb-istiod-ca-cert\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-jbtqd\" (UID: \"baacee7f-b912-4964-9855-be2beb0094cb\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-jbtqd"
Nov 03 20:16:41.134523 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:41.134497    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/baacee7f-b912-4964-9855-be2beb0094cb-workload-socket\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-jbtqd\" (UID: \"baacee7f-b912-4964-9855-be2beb0094cb\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-jbtqd"
Nov 03 20:16:41.135273 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:41.135110    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/baacee7f-b912-4964-9855-be2beb0094cb-credential-socket\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-jbtqd\" (UID: \"baacee7f-b912-4964-9855-be2beb0094cb\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-jbtqd"
Nov 03 20:16:41.135673 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:41.135652    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/baacee7f-b912-4964-9855-be2beb0094cb-istio-data\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-jbtqd\" (UID: \"baacee7f-b912-4964-9855-be2beb0094cb\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-jbtqd"
Nov 03 20:16:41.135820 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:41.135721    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/baacee7f-b912-4964-9855-be2beb0094cb-workload-certs\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-jbtqd\" (UID: \"baacee7f-b912-4964-9855-be2beb0094cb\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-jbtqd"
Nov 03 20:16:41.137550 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:41.137526    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/baacee7f-b912-4964-9855-be2beb0094cb-istio-envoy\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-jbtqd\" (UID: \"baacee7f-b912-4964-9855-be2beb0094cb\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-jbtqd"
Nov 03 20:16:41.137752 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:41.137729    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/baacee7f-b912-4964-9855-be2beb0094cb-istio-podinfo\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-jbtqd\" (UID: \"baacee7f-b912-4964-9855-be2beb0094cb\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-jbtqd"
Nov 03 20:16:41.155419 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:41.155391    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/baacee7f-b912-4964-9855-be2beb0094cb-istio-token\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-jbtqd\" (UID: \"baacee7f-b912-4964-9855-be2beb0094cb\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-jbtqd"
Nov 03 20:16:41.155586 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:41.155523    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cw925\" (UniqueName: \"kubernetes.io/projected/baacee7f-b912-4964-9855-be2beb0094cb-kube-api-access-cw925\") pod \"e2e-worker-w1-6fc748-openshift-default-665546959f-jbtqd\" (UID: \"baacee7f-b912-4964-9855-be2beb0094cb\") " pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-jbtqd"
Nov 03 20:16:41.186022 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:41.185986    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-jbtqd"
Nov 03 20:16:41.509840 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:41.509811    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-jbtqd"]
Nov 03 20:16:41.512009 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:16:41.511979    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbaacee7f_b912_4964_9855_be2beb0094cb.slice/crio-b48d415fd6952fb133ce8433e58075bd9c80126def429374aa8d77c457723a20 WatchSource:0}: Error finding container b48d415fd6952fb133ce8433e58075bd9c80126def429374aa8d77c457723a20: Status 404 returned error can't find the container with id b48d415fd6952fb133ce8433e58075bd9c80126def429374aa8d77c457723a20
Nov 03 20:16:41.514394 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:41.514364    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:16:41.514491 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:41.514430    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:16:41.514491 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:41.514460    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:16:41.776158 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:41.776062    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-jbtqd" event={"ID":"baacee7f-b912-4964-9855-be2beb0094cb","Type":"ContainerStarted","Data":"f88e91ce1e2d9b540bdd3ec4f7167d68046dc7bc6e871d1e1f9efc08d87c6965"}
Nov 03 20:16:41.776158 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:41.776125    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-jbtqd" event={"ID":"baacee7f-b912-4964-9855-be2beb0094cb","Type":"ContainerStarted","Data":"b48d415fd6952fb133ce8433e58075bd9c80126def429374aa8d77c457723a20"}
Nov 03 20:16:41.824813 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:41.824737    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-jbtqd" podStartSLOduration=1.824720822 podStartE2EDuration="1.824720822s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:16:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:16:41.823852822 +0000 UTC m=+735.976514709" watchObservedRunningTime="2026-11-03 20:16:41.824720822 +0000 UTC m=+735.977382709"
Nov 03 20:16:42.186395 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:42.186301    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-jbtqd"
Nov 03 20:16:42.188248 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:42.188213    5570 patch_prober.go:29] interesting pod/e2e-worker-w1-6fc748-openshift-default-665546959f-jbtqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Nov 03 20:16:42.188414 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:42.188265    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-jbtqd" podUID="baacee7f-b912-4964-9855-be2beb0094cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Nov 03 20:16:43.186712 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:43.186679    5570 patch_prober.go:29] interesting pod/e2e-worker-w1-6fc748-openshift-default-665546959f-jbtqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Nov 03 20:16:43.187139 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:43.186733    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-jbtqd" podUID="baacee7f-b912-4964-9855-be2beb0094cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Nov 03 20:16:43.505696 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:43.505655    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-9973d9/premium-w2-9973d9-kserve-5c6dd4669c-5gv8x"]
Nov 03 20:16:43.510670 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:43.510639    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-9973d9/premium-w2-9973d9-kserve-5c6dd4669c-5gv8x"
Nov 03 20:16:43.527574 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:43.527547    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-9973d9/premium-w2-9973d9-kserve-5c6dd4669c-5gv8x"]
Nov 03 20:16:43.552821 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:43.552179    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-9973d9\"/\"default-dockercfg-rxqtx\""
Nov 03 20:16:43.553055 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:43.552946    5570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-9973d9\"/\"kube-root-ca.crt\""
Nov 03 20:16:43.553055 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:43.553035    5570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-9973d9\"/\"openshift-service-ca.crt\""
Nov 03 20:16:43.553206 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:43.553147    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-9973d9\"/\"premium-w2-9973d9-kserve-self-signed-certs\""
Nov 03 20:16:43.675623 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:43.675592    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cefc89ad-7ae1-433b-979d-6601678a6d98-dshm\") pod \"premium-w2-9973d9-kserve-5c6dd4669c-5gv8x\" (UID: \"cefc89ad-7ae1-433b-979d-6601678a6d98\") " pod="e2e-models-e2e-worker-w2-9973d9/premium-w2-9973d9-kserve-5c6dd4669c-5gv8x"
Nov 03 20:16:43.675861 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:43.675652    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cefc89ad-7ae1-433b-979d-6601678a6d98-model-cache\") pod \"premium-w2-9973d9-kserve-5c6dd4669c-5gv8x\" (UID: \"cefc89ad-7ae1-433b-979d-6601678a6d98\") " pod="e2e-models-e2e-worker-w2-9973d9/premium-w2-9973d9-kserve-5c6dd4669c-5gv8x"
Nov 03 20:16:43.675861 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:43.675703    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqth8\" (UniqueName: \"kubernetes.io/projected/cefc89ad-7ae1-433b-979d-6601678a6d98-kube-api-access-xqth8\") pod \"premium-w2-9973d9-kserve-5c6dd4669c-5gv8x\" (UID: \"cefc89ad-7ae1-433b-979d-6601678a6d98\") " pod="e2e-models-e2e-worker-w2-9973d9/premium-w2-9973d9-kserve-5c6dd4669c-5gv8x"
Nov 03 20:16:43.675861 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:43.675813    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cefc89ad-7ae1-433b-979d-6601678a6d98-tmp-dir\") pod \"premium-w2-9973d9-kserve-5c6dd4669c-5gv8x\" (UID: \"cefc89ad-7ae1-433b-979d-6601678a6d98\") " pod="e2e-models-e2e-worker-w2-9973d9/premium-w2-9973d9-kserve-5c6dd4669c-5gv8x"
Nov 03 20:16:43.675861 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:43.675851    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cefc89ad-7ae1-433b-979d-6601678a6d98-tls-certs\") pod \"premium-w2-9973d9-kserve-5c6dd4669c-5gv8x\" (UID: \"cefc89ad-7ae1-433b-979d-6601678a6d98\") " pod="e2e-models-e2e-worker-w2-9973d9/premium-w2-9973d9-kserve-5c6dd4669c-5gv8x"
Nov 03 20:16:43.676022 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:43.675942    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cefc89ad-7ae1-433b-979d-6601678a6d98-home\") pod \"premium-w2-9973d9-kserve-5c6dd4669c-5gv8x\" (UID: \"cefc89ad-7ae1-433b-979d-6601678a6d98\") " pod="e2e-models-e2e-worker-w2-9973d9/premium-w2-9973d9-kserve-5c6dd4669c-5gv8x"
Nov 03 20:16:43.777861 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:43.777568    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cefc89ad-7ae1-433b-979d-6601678a6d98-dshm\") pod \"premium-w2-9973d9-kserve-5c6dd4669c-5gv8x\" (UID: \"cefc89ad-7ae1-433b-979d-6601678a6d98\") " pod="e2e-models-e2e-worker-w2-9973d9/premium-w2-9973d9-kserve-5c6dd4669c-5gv8x"
Nov 03 20:16:43.777861 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:43.777676    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cefc89ad-7ae1-433b-979d-6601678a6d98-model-cache\") pod \"premium-w2-9973d9-kserve-5c6dd4669c-5gv8x\" (UID: \"cefc89ad-7ae1-433b-979d-6601678a6d98\") " pod="e2e-models-e2e-worker-w2-9973d9/premium-w2-9973d9-kserve-5c6dd4669c-5gv8x"
Nov 03 20:16:43.777861 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:43.777717    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqth8\" (UniqueName: \"kubernetes.io/projected/cefc89ad-7ae1-433b-979d-6601678a6d98-kube-api-access-xqth8\") pod \"premium-w2-9973d9-kserve-5c6dd4669c-5gv8x\" (UID: \"cefc89ad-7ae1-433b-979d-6601678a6d98\") " pod="e2e-models-e2e-worker-w2-9973d9/premium-w2-9973d9-kserve-5c6dd4669c-5gv8x"
Nov 03 20:16:43.777861 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:43.777788    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cefc89ad-7ae1-433b-979d-6601678a6d98-tmp-dir\") pod \"premium-w2-9973d9-kserve-5c6dd4669c-5gv8x\" (UID: \"cefc89ad-7ae1-433b-979d-6601678a6d98\") " pod="e2e-models-e2e-worker-w2-9973d9/premium-w2-9973d9-kserve-5c6dd4669c-5gv8x"
Nov 03 20:16:43.777861 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:43.777828    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cefc89ad-7ae1-433b-979d-6601678a6d98-tls-certs\") pod \"premium-w2-9973d9-kserve-5c6dd4669c-5gv8x\" (UID: \"cefc89ad-7ae1-433b-979d-6601678a6d98\") " pod="e2e-models-e2e-worker-w2-9973d9/premium-w2-9973d9-kserve-5c6dd4669c-5gv8x"
Nov 03 20:16:43.778295 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:43.777930    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cefc89ad-7ae1-433b-979d-6601678a6d98-home\") pod \"premium-w2-9973d9-kserve-5c6dd4669c-5gv8x\" (UID: \"cefc89ad-7ae1-433b-979d-6601678a6d98\") " pod="e2e-models-e2e-worker-w2-9973d9/premium-w2-9973d9-kserve-5c6dd4669c-5gv8x"
Nov 03 20:16:43.778610 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:43.778532    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cefc89ad-7ae1-433b-979d-6601678a6d98-home\") pod \"premium-w2-9973d9-kserve-5c6dd4669c-5gv8x\" (UID: \"cefc89ad-7ae1-433b-979d-6601678a6d98\") " pod="e2e-models-e2e-worker-w2-9973d9/premium-w2-9973d9-kserve-5c6dd4669c-5gv8x"
Nov 03 20:16:43.788861 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:43.779544    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cefc89ad-7ae1-433b-979d-6601678a6d98-model-cache\") pod \"premium-w2-9973d9-kserve-5c6dd4669c-5gv8x\" (UID: \"cefc89ad-7ae1-433b-979d-6601678a6d98\") " pod="e2e-models-e2e-worker-w2-9973d9/premium-w2-9973d9-kserve-5c6dd4669c-5gv8x"
Nov 03 20:16:43.788861 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:43.779820    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cefc89ad-7ae1-433b-979d-6601678a6d98-tmp-dir\") pod \"premium-w2-9973d9-kserve-5c6dd4669c-5gv8x\" (UID: \"cefc89ad-7ae1-433b-979d-6601678a6d98\") " pod="e2e-models-e2e-worker-w2-9973d9/premium-w2-9973d9-kserve-5c6dd4669c-5gv8x"
Nov 03 20:16:43.790598 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:43.790549    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cefc89ad-7ae1-433b-979d-6601678a6d98-dshm\") pod \"premium-w2-9973d9-kserve-5c6dd4669c-5gv8x\" (UID: \"cefc89ad-7ae1-433b-979d-6601678a6d98\") " pod="e2e-models-e2e-worker-w2-9973d9/premium-w2-9973d9-kserve-5c6dd4669c-5gv8x"
Nov 03 20:16:43.792083 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:43.792044    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cefc89ad-7ae1-433b-979d-6601678a6d98-tls-certs\") pod \"premium-w2-9973d9-kserve-5c6dd4669c-5gv8x\" (UID: \"cefc89ad-7ae1-433b-979d-6601678a6d98\") " pod="e2e-models-e2e-worker-w2-9973d9/premium-w2-9973d9-kserve-5c6dd4669c-5gv8x"
Nov 03 20:16:43.809708 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:43.809672    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqth8\" (UniqueName: \"kubernetes.io/projected/cefc89ad-7ae1-433b-979d-6601678a6d98-kube-api-access-xqth8\") pod \"premium-w2-9973d9-kserve-5c6dd4669c-5gv8x\" (UID: \"cefc89ad-7ae1-433b-979d-6601678a6d98\") " pod="e2e-models-e2e-worker-w2-9973d9/premium-w2-9973d9-kserve-5c6dd4669c-5gv8x"
Nov 03 20:16:43.825264 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:43.825228    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-9973d9/premium-w2-9973d9-kserve-5c6dd4669c-5gv8x"
Nov 03 20:16:44.164217 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:44.160659    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-9973d9/premium-w2-9973d9-kserve-5c6dd4669c-5gv8x"]
Nov 03 20:16:44.166246 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:16:44.166210    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcefc89ad_7ae1_433b_979d_6601678a6d98.slice/crio-31a655f8b91204ee832414b270ae7631816d077d59bb13d5dc61658121746f10 WatchSource:0}: Error finding container 31a655f8b91204ee832414b270ae7631816d077d59bb13d5dc61658121746f10: Status 404 returned error can't find the container with id 31a655f8b91204ee832414b270ae7631816d077d59bb13d5dc61658121746f10
Nov 03 20:16:44.187251 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:44.187219    5570 patch_prober.go:29] interesting pod/e2e-worker-w1-6fc748-openshift-default-665546959f-jbtqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Nov 03 20:16:44.187679 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:44.187269    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-jbtqd" podUID="baacee7f-b912-4964-9855-be2beb0094cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Nov 03 20:16:44.555262 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:44.550212    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-482722/premium-w3-482722-kserve-689dc7bd5d-x8rjp"
Nov 03 20:16:44.575012 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:44.574981    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-482722/premium-w3-482722-kserve-689dc7bd5d-x8rjp"
Nov 03 20:16:44.802761 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:44.802720    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-9973d9/premium-w2-9973d9-kserve-5c6dd4669c-5gv8x" event={"ID":"cefc89ad-7ae1-433b-979d-6601678a6d98","Type":"ContainerStarted","Data":"5621b0943b02c827ef26cd3d11392f6b882f3c7f2ffbbdb118669c90ff795faa"}
Nov 03 20:16:44.802973 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:44.802789    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-9973d9/premium-w2-9973d9-kserve-5c6dd4669c-5gv8x" event={"ID":"cefc89ad-7ae1-433b-979d-6601678a6d98","Type":"ContainerStarted","Data":"31a655f8b91204ee832414b270ae7631816d077d59bb13d5dc61658121746f10"}
Nov 03 20:16:44.814498 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:44.814418    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-482722/premium-w3-482722-kserve-689dc7bd5d-x8rjp"
Nov 03 20:16:44.825658 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:44.825629    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9973d9/premium-w2-9973d9-kserve-5c6dd4669c-5gv8x"
Nov 03 20:16:44.861796 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:44.861217    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-9973d9/premium-w2-9973d9-kserve-5c6dd4669c-5gv8x" podStartSLOduration=1.861197635 podStartE2EDuration="1.861197635s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:16:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:16:44.86038517 +0000 UTC m=+739.013047055" watchObservedRunningTime="2026-11-03 20:16:44.861197635 +0000 UTC m=+739.013859525"
Nov 03 20:16:45.187449 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:45.187423    5570 patch_prober.go:29] interesting pod/e2e-worker-w1-6fc748-openshift-default-665546959f-jbtqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Nov 03 20:16:45.187902 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:45.187475    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-jbtqd" podUID="baacee7f-b912-4964-9855-be2beb0094cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Nov 03 20:16:46.186745 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:46.186700    5570 patch_prober.go:29] interesting pod/e2e-worker-w1-6fc748-openshift-default-665546959f-jbtqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Nov 03 20:16:46.186940 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:46.186761    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-jbtqd" podUID="baacee7f-b912-4964-9855-be2beb0094cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Nov 03 20:16:47.186469 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:47.186426    5570 patch_prober.go:29] interesting pod/e2e-worker-w1-6fc748-openshift-default-665546959f-jbtqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Nov 03 20:16:47.186971 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:47.186485    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-jbtqd" podUID="baacee7f-b912-4964-9855-be2beb0094cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Nov 03 20:16:48.191618 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:48.191584    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-jbtqd"
Nov 03 20:16:48.192050 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:48.191995    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-jbtqd"
Nov 03 20:16:48.192647 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:48.192628    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-jbtqd"
Nov 03 20:16:48.763956 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:48.763922    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g69f8"]
Nov 03 20:16:53.825680 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:53.825645    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-9973d9/premium-w2-9973d9-kserve-5c6dd4669c-5gv8x"
Nov 03 20:16:53.839008 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:53.838982    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-9973d9/premium-w2-9973d9-kserve-5c6dd4669c-5gv8x"
Nov 03 20:16:53.873901 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:53.873869    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9973d9/premium-w2-9973d9-kserve-5c6dd4669c-5gv8x"
Nov 03 20:16:55.589496 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:55.589460    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-b4zmq"]
Nov 03 20:16:55.594535 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:55.594514    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-b4zmq"
Nov 03 20:16:55.640057 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:55.640018    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-b4zmq"]
Nov 03 20:16:55.674369 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:55.674328    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5kvd\" (UniqueName: \"kubernetes.io/projected/8f7ff673-520b-4dd1-a76d-0e7aa1053c3f-kube-api-access-x5kvd\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-b4zmq\" (UID: \"8f7ff673-520b-4dd1-a76d-0e7aa1053c3f\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-b4zmq"
Nov 03 20:16:55.674570 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:55.674387    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f7ff673-520b-4dd1-a76d-0e7aa1053c3f-workload-socket\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-b4zmq\" (UID: \"8f7ff673-520b-4dd1-a76d-0e7aa1053c3f\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-b4zmq"
Nov 03 20:16:55.674570 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:55.674432    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f7ff673-520b-4dd1-a76d-0e7aa1053c3f-istio-data\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-b4zmq\" (UID: \"8f7ff673-520b-4dd1-a76d-0e7aa1053c3f\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-b4zmq"
Nov 03 20:16:55.674982 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:55.674954    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f7ff673-520b-4dd1-a76d-0e7aa1053c3f-workload-certs\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-b4zmq\" (UID: \"8f7ff673-520b-4dd1-a76d-0e7aa1053c3f\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-b4zmq"
Nov 03 20:16:55.675098 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:55.675074    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f7ff673-520b-4dd1-a76d-0e7aa1053c3f-credential-socket\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-b4zmq\" (UID: \"8f7ff673-520b-4dd1-a76d-0e7aa1053c3f\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-b4zmq"
Nov 03 20:16:55.675162 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:55.675106    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f7ff673-520b-4dd1-a76d-0e7aa1053c3f-istio-podinfo\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-b4zmq\" (UID: \"8f7ff673-520b-4dd1-a76d-0e7aa1053c3f\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-b4zmq"
Nov 03 20:16:55.675227 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:55.675212    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f7ff673-520b-4dd1-a76d-0e7aa1053c3f-istiod-ca-cert\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-b4zmq\" (UID: \"8f7ff673-520b-4dd1-a76d-0e7aa1053c3f\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-b4zmq"
Nov 03 20:16:55.675283 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:55.675239    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f7ff673-520b-4dd1-a76d-0e7aa1053c3f-istio-envoy\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-b4zmq\" (UID: \"8f7ff673-520b-4dd1-a76d-0e7aa1053c3f\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-b4zmq"
Nov 03 20:16:55.675333 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:55.675289    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f7ff673-520b-4dd1-a76d-0e7aa1053c3f-istio-token\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-b4zmq\" (UID: \"8f7ff673-520b-4dd1-a76d-0e7aa1053c3f\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-b4zmq"
Nov 03 20:16:55.776766 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:55.776729    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f7ff673-520b-4dd1-a76d-0e7aa1053c3f-credential-socket\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-b4zmq\" (UID: \"8f7ff673-520b-4dd1-a76d-0e7aa1053c3f\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-b4zmq"
Nov 03 20:16:55.776969 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:55.776813    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f7ff673-520b-4dd1-a76d-0e7aa1053c3f-istio-podinfo\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-b4zmq\" (UID: \"8f7ff673-520b-4dd1-a76d-0e7aa1053c3f\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-b4zmq"
Nov 03 20:16:55.776969 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:55.776931    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f7ff673-520b-4dd1-a76d-0e7aa1053c3f-istiod-ca-cert\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-b4zmq\" (UID: \"8f7ff673-520b-4dd1-a76d-0e7aa1053c3f\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-b4zmq"
Nov 03 20:16:55.776969 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:55.776959    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f7ff673-520b-4dd1-a76d-0e7aa1053c3f-istio-envoy\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-b4zmq\" (UID: \"8f7ff673-520b-4dd1-a76d-0e7aa1053c3f\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-b4zmq"
Nov 03 20:16:55.777113 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:55.777012    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f7ff673-520b-4dd1-a76d-0e7aa1053c3f-istio-token\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-b4zmq\" (UID: \"8f7ff673-520b-4dd1-a76d-0e7aa1053c3f\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-b4zmq"
Nov 03 20:16:55.777169 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:55.777125    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5kvd\" (UniqueName: \"kubernetes.io/projected/8f7ff673-520b-4dd1-a76d-0e7aa1053c3f-kube-api-access-x5kvd\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-b4zmq\" (UID: \"8f7ff673-520b-4dd1-a76d-0e7aa1053c3f\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-b4zmq"
Nov 03 20:16:55.777222 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:55.777173    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f7ff673-520b-4dd1-a76d-0e7aa1053c3f-workload-socket\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-b4zmq\" (UID: \"8f7ff673-520b-4dd1-a76d-0e7aa1053c3f\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-b4zmq"
Nov 03 20:16:55.777278 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:55.777218    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f7ff673-520b-4dd1-a76d-0e7aa1053c3f-istio-data\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-b4zmq\" (UID: \"8f7ff673-520b-4dd1-a76d-0e7aa1053c3f\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-b4zmq"
Nov 03 20:16:55.777278 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:55.777235    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f7ff673-520b-4dd1-a76d-0e7aa1053c3f-credential-socket\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-b4zmq\" (UID: \"8f7ff673-520b-4dd1-a76d-0e7aa1053c3f\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-b4zmq"
Nov 03 20:16:55.777278 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:55.777248    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f7ff673-520b-4dd1-a76d-0e7aa1053c3f-workload-certs\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-b4zmq\" (UID: \"8f7ff673-520b-4dd1-a76d-0e7aa1053c3f\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-b4zmq"
Nov 03 20:16:55.777543 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:55.777508    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f7ff673-520b-4dd1-a76d-0e7aa1053c3f-workload-socket\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-b4zmq\" (UID: \"8f7ff673-520b-4dd1-a76d-0e7aa1053c3f\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-b4zmq"
Nov 03 20:16:55.777651 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:55.777549    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f7ff673-520b-4dd1-a76d-0e7aa1053c3f-workload-certs\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-b4zmq\" (UID: \"8f7ff673-520b-4dd1-a76d-0e7aa1053c3f\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-b4zmq"
Nov 03 20:16:55.777721 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:55.777690    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f7ff673-520b-4dd1-a76d-0e7aa1053c3f-istio-data\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-b4zmq\" (UID: \"8f7ff673-520b-4dd1-a76d-0e7aa1053c3f\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-b4zmq"
Nov 03 20:16:55.777798 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:55.777727    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f7ff673-520b-4dd1-a76d-0e7aa1053c3f-istiod-ca-cert\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-b4zmq\" (UID: \"8f7ff673-520b-4dd1-a76d-0e7aa1053c3f\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-b4zmq"
Nov 03 20:16:55.779356 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:55.779325    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f7ff673-520b-4dd1-a76d-0e7aa1053c3f-istio-envoy\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-b4zmq\" (UID: \"8f7ff673-520b-4dd1-a76d-0e7aa1053c3f\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-b4zmq"
Nov 03 20:16:55.779469 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:55.779453    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f7ff673-520b-4dd1-a76d-0e7aa1053c3f-istio-podinfo\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-b4zmq\" (UID: \"8f7ff673-520b-4dd1-a76d-0e7aa1053c3f\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-b4zmq"
Nov 03 20:16:55.799498 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:55.799465    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5kvd\" (UniqueName: \"kubernetes.io/projected/8f7ff673-520b-4dd1-a76d-0e7aa1053c3f-kube-api-access-x5kvd\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-b4zmq\" (UID: \"8f7ff673-520b-4dd1-a76d-0e7aa1053c3f\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-b4zmq"
Nov 03 20:16:55.799631 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:55.799531    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f7ff673-520b-4dd1-a76d-0e7aa1053c3f-istio-token\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-b4zmq\" (UID: \"8f7ff673-520b-4dd1-a76d-0e7aa1053c3f\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-b4zmq"
Nov 03 20:16:55.931164 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:55.931053    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-b4zmq"
Nov 03 20:16:56.259092 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:56.259048    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-b4zmq"]
Nov 03 20:16:56.260594 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:16:56.260563    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f7ff673_520b_4dd1_a76d_0e7aa1053c3f.slice/crio-743236b596fb07582b4fc24ccf3b8e10f871c0fb0f63f7ce1bb5a5b47e890382 WatchSource:0}: Error finding container 743236b596fb07582b4fc24ccf3b8e10f871c0fb0f63f7ce1bb5a5b47e890382: Status 404 returned error can't find the container with id 743236b596fb07582b4fc24ccf3b8e10f871c0fb0f63f7ce1bb5a5b47e890382
Nov 03 20:16:56.263092 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:56.263053    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:16:56.263172 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:56.263130    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:16:56.263172 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:56.263161    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:16:56.883150 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:56.883113    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-b4zmq" event={"ID":"8f7ff673-520b-4dd1-a76d-0e7aa1053c3f","Type":"ContainerStarted","Data":"c168feccf6d7ffe37ff51f11a79108fcc28d009b7994b2ad80a8a06cb49bd947"}
Nov 03 20:16:56.883574 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:56.883173    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-b4zmq" event={"ID":"8f7ff673-520b-4dd1-a76d-0e7aa1053c3f","Type":"ContainerStarted","Data":"743236b596fb07582b4fc24ccf3b8e10f871c0fb0f63f7ce1bb5a5b47e890382"}
Nov 03 20:16:56.932254 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:56.932213    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-b4zmq"
Nov 03 20:16:56.959991 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:56.959910    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-b4zmq" podStartSLOduration=1.9598919430000001 podStartE2EDuration="1.959891943s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:16:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:16:56.95831148 +0000 UTC m=+751.110973367" watchObservedRunningTime="2026-11-03 20:16:56.959891943 +0000 UTC m=+751.112553832"
Nov 03 20:16:57.228763 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:16:57.228725    5570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8bwhz" containerName="extractor"
Nov 03 20:16:57.933375 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:57.933340    5570 patch_prober.go:29] interesting pod/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-b4zmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Nov 03 20:16:57.933912 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:57.933401    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-b4zmq" podUID="8f7ff673-520b-4dd1-a76d-0e7aa1053c3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Nov 03 20:16:58.932048 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:58.932010    5570 patch_prober.go:29] interesting pod/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-b4zmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Nov 03 20:16:58.932245 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:58.932074    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-b4zmq" podUID="8f7ff673-520b-4dd1-a76d-0e7aa1053c3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Nov 03 20:16:59.932453 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:59.932413    5570 patch_prober.go:29] interesting pod/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-b4zmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Nov 03 20:16:59.932874 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:16:59.932472    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-b4zmq" podUID="8f7ff673-520b-4dd1-a76d-0e7aa1053c3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Nov 03 20:17:00.931953 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:00.931913    5570 patch_prober.go:29] interesting pod/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-b4zmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Nov 03 20:17:00.932122 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:00.931971    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-b4zmq" podUID="8f7ff673-520b-4dd1-a76d-0e7aa1053c3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Nov 03 20:17:01.551322 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:01.551264    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-6fc748/premium-w1-6fc748-kserve-7474db9cd7-nf2xk"]
Nov 03 20:17:01.573811 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:01.573739    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-6fc748/premium-w1-6fc748-kserve-7474db9cd7-nf2xk"]
Nov 03 20:17:01.573999 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:01.573877    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-6fc748/premium-w1-6fc748-kserve-7474db9cd7-nf2xk"
Nov 03 20:17:01.590349 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:01.590322    5570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-6fc748\"/\"openshift-service-ca.crt\""
Nov 03 20:17:01.604019 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:01.603984    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-6fc748\"/\"default-dockercfg-2v8qx\""
Nov 03 20:17:01.604201 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:01.604072    5570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-6fc748\"/\"kube-root-ca.crt\""
Nov 03 20:17:01.604365 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:01.604340    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-6fc748\"/\"premium-w1-6fc748-kserve-self-signed-certs\""
Nov 03 20:17:01.670560 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:01.670524    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sc2wj\" (UniqueName: \"kubernetes.io/projected/830e062e-83fc-4de7-91a7-5534dac1d46b-kube-api-access-sc2wj\") pod \"premium-w1-6fc748-kserve-7474db9cd7-nf2xk\" (UID: \"830e062e-83fc-4de7-91a7-5534dac1d46b\") " pod="e2e-models-e2e-worker-w1-6fc748/premium-w1-6fc748-kserve-7474db9cd7-nf2xk"
Nov 03 20:17:01.670755 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:01.670600    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/830e062e-83fc-4de7-91a7-5534dac1d46b-tls-certs\") pod \"premium-w1-6fc748-kserve-7474db9cd7-nf2xk\" (UID: \"830e062e-83fc-4de7-91a7-5534dac1d46b\") " pod="e2e-models-e2e-worker-w1-6fc748/premium-w1-6fc748-kserve-7474db9cd7-nf2xk"
Nov 03 20:17:01.670755 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:01.670733    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/830e062e-83fc-4de7-91a7-5534dac1d46b-dshm\") pod \"premium-w1-6fc748-kserve-7474db9cd7-nf2xk\" (UID: \"830e062e-83fc-4de7-91a7-5534dac1d46b\") " pod="e2e-models-e2e-worker-w1-6fc748/premium-w1-6fc748-kserve-7474db9cd7-nf2xk"
Nov 03 20:17:01.670874 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:01.670843    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/830e062e-83fc-4de7-91a7-5534dac1d46b-tmp-dir\") pod \"premium-w1-6fc748-kserve-7474db9cd7-nf2xk\" (UID: \"830e062e-83fc-4de7-91a7-5534dac1d46b\") " pod="e2e-models-e2e-worker-w1-6fc748/premium-w1-6fc748-kserve-7474db9cd7-nf2xk"
Nov 03 20:17:01.670874 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:01.670865    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/830e062e-83fc-4de7-91a7-5534dac1d46b-home\") pod \"premium-w1-6fc748-kserve-7474db9cd7-nf2xk\" (UID: \"830e062e-83fc-4de7-91a7-5534dac1d46b\") " pod="e2e-models-e2e-worker-w1-6fc748/premium-w1-6fc748-kserve-7474db9cd7-nf2xk"
Nov 03 20:17:01.671028 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:01.671014    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/830e062e-83fc-4de7-91a7-5534dac1d46b-model-cache\") pod \"premium-w1-6fc748-kserve-7474db9cd7-nf2xk\" (UID: \"830e062e-83fc-4de7-91a7-5534dac1d46b\") " pod="e2e-models-e2e-worker-w1-6fc748/premium-w1-6fc748-kserve-7474db9cd7-nf2xk"
Nov 03 20:17:01.778552 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:01.778503    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/830e062e-83fc-4de7-91a7-5534dac1d46b-tls-certs\") pod \"premium-w1-6fc748-kserve-7474db9cd7-nf2xk\" (UID: \"830e062e-83fc-4de7-91a7-5534dac1d46b\") " pod="e2e-models-e2e-worker-w1-6fc748/premium-w1-6fc748-kserve-7474db9cd7-nf2xk"
Nov 03 20:17:01.779417 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:01.779392    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/830e062e-83fc-4de7-91a7-5534dac1d46b-dshm\") pod \"premium-w1-6fc748-kserve-7474db9cd7-nf2xk\" (UID: \"830e062e-83fc-4de7-91a7-5534dac1d46b\") " pod="e2e-models-e2e-worker-w1-6fc748/premium-w1-6fc748-kserve-7474db9cd7-nf2xk"
Nov 03 20:17:01.779618 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:01.779602    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/830e062e-83fc-4de7-91a7-5534dac1d46b-tmp-dir\") pod \"premium-w1-6fc748-kserve-7474db9cd7-nf2xk\" (UID: \"830e062e-83fc-4de7-91a7-5534dac1d46b\") " pod="e2e-models-e2e-worker-w1-6fc748/premium-w1-6fc748-kserve-7474db9cd7-nf2xk"
Nov 03 20:17:01.779722 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:01.779709    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/830e062e-83fc-4de7-91a7-5534dac1d46b-home\") pod \"premium-w1-6fc748-kserve-7474db9cd7-nf2xk\" (UID: \"830e062e-83fc-4de7-91a7-5534dac1d46b\") " pod="e2e-models-e2e-worker-w1-6fc748/premium-w1-6fc748-kserve-7474db9cd7-nf2xk"
Nov 03 20:17:01.779994 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:01.779974    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/830e062e-83fc-4de7-91a7-5534dac1d46b-model-cache\") pod \"premium-w1-6fc748-kserve-7474db9cd7-nf2xk\" (UID: \"830e062e-83fc-4de7-91a7-5534dac1d46b\") " pod="e2e-models-e2e-worker-w1-6fc748/premium-w1-6fc748-kserve-7474db9cd7-nf2xk"
Nov 03 20:17:01.787814 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:01.780143    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sc2wj\" (UniqueName: \"kubernetes.io/projected/830e062e-83fc-4de7-91a7-5534dac1d46b-kube-api-access-sc2wj\") pod \"premium-w1-6fc748-kserve-7474db9cd7-nf2xk\" (UID: \"830e062e-83fc-4de7-91a7-5534dac1d46b\") " pod="e2e-models-e2e-worker-w1-6fc748/premium-w1-6fc748-kserve-7474db9cd7-nf2xk"
Nov 03 20:17:01.788200 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:01.780507    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/830e062e-83fc-4de7-91a7-5534dac1d46b-tmp-dir\") pod \"premium-w1-6fc748-kserve-7474db9cd7-nf2xk\" (UID: \"830e062e-83fc-4de7-91a7-5534dac1d46b\") " pod="e2e-models-e2e-worker-w1-6fc748/premium-w1-6fc748-kserve-7474db9cd7-nf2xk"
Nov 03 20:17:01.788350 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:01.780732    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/830e062e-83fc-4de7-91a7-5534dac1d46b-model-cache\") pod \"premium-w1-6fc748-kserve-7474db9cd7-nf2xk\" (UID: \"830e062e-83fc-4de7-91a7-5534dac1d46b\") " pod="e2e-models-e2e-worker-w1-6fc748/premium-w1-6fc748-kserve-7474db9cd7-nf2xk"
Nov 03 20:17:01.788447 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:01.782379    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/830e062e-83fc-4de7-91a7-5534dac1d46b-home\") pod \"premium-w1-6fc748-kserve-7474db9cd7-nf2xk\" (UID: \"830e062e-83fc-4de7-91a7-5534dac1d46b\") " pod="e2e-models-e2e-worker-w1-6fc748/premium-w1-6fc748-kserve-7474db9cd7-nf2xk"
Nov 03 20:17:01.807551 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:01.807469    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/830e062e-83fc-4de7-91a7-5534dac1d46b-tls-certs\") pod \"premium-w1-6fc748-kserve-7474db9cd7-nf2xk\" (UID: \"830e062e-83fc-4de7-91a7-5534dac1d46b\") " pod="e2e-models-e2e-worker-w1-6fc748/premium-w1-6fc748-kserve-7474db9cd7-nf2xk"
Nov 03 20:17:01.812382 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:01.812351    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/830e062e-83fc-4de7-91a7-5534dac1d46b-dshm\") pod \"premium-w1-6fc748-kserve-7474db9cd7-nf2xk\" (UID: \"830e062e-83fc-4de7-91a7-5534dac1d46b\") " pod="e2e-models-e2e-worker-w1-6fc748/premium-w1-6fc748-kserve-7474db9cd7-nf2xk"
Nov 03 20:17:01.821893 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:01.821856    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sc2wj\" (UniqueName: \"kubernetes.io/projected/830e062e-83fc-4de7-91a7-5534dac1d46b-kube-api-access-sc2wj\") pod \"premium-w1-6fc748-kserve-7474db9cd7-nf2xk\" (UID: \"830e062e-83fc-4de7-91a7-5534dac1d46b\") " pod="e2e-models-e2e-worker-w1-6fc748/premium-w1-6fc748-kserve-7474db9cd7-nf2xk"
Nov 03 20:17:01.892094 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:01.892056    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-6fc748/premium-w1-6fc748-kserve-7474db9cd7-nf2xk"
Nov 03 20:17:01.932478 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:01.932441    5570 patch_prober.go:29] interesting pod/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-b4zmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Nov 03 20:17:01.932663 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:01.932502    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-b4zmq" podUID="8f7ff673-520b-4dd1-a76d-0e7aa1053c3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Nov 03 20:17:02.241731 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:02.241702    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-6fc748/premium-w1-6fc748-kserve-7474db9cd7-nf2xk"]
Nov 03 20:17:02.244597 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:17:02.244560    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod830e062e_83fc_4de7_91a7_5534dac1d46b.slice/crio-5fe19cda93a67d49b6b53e87a796287bcb8391ea55c870857d69d6a53ea458f6 WatchSource:0}: Error finding container 5fe19cda93a67d49b6b53e87a796287bcb8391ea55c870857d69d6a53ea458f6: Status 404 returned error can't find the container with id 5fe19cda93a67d49b6b53e87a796287bcb8391ea55c870857d69d6a53ea458f6
Nov 03 20:17:02.935214 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:02.935169    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-6fc748/premium-w1-6fc748-kserve-7474db9cd7-nf2xk" event={"ID":"830e062e-83fc-4de7-91a7-5534dac1d46b","Type":"ContainerStarted","Data":"a24442a935efa7166e38699d8b414a492546737e47c8e40770013fb03915efb0"}
Nov 03 20:17:02.935860 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:02.935826    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-6fc748/premium-w1-6fc748-kserve-7474db9cd7-nf2xk" event={"ID":"830e062e-83fc-4de7-91a7-5534dac1d46b","Type":"ContainerStarted","Data":"5fe19cda93a67d49b6b53e87a796287bcb8391ea55c870857d69d6a53ea458f6"}
Nov 03 20:17:03.060814 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:03.059742    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-b4zmq"
Nov 03 20:17:03.061761 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:03.061587    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-b4zmq"
Nov 03 20:17:03.063280 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:03.063222    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-b4zmq"
Nov 03 20:17:03.121169 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:03.121100    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-6fc748/premium-w1-6fc748-kserve-7474db9cd7-nf2xk" podStartSLOduration=2.121075314 podStartE2EDuration="2.121075314s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:17:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:17:02.989056258 +0000 UTC m=+757.141718149" watchObservedRunningTime="2026-11-03 20:17:03.121075314 +0000 UTC m=+757.273737203"
Nov 03 20:17:03.893472 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:03.893434    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6fc748/premium-w1-6fc748-kserve-7474db9cd7-nf2xk"
Nov 03 20:17:11.642802 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:11.642708    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g69f8"]
Nov 03 20:17:11.893179 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:11.893089    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-6fc748/premium-w1-6fc748-kserve-7474db9cd7-nf2xk"
Nov 03 20:17:11.909760 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:11.909730    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-6fc748/premium-w1-6fc748-kserve-7474db9cd7-nf2xk"
Nov 03 20:17:11.997667 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:11.997636    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6fc748/premium-w1-6fc748-kserve-7474db9cd7-nf2xk"
Nov 03 20:17:16.364296 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:16.364253    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-9e1bcb/premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh"]
Nov 03 20:17:16.369237 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:16.369213    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-9e1bcb/premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh"
Nov 03 20:17:16.385674 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:16.385643    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-9e1bcb/premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh"]
Nov 03 20:17:16.396697 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:16.396662    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-9e1bcb\"/\"default-dockercfg-9ptvd\""
Nov 03 20:17:16.396903 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:16.396733    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-9e1bcb\"/\"premium-w6-9e1bcb-kserve-self-signed-certs\""
Nov 03 20:17:16.400324 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:16.400297    5570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-9e1bcb\"/\"openshift-service-ca.crt\""
Nov 03 20:17:16.400402 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:16.400360    5570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-9e1bcb\"/\"kube-root-ca.crt\""
Nov 03 20:17:16.483491 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:16.483450    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8r4mh\" (UniqueName: \"kubernetes.io/projected/2d3647f2-89e9-4908-a8ee-d84e527abeb0-kube-api-access-8r4mh\") pod \"premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh\" (UID: \"2d3647f2-89e9-4908-a8ee-d84e527abeb0\") " pod="e2e-models-e2e-worker-w6-9e1bcb/premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh"
Nov 03 20:17:16.483698 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:16.483515    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d3647f2-89e9-4908-a8ee-d84e527abeb0-home\") pod \"premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh\" (UID: \"2d3647f2-89e9-4908-a8ee-d84e527abeb0\") " pod="e2e-models-e2e-worker-w6-9e1bcb/premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh"
Nov 03 20:17:16.483698 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:16.483619    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d3647f2-89e9-4908-a8ee-d84e527abeb0-model-cache\") pod \"premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh\" (UID: \"2d3647f2-89e9-4908-a8ee-d84e527abeb0\") " pod="e2e-models-e2e-worker-w6-9e1bcb/premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh"
Nov 03 20:17:16.483698 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:16.483651    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d3647f2-89e9-4908-a8ee-d84e527abeb0-tls-certs\") pod \"premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh\" (UID: \"2d3647f2-89e9-4908-a8ee-d84e527abeb0\") " pod="e2e-models-e2e-worker-w6-9e1bcb/premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh"
Nov 03 20:17:16.483860 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:16.483838    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d3647f2-89e9-4908-a8ee-d84e527abeb0-dshm\") pod \"premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh\" (UID: \"2d3647f2-89e9-4908-a8ee-d84e527abeb0\") " pod="e2e-models-e2e-worker-w6-9e1bcb/premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh"
Nov 03 20:17:16.484011 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:16.483991    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d3647f2-89e9-4908-a8ee-d84e527abeb0-tmp-dir\") pod \"premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh\" (UID: \"2d3647f2-89e9-4908-a8ee-d84e527abeb0\") " pod="e2e-models-e2e-worker-w6-9e1bcb/premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh"
Nov 03 20:17:16.597956 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:16.587564    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d3647f2-89e9-4908-a8ee-d84e527abeb0-dshm\") pod \"premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh\" (UID: \"2d3647f2-89e9-4908-a8ee-d84e527abeb0\") " pod="e2e-models-e2e-worker-w6-9e1bcb/premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh"
Nov 03 20:17:16.599372 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:16.599341    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d3647f2-89e9-4908-a8ee-d84e527abeb0-tmp-dir\") pod \"premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh\" (UID: \"2d3647f2-89e9-4908-a8ee-d84e527abeb0\") " pod="e2e-models-e2e-worker-w6-9e1bcb/premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh"
Nov 03 20:17:16.604862 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:16.604828    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8r4mh\" (UniqueName: \"kubernetes.io/projected/2d3647f2-89e9-4908-a8ee-d84e527abeb0-kube-api-access-8r4mh\") pod \"premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh\" (UID: \"2d3647f2-89e9-4908-a8ee-d84e527abeb0\") " pod="e2e-models-e2e-worker-w6-9e1bcb/premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh"
Nov 03 20:17:16.612610 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:16.612577    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d3647f2-89e9-4908-a8ee-d84e527abeb0-home\") pod \"premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh\" (UID: \"2d3647f2-89e9-4908-a8ee-d84e527abeb0\") " pod="e2e-models-e2e-worker-w6-9e1bcb/premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh"
Nov 03 20:17:16.613262 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:16.613241    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d3647f2-89e9-4908-a8ee-d84e527abeb0-model-cache\") pod \"premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh\" (UID: \"2d3647f2-89e9-4908-a8ee-d84e527abeb0\") " pod="e2e-models-e2e-worker-w6-9e1bcb/premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh"
Nov 03 20:17:16.613710 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:16.613693    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d3647f2-89e9-4908-a8ee-d84e527abeb0-tls-certs\") pod \"premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh\" (UID: \"2d3647f2-89e9-4908-a8ee-d84e527abeb0\") " pod="e2e-models-e2e-worker-w6-9e1bcb/premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh"
Nov 03 20:17:16.614014 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:16.613159    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d3647f2-89e9-4908-a8ee-d84e527abeb0-home\") pod \"premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh\" (UID: \"2d3647f2-89e9-4908-a8ee-d84e527abeb0\") " pod="e2e-models-e2e-worker-w6-9e1bcb/premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh"
Nov 03 20:17:16.614127 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:16.599940    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d3647f2-89e9-4908-a8ee-d84e527abeb0-tmp-dir\") pod \"premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh\" (UID: \"2d3647f2-89e9-4908-a8ee-d84e527abeb0\") " pod="e2e-models-e2e-worker-w6-9e1bcb/premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh"
Nov 03 20:17:16.614217 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:16.613648    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d3647f2-89e9-4908-a8ee-d84e527abeb0-model-cache\") pod \"premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh\" (UID: \"2d3647f2-89e9-4908-a8ee-d84e527abeb0\") " pod="e2e-models-e2e-worker-w6-9e1bcb/premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh"
Nov 03 20:17:16.614348 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:16.610177    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d3647f2-89e9-4908-a8ee-d84e527abeb0-dshm\") pod \"premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh\" (UID: \"2d3647f2-89e9-4908-a8ee-d84e527abeb0\") " pod="e2e-models-e2e-worker-w6-9e1bcb/premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh"
Nov 03 20:17:16.625043 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:16.625010    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d3647f2-89e9-4908-a8ee-d84e527abeb0-tls-certs\") pod \"premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh\" (UID: \"2d3647f2-89e9-4908-a8ee-d84e527abeb0\") " pod="e2e-models-e2e-worker-w6-9e1bcb/premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh"
Nov 03 20:17:16.639796 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:16.635970    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8r4mh\" (UniqueName: \"kubernetes.io/projected/2d3647f2-89e9-4908-a8ee-d84e527abeb0-kube-api-access-8r4mh\") pod \"premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh\" (UID: \"2d3647f2-89e9-4908-a8ee-d84e527abeb0\") " pod="e2e-models-e2e-worker-w6-9e1bcb/premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh"
Nov 03 20:17:16.686818 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:16.686743    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-9e1bcb/premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh"
Nov 03 20:17:17.000055 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:17.000024    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-9e1bcb/premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh"]
Nov 03 20:17:17.002331 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:17:17.002302    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2d3647f2_89e9_4908_a8ee_d84e527abeb0.slice/crio-e89a95af30ee334c8c6ef4ce4007d0ca1a4b4a9d9aa23c8971cc51b2edaa742b WatchSource:0}: Error finding container e89a95af30ee334c8c6ef4ce4007d0ca1a4b4a9d9aa23c8971cc51b2edaa742b: Status 404 returned error can't find the container with id e89a95af30ee334c8c6ef4ce4007d0ca1a4b4a9d9aa23c8971cc51b2edaa742b
Nov 03 20:17:17.055588 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:17.055551    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-9e1bcb/premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh" event={"ID":"2d3647f2-89e9-4908-a8ee-d84e527abeb0","Type":"ContainerStarted","Data":"e89a95af30ee334c8c6ef4ce4007d0ca1a4b4a9d9aa23c8971cc51b2edaa742b"}
Nov 03 20:17:18.063726 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:18.063676    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-9e1bcb/premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh" event={"ID":"2d3647f2-89e9-4908-a8ee-d84e527abeb0","Type":"ContainerStarted","Data":"e5774781017bffdb309abab3f3038fc990f221a8bb660c664b236d0529e6dce2"}
Nov 03 20:17:18.108134 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:18.108078    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-9e1bcb/premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh" podStartSLOduration=2.108059767 podStartE2EDuration="2.108059767s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:17:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:17:18.106098211 +0000 UTC m=+772.258760098" watchObservedRunningTime="2026-11-03 20:17:18.108059767 +0000 UTC m=+772.260721647"
Nov 03 20:17:18.688242 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:18.688203    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-9e1bcb/premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh"
Nov 03 20:17:24.639200 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:24.639132    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/simulator-w0-e40816-kserve-688f79796f-2tvw5" podUID="c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:17:24.693804 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:24.692963    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/premium-w4-e80203-kserve-5f9f445456-bms9d" podUID="6ad76efc-20c2-4951-b22a-c5a39cd39ed4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:17:24.804931 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:24.804875    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/premium-w3-482722-kserve-689dc7bd5d-x8rjp" podUID="369045c3-4d6a-49a7-9bc6-f2939883bee8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:17:24.834949 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:24.834888    5570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-9973d9/premium-w2-9973d9-kserve-5c6dd4669c-5gv8x" podUID="cefc89ad-7ae1-433b-979d-6601678a6d98" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:17:24.871948 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:24.871892    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/premium-w2-9973d9-kserve-5c6dd4669c-5gv8x" podUID="cefc89ad-7ae1-433b-979d-6601678a6d98" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:17:24.988941 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:24.988886    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/premium-w1-6fc748-kserve-7474db9cd7-nf2xk" podUID="830e062e-83fc-4de7-91a7-5534dac1d46b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:17:25.558797 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:25.511658    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz" podUID="3a715ef1-38e9-4386-9eec-e10e8df11a10" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:17:25.596801 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:25.535178    5570 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz" podUID="3a715ef1-38e9-4386-9eec-e10e8df11a10" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:17:25.740806 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:25.599331    5570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-482722/premium-w3-482722-kserve-689dc7bd5d-x8rjp" podUID="369045c3-4d6a-49a7-9bc6-f2939883bee8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": context deadline exceeded"
Nov 03 20:17:25.854856 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:25.741854    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/premium-w4-e80203-kserve-5f9f445456-bms9d" podUID="6ad76efc-20c2-4951-b22a-c5a39cd39ed4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:17:25.854856 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:25.851404    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-e80203/premium-w4-e80203-kserve-5f9f445456-bms9d"
Nov 03 20:17:25.909327 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:25.908859    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/premium-w2-9973d9-kserve-5c6dd4669c-5gv8x" podUID="cefc89ad-7ae1-433b-979d-6601678a6d98" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Nov 03 20:17:25.909509 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:25.909354    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9973d9/premium-w2-9973d9-kserve-5c6dd4669c-5gv8x"
Nov 03 20:17:26.018883 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:26.018822    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/premium-w1-6fc748-kserve-7474db9cd7-nf2xk" podUID="830e062e-83fc-4de7-91a7-5534dac1d46b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Nov 03 20:17:26.285796 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:26.284840    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6fc748/premium-w1-6fc748-kserve-7474db9cd7-nf2xk"
Nov 03 20:17:26.285796 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:26.285566    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-e80203/premium-w4-e80203-kserve-5f9f445456-bms9d"
Nov 03 20:17:26.289808 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:26.286218    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9973d9/premium-w2-9973d9-kserve-5c6dd4669c-5gv8x"
Nov 03 20:17:26.289808 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:26.286253    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-c9kvc"]
Nov 03 20:17:26.330207 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:26.330177    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-c9kvc"]
Nov 03 20:17:26.330688 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:26.330663    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-c9kvc"
Nov 03 20:17:26.357849 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:26.346331    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6fc748/premium-w1-6fc748-kserve-7474db9cd7-nf2xk"
Nov 03 20:17:26.357849 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:26.349280    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-482722-openshift-default-dockercfg-6d2xm\""
Nov 03 20:17:26.437789 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:26.437730    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qw8fv\" (UniqueName: \"kubernetes.io/projected/1351662d-6ca4-430b-8bbe-0ca1429df833-kube-api-access-qw8fv\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-c9kvc\" (UID: \"1351662d-6ca4-430b-8bbe-0ca1429df833\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-c9kvc"
Nov 03 20:17:26.438071 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:26.438046    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1351662d-6ca4-430b-8bbe-0ca1429df833-istiod-ca-cert\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-c9kvc\" (UID: \"1351662d-6ca4-430b-8bbe-0ca1429df833\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-c9kvc"
Nov 03 20:17:26.438348 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:26.438328    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1351662d-6ca4-430b-8bbe-0ca1429df833-istio-data\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-c9kvc\" (UID: \"1351662d-6ca4-430b-8bbe-0ca1429df833\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-c9kvc"
Nov 03 20:17:26.438538 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:26.438522    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1351662d-6ca4-430b-8bbe-0ca1429df833-istio-envoy\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-c9kvc\" (UID: \"1351662d-6ca4-430b-8bbe-0ca1429df833\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-c9kvc"
Nov 03 20:17:26.438754 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:26.438731    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1351662d-6ca4-430b-8bbe-0ca1429df833-workload-certs\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-c9kvc\" (UID: \"1351662d-6ca4-430b-8bbe-0ca1429df833\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-c9kvc"
Nov 03 20:17:26.438918 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:26.438902    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1351662d-6ca4-430b-8bbe-0ca1429df833-workload-socket\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-c9kvc\" (UID: \"1351662d-6ca4-430b-8bbe-0ca1429df833\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-c9kvc"
Nov 03 20:17:26.439081 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:26.439063    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1351662d-6ca4-430b-8bbe-0ca1429df833-credential-socket\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-c9kvc\" (UID: \"1351662d-6ca4-430b-8bbe-0ca1429df833\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-c9kvc"
Nov 03 20:17:26.439300 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:26.439282    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1351662d-6ca4-430b-8bbe-0ca1429df833-istio-token\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-c9kvc\" (UID: \"1351662d-6ca4-430b-8bbe-0ca1429df833\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-c9kvc"
Nov 03 20:17:26.439461 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:26.439445    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1351662d-6ca4-430b-8bbe-0ca1429df833-istio-podinfo\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-c9kvc\" (UID: \"1351662d-6ca4-430b-8bbe-0ca1429df833\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-c9kvc"
Nov 03 20:17:26.541651 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:26.541552    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1351662d-6ca4-430b-8bbe-0ca1429df833-workload-certs\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-c9kvc\" (UID: \"1351662d-6ca4-430b-8bbe-0ca1429df833\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-c9kvc"
Nov 03 20:17:26.541926 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:26.541904    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1351662d-6ca4-430b-8bbe-0ca1429df833-workload-socket\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-c9kvc\" (UID: \"1351662d-6ca4-430b-8bbe-0ca1429df833\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-c9kvc"
Nov 03 20:17:26.542068 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:26.542054    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1351662d-6ca4-430b-8bbe-0ca1429df833-credential-socket\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-c9kvc\" (UID: \"1351662d-6ca4-430b-8bbe-0ca1429df833\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-c9kvc"
Nov 03 20:17:26.542157 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:26.542129    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1351662d-6ca4-430b-8bbe-0ca1429df833-workload-certs\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-c9kvc\" (UID: \"1351662d-6ca4-430b-8bbe-0ca1429df833\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-c9kvc"
Nov 03 20:17:26.542326 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:26.542303    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1351662d-6ca4-430b-8bbe-0ca1429df833-istio-token\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-c9kvc\" (UID: \"1351662d-6ca4-430b-8bbe-0ca1429df833\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-c9kvc"
Nov 03 20:17:26.542467 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:26.542449    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1351662d-6ca4-430b-8bbe-0ca1429df833-istio-podinfo\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-c9kvc\" (UID: \"1351662d-6ca4-430b-8bbe-0ca1429df833\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-c9kvc"
Nov 03 20:17:26.542718 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:26.542692    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qw8fv\" (UniqueName: \"kubernetes.io/projected/1351662d-6ca4-430b-8bbe-0ca1429df833-kube-api-access-qw8fv\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-c9kvc\" (UID: \"1351662d-6ca4-430b-8bbe-0ca1429df833\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-c9kvc"
Nov 03 20:17:26.542928 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:26.542837    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1351662d-6ca4-430b-8bbe-0ca1429df833-istiod-ca-cert\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-c9kvc\" (UID: \"1351662d-6ca4-430b-8bbe-0ca1429df833\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-c9kvc"
Nov 03 20:17:26.543131 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:26.542934    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1351662d-6ca4-430b-8bbe-0ca1429df833-istio-data\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-c9kvc\" (UID: \"1351662d-6ca4-430b-8bbe-0ca1429df833\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-c9kvc"
Nov 03 20:17:26.543131 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:26.542975    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1351662d-6ca4-430b-8bbe-0ca1429df833-istio-envoy\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-c9kvc\" (UID: \"1351662d-6ca4-430b-8bbe-0ca1429df833\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-c9kvc"
Nov 03 20:17:26.543131 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:26.542996    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1351662d-6ca4-430b-8bbe-0ca1429df833-workload-socket\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-c9kvc\" (UID: \"1351662d-6ca4-430b-8bbe-0ca1429df833\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-c9kvc"
Nov 03 20:17:26.543340 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:26.542525    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1351662d-6ca4-430b-8bbe-0ca1429df833-credential-socket\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-c9kvc\" (UID: \"1351662d-6ca4-430b-8bbe-0ca1429df833\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-c9kvc"
Nov 03 20:17:26.543646 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:26.543624    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1351662d-6ca4-430b-8bbe-0ca1429df833-istio-data\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-c9kvc\" (UID: \"1351662d-6ca4-430b-8bbe-0ca1429df833\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-c9kvc"
Nov 03 20:17:26.544227 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:26.544200    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1351662d-6ca4-430b-8bbe-0ca1429df833-istiod-ca-cert\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-c9kvc\" (UID: \"1351662d-6ca4-430b-8bbe-0ca1429df833\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-c9kvc"
Nov 03 20:17:26.546572 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:26.546546    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1351662d-6ca4-430b-8bbe-0ca1429df833-istio-envoy\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-c9kvc\" (UID: \"1351662d-6ca4-430b-8bbe-0ca1429df833\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-c9kvc"
Nov 03 20:17:26.568562 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:26.568522    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1351662d-6ca4-430b-8bbe-0ca1429df833-istio-podinfo\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-c9kvc\" (UID: \"1351662d-6ca4-430b-8bbe-0ca1429df833\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-c9kvc"
Nov 03 20:17:26.572553 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:26.572519    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qw8fv\" (UniqueName: \"kubernetes.io/projected/1351662d-6ca4-430b-8bbe-0ca1429df833-kube-api-access-qw8fv\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-c9kvc\" (UID: \"1351662d-6ca4-430b-8bbe-0ca1429df833\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-c9kvc"
Nov 03 20:17:26.580572 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:26.580540    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1351662d-6ca4-430b-8bbe-0ca1429df833-istio-token\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-c9kvc\" (UID: \"1351662d-6ca4-430b-8bbe-0ca1429df833\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-c9kvc"
Nov 03 20:17:26.704054 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:26.703087    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-9e1bcb/premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh"
Nov 03 20:17:26.726138 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:26.726089    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-c9kvc"
Nov 03 20:17:26.735174 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:26.735144    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-9e1bcb/premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh"
Nov 03 20:17:27.241569 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:17:27.241526    5570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8bwhz" containerName="extractor"
Nov 03 20:17:27.245986 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:27.245952    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-c9kvc"]
Nov 03 20:17:27.251296 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:17:27.251232    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1351662d_6ca4_430b_8bbe_0ca1429df833.slice/crio-da42034e32112e0677143e671754162e71f1094f4321bf952e8b0c9de60c2720 WatchSource:0}: Error finding container da42034e32112e0677143e671754162e71f1094f4321bf952e8b0c9de60c2720: Status 404 returned error can't find the container with id da42034e32112e0677143e671754162e71f1094f4321bf952e8b0c9de60c2720
Nov 03 20:17:27.254280 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:27.254250    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:17:27.254391 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:27.254319    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:17:27.254391 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:27.254365    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:17:27.274137 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:27.274097    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-9e1bcb/premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh"
Nov 03 20:17:28.270821 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:28.270764    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-c9kvc" event={"ID":"1351662d-6ca4-430b-8bbe-0ca1429df833","Type":"ContainerStarted","Data":"8977e092d4a9cfa729f07741dd5894f87839bdf7aa38cf0b489c3d40058255c4"}
Nov 03 20:17:28.271254 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:28.270833    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-c9kvc" event={"ID":"1351662d-6ca4-430b-8bbe-0ca1429df833","Type":"ContainerStarted","Data":"da42034e32112e0677143e671754162e71f1094f4321bf952e8b0c9de60c2720"}
Nov 03 20:17:28.333222 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:28.333159    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-c9kvc" podStartSLOduration=2.33314138 podStartE2EDuration="2.33314138s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:17:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:17:28.331393128 +0000 UTC m=+782.484055043" watchObservedRunningTime="2026-11-03 20:17:28.33314138 +0000 UTC m=+782.485803264"
Nov 03 20:17:28.727512 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:28.727475    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-c9kvc"
Nov 03 20:17:28.729476 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:28.729442    5570 patch_prober.go:29] interesting pod/e2e-worker-w3-482722-openshift-default-7645487df-c9kvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Nov 03 20:17:28.729635 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:28.729509    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-c9kvc" podUID="1351662d-6ca4-430b-8bbe-0ca1429df833" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Nov 03 20:17:29.727303 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:29.727262    5570 patch_prober.go:29] interesting pod/e2e-worker-w3-482722-openshift-default-7645487df-c9kvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Nov 03 20:17:29.727830 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:29.727325    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-c9kvc" podUID="1351662d-6ca4-430b-8bbe-0ca1429df833" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Nov 03 20:17:30.727269 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:30.727225    5570 patch_prober.go:29] interesting pod/e2e-worker-w3-482722-openshift-default-7645487df-c9kvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Nov 03 20:17:30.727456 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:30.727292    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-c9kvc" podUID="1351662d-6ca4-430b-8bbe-0ca1429df833" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Nov 03 20:17:31.727893 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:31.727846    5570 patch_prober.go:29] interesting pod/e2e-worker-w3-482722-openshift-default-7645487df-c9kvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Nov 03 20:17:31.728430 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:31.727926    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-c9kvc" podUID="1351662d-6ca4-430b-8bbe-0ca1429df833" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Nov 03 20:17:32.727300 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:32.727260    5570 patch_prober.go:29] interesting pod/e2e-worker-w3-482722-openshift-default-7645487df-c9kvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Nov 03 20:17:32.727564 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:32.727339    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-c9kvc" podUID="1351662d-6ca4-430b-8bbe-0ca1429df833" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Nov 03 20:17:32.830560 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:32.830525    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-649609db-pqmqh"]
Nov 03 20:17:32.835536 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:32.835516    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-649609db-pqmqh"
Nov 03 20:17:32.866677 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:32.860071    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:17:32.866677 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:32.860273    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l7s5w\""
Nov 03 20:17:32.866677 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:32.861256    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-649609db-pqmqh"]
Nov 03 20:17:32.980107 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:32.980016    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a833c968-f463-4a3a-9451-57c0ee19a768-maas-api-xks-ca\") pod \"maas-api-revoke-sub-649609db-pqmqh\" (UID: \"a833c968-f463-4a3a-9451-57c0ee19a768\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-649609db-pqmqh"
Nov 03 20:17:32.980107 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:32.980071    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5r8c\" (UniqueName: \"kubernetes.io/projected/a833c968-f463-4a3a-9451-57c0ee19a768-kube-api-access-m5r8c\") pod \"maas-api-revoke-sub-649609db-pqmqh\" (UID: \"a833c968-f463-4a3a-9451-57c0ee19a768\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-649609db-pqmqh"
Nov 03 20:17:32.980356 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:32.980272    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a833c968-f463-4a3a-9451-57c0ee19a768-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-649609db-pqmqh\" (UID: \"a833c968-f463-4a3a-9451-57c0ee19a768\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-649609db-pqmqh"
Nov 03 20:17:33.081293 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:33.081261    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a833c968-f463-4a3a-9451-57c0ee19a768-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-649609db-pqmqh\" (UID: \"a833c968-f463-4a3a-9451-57c0ee19a768\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-649609db-pqmqh"
Nov 03 20:17:33.081514 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:33.081348    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a833c968-f463-4a3a-9451-57c0ee19a768-maas-api-xks-ca\") pod \"maas-api-revoke-sub-649609db-pqmqh\" (UID: \"a833c968-f463-4a3a-9451-57c0ee19a768\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-649609db-pqmqh"
Nov 03 20:17:33.081514 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:33.081389    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5r8c\" (UniqueName: \"kubernetes.io/projected/a833c968-f463-4a3a-9451-57c0ee19a768-kube-api-access-m5r8c\") pod \"maas-api-revoke-sub-649609db-pqmqh\" (UID: \"a833c968-f463-4a3a-9451-57c0ee19a768\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-649609db-pqmqh"
Nov 03 20:17:33.082107 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:33.082077    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a833c968-f463-4a3a-9451-57c0ee19a768-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-649609db-pqmqh\" (UID: \"a833c968-f463-4a3a-9451-57c0ee19a768\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-649609db-pqmqh"
Nov 03 20:17:33.083970 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:33.083948    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a833c968-f463-4a3a-9451-57c0ee19a768-maas-api-xks-ca\") pod \"maas-api-revoke-sub-649609db-pqmqh\" (UID: \"a833c968-f463-4a3a-9451-57c0ee19a768\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-649609db-pqmqh"
Nov 03 20:17:33.101484 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:33.101459    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5r8c\" (UniqueName: \"kubernetes.io/projected/a833c968-f463-4a3a-9451-57c0ee19a768-kube-api-access-m5r8c\") pod \"maas-api-revoke-sub-649609db-pqmqh\" (UID: \"a833c968-f463-4a3a-9451-57c0ee19a768\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-649609db-pqmqh"
Nov 03 20:17:33.178187 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:33.178154    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-649609db-pqmqh"
Nov 03 20:17:33.498525 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:33.498496    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-649609db-pqmqh"]
Nov 03 20:17:33.500510 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:17:33.500469    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda833c968_f463_4a3a_9451_57c0ee19a768.slice/crio-9fd01a16da6d8031f1c66770c6a11eef6f49e9eeca7fca400be185642517767e WatchSource:0}: Error finding container 9fd01a16da6d8031f1c66770c6a11eef6f49e9eeca7fca400be185642517767e: Status 404 returned error can't find the container with id 9fd01a16da6d8031f1c66770c6a11eef6f49e9eeca7fca400be185642517767e
Nov 03 20:17:33.727484 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:33.727449    5570 patch_prober.go:29] interesting pod/e2e-worker-w3-482722-openshift-default-7645487df-c9kvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Nov 03 20:17:33.727713 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:33.727501    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-c9kvc" podUID="1351662d-6ca4-430b-8bbe-0ca1429df833" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Nov 03 20:17:34.313188 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:34.313146    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-649609db-pqmqh" event={"ID":"a833c968-f463-4a3a-9451-57c0ee19a768","Type":"ContainerStarted","Data":"d0ffb9fc1f732a4ecad34a9b58046139943f0e616a1eccf1d8d21a3e9902aac5"}
Nov 03 20:17:34.313605 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:34.313199    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-649609db-pqmqh" event={"ID":"a833c968-f463-4a3a-9451-57c0ee19a768","Type":"ContainerStarted","Data":"9fd01a16da6d8031f1c66770c6a11eef6f49e9eeca7fca400be185642517767e"}
Nov 03 20:17:34.354931 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:34.354873    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-649609db-pqmqh" podStartSLOduration=2.354856993 podStartE2EDuration="2.354856993s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:17:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:17:34.35256876 +0000 UTC m=+788.505230672" watchObservedRunningTime="2026-11-03 20:17:34.354856993 +0000 UTC m=+788.507518881"
Nov 03 20:17:34.731555 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:34.731521    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-c9kvc"
Nov 03 20:17:34.731837 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:34.731814    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-c9kvc"
Nov 03 20:17:34.732700 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:34.732683    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-c9kvc"
Nov 03 20:17:35.320547 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:35.320506    5570 generic.go:356] "Generic (PLEG): container finished" podID="a833c968-f463-4a3a-9451-57c0ee19a768" containerID="d0ffb9fc1f732a4ecad34a9b58046139943f0e616a1eccf1d8d21a3e9902aac5" exitCode=0
Nov 03 20:17:35.321014 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:35.320588    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-649609db-pqmqh" event={"ID":"a833c968-f463-4a3a-9451-57c0ee19a768","Type":"ContainerDied","Data":"d0ffb9fc1f732a4ecad34a9b58046139943f0e616a1eccf1d8d21a3e9902aac5"}
Nov 03 20:17:36.487579 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:36.487554    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-649609db-pqmqh"
Nov 03 20:17:36.636714 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:36.636621    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a833c968-f463-4a3a-9451-57c0ee19a768-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a833c968-f463-4a3a-9451-57c0ee19a768-maas-api-openshift-ca\") pod \"a833c968-f463-4a3a-9451-57c0ee19a768\" (UID: \"a833c968-f463-4a3a-9451-57c0ee19a768\") "
Nov 03 20:17:36.636714 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:36.636679    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a833c968-f463-4a3a-9451-57c0ee19a768-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a833c968-f463-4a3a-9451-57c0ee19a768-maas-api-xks-ca\") pod \"a833c968-f463-4a3a-9451-57c0ee19a768\" (UID: \"a833c968-f463-4a3a-9451-57c0ee19a768\") "
Nov 03 20:17:36.637008 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:36.636882    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a833c968-f463-4a3a-9451-57c0ee19a768-kube-api-access-m5r8c\" (UniqueName: \"kubernetes.io/projected/a833c968-f463-4a3a-9451-57c0ee19a768-kube-api-access-m5r8c\") pod \"a833c968-f463-4a3a-9451-57c0ee19a768\" (UID: \"a833c968-f463-4a3a-9451-57c0ee19a768\") "
Nov 03 20:17:36.637111 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:36.637084    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a833c968-f463-4a3a-9451-57c0ee19a768-maas-api-openshift-ca" pod "a833c968-f463-4a3a-9451-57c0ee19a768" (UID: "a833c968-f463-4a3a-9451-57c0ee19a768"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:17:36.637360 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:36.637343    5570 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a833c968-f463-4a3a-9451-57c0ee19a768-maas-api-openshift-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:17:36.638904 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:36.638872    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a833c968-f463-4a3a-9451-57c0ee19a768-maas-api-xks-ca" pod "a833c968-f463-4a3a-9451-57c0ee19a768" (UID: "a833c968-f463-4a3a-9451-57c0ee19a768"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:17:36.639012 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:36.638954    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a833c968-f463-4a3a-9451-57c0ee19a768-kube-api-access-m5r8c" pod "a833c968-f463-4a3a-9451-57c0ee19a768" (UID: "a833c968-f463-4a3a-9451-57c0ee19a768"). InnerVolumeSpecName "kube-api-access-m5r8c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:17:36.732075 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:36.732030    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g69f8"]
Nov 03 20:17:36.738202 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:36.738168    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5r8c\" (UniqueName: \"kubernetes.io/projected/a833c968-f463-4a3a-9451-57c0ee19a768-kube-api-access-m5r8c\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:17:36.738202 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:36.738201    5570 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a833c968-f463-4a3a-9451-57c0ee19a768-maas-api-xks-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:17:37.397797 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:37.397121    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-649609db-pqmqh" event={"ID":"a833c968-f463-4a3a-9451-57c0ee19a768","Type":"ContainerDied","Data":"9fd01a16da6d8031f1c66770c6a11eef6f49e9eeca7fca400be185642517767e"}
Nov 03 20:17:37.397797 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:37.397190    5570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9fd01a16da6d8031f1c66770c6a11eef6f49e9eeca7fca400be185642517767e"
Nov 03 20:17:37.397797 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:37.397262    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-649609db-pqmqh"
Nov 03 20:17:40.712050 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:40.712013    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-z4kzq"]
Nov 03 20:17:40.712911 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:40.712886    5570 cpu_manager.go:414] "containerMap: removing container" podUID="a833c968-f463-4a3a-9451-57c0ee19a768" containerName="revoke-keys"
Nov 03 20:17:40.712911 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:40.712910    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a833c968-f463-4a3a-9451-57c0ee19a768" containerName="revoke-keys"
Nov 03 20:17:40.713127 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:40.713099    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="a833c968-f463-4a3a-9451-57c0ee19a768" containerName="revoke-keys"
Nov 03 20:17:40.717078 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:40.717054    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-z4kzq"
Nov 03 20:17:40.753584 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:40.753553    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-z4kzq"]
Nov 03 20:17:40.774683 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:40.774637    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e185c296-cd84-4b0c-9316-ef10085cda0a-istiod-ca-cert\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-z4kzq\" (UID: \"e185c296-cd84-4b0c-9316-ef10085cda0a\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-z4kzq"
Nov 03 20:17:40.774919 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:40.774709    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e185c296-cd84-4b0c-9316-ef10085cda0a-credential-socket\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-z4kzq\" (UID: \"e185c296-cd84-4b0c-9316-ef10085cda0a\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-z4kzq"
Nov 03 20:17:40.774919 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:40.774746    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e185c296-cd84-4b0c-9316-ef10085cda0a-istio-data\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-z4kzq\" (UID: \"e185c296-cd84-4b0c-9316-ef10085cda0a\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-z4kzq"
Nov 03 20:17:40.774919 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:40.774876    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4m4g\" (UniqueName: \"kubernetes.io/projected/e185c296-cd84-4b0c-9316-ef10085cda0a-kube-api-access-j4m4g\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-z4kzq\" (UID: \"e185c296-cd84-4b0c-9316-ef10085cda0a\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-z4kzq"
Nov 03 20:17:40.774919 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:40.774907    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e185c296-cd84-4b0c-9316-ef10085cda0a-istio-token\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-z4kzq\" (UID: \"e185c296-cd84-4b0c-9316-ef10085cda0a\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-z4kzq"
Nov 03 20:17:40.775148 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:40.774974    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e185c296-cd84-4b0c-9316-ef10085cda0a-istio-podinfo\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-z4kzq\" (UID: \"e185c296-cd84-4b0c-9316-ef10085cda0a\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-z4kzq"
Nov 03 20:17:40.775148 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:40.775040    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e185c296-cd84-4b0c-9316-ef10085cda0a-workload-socket\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-z4kzq\" (UID: \"e185c296-cd84-4b0c-9316-ef10085cda0a\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-z4kzq"
Nov 03 20:17:40.775148 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:40.775113    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e185c296-cd84-4b0c-9316-ef10085cda0a-istio-envoy\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-z4kzq\" (UID: \"e185c296-cd84-4b0c-9316-ef10085cda0a\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-z4kzq"
Nov 03 20:17:40.775305 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:40.775191    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e185c296-cd84-4b0c-9316-ef10085cda0a-workload-certs\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-z4kzq\" (UID: \"e185c296-cd84-4b0c-9316-ef10085cda0a\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-z4kzq"
Nov 03 20:17:40.877213 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:40.877166    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4m4g\" (UniqueName: \"kubernetes.io/projected/e185c296-cd84-4b0c-9316-ef10085cda0a-kube-api-access-j4m4g\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-z4kzq\" (UID: \"e185c296-cd84-4b0c-9316-ef10085cda0a\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-z4kzq"
Nov 03 20:17:40.877418 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:40.877227    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e185c296-cd84-4b0c-9316-ef10085cda0a-istio-token\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-z4kzq\" (UID: \"e185c296-cd84-4b0c-9316-ef10085cda0a\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-z4kzq"
Nov 03 20:17:40.877418 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:40.877298    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e185c296-cd84-4b0c-9316-ef10085cda0a-istio-podinfo\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-z4kzq\" (UID: \"e185c296-cd84-4b0c-9316-ef10085cda0a\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-z4kzq"
Nov 03 20:17:40.877418 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:40.877363    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e185c296-cd84-4b0c-9316-ef10085cda0a-workload-socket\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-z4kzq\" (UID: \"e185c296-cd84-4b0c-9316-ef10085cda0a\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-z4kzq"
Nov 03 20:17:40.877611 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:40.877440    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e185c296-cd84-4b0c-9316-ef10085cda0a-istio-envoy\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-z4kzq\" (UID: \"e185c296-cd84-4b0c-9316-ef10085cda0a\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-z4kzq"
Nov 03 20:17:40.877611 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:40.877513    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e185c296-cd84-4b0c-9316-ef10085cda0a-workload-certs\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-z4kzq\" (UID: \"e185c296-cd84-4b0c-9316-ef10085cda0a\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-z4kzq"
Nov 03 20:17:40.877611 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:40.877581    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e185c296-cd84-4b0c-9316-ef10085cda0a-istiod-ca-cert\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-z4kzq\" (UID: \"e185c296-cd84-4b0c-9316-ef10085cda0a\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-z4kzq"
Nov 03 20:17:40.877792 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:40.877636    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e185c296-cd84-4b0c-9316-ef10085cda0a-credential-socket\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-z4kzq\" (UID: \"e185c296-cd84-4b0c-9316-ef10085cda0a\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-z4kzq"
Nov 03 20:17:40.877792 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:40.877671    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e185c296-cd84-4b0c-9316-ef10085cda0a-istio-data\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-z4kzq\" (UID: \"e185c296-cd84-4b0c-9316-ef10085cda0a\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-z4kzq"
Nov 03 20:17:40.879129 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:40.879094    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e185c296-cd84-4b0c-9316-ef10085cda0a-workload-certs\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-z4kzq\" (UID: \"e185c296-cd84-4b0c-9316-ef10085cda0a\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-z4kzq"
Nov 03 20:17:40.879251 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:40.879219    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e185c296-cd84-4b0c-9316-ef10085cda0a-credential-socket\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-z4kzq\" (UID: \"e185c296-cd84-4b0c-9316-ef10085cda0a\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-z4kzq"
Nov 03 20:17:40.879402 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:40.879379    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e185c296-cd84-4b0c-9316-ef10085cda0a-workload-socket\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-z4kzq\" (UID: \"e185c296-cd84-4b0c-9316-ef10085cda0a\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-z4kzq"
Nov 03 20:17:40.882424 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:40.882393    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e185c296-cd84-4b0c-9316-ef10085cda0a-istio-envoy\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-z4kzq\" (UID: \"e185c296-cd84-4b0c-9316-ef10085cda0a\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-z4kzq"
Nov 03 20:17:40.883400 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:40.883378    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e185c296-cd84-4b0c-9316-ef10085cda0a-istio-data\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-z4kzq\" (UID: \"e185c296-cd84-4b0c-9316-ef10085cda0a\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-z4kzq"
Nov 03 20:17:40.884364 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:40.884335    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e185c296-cd84-4b0c-9316-ef10085cda0a-istiod-ca-cert\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-z4kzq\" (UID: \"e185c296-cd84-4b0c-9316-ef10085cda0a\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-z4kzq"
Nov 03 20:17:40.884490 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:40.884365    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e185c296-cd84-4b0c-9316-ef10085cda0a-istio-podinfo\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-z4kzq\" (UID: \"e185c296-cd84-4b0c-9316-ef10085cda0a\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-z4kzq"
Nov 03 20:17:40.900967 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:40.900937    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4m4g\" (UniqueName: \"kubernetes.io/projected/e185c296-cd84-4b0c-9316-ef10085cda0a-kube-api-access-j4m4g\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-z4kzq\" (UID: \"e185c296-cd84-4b0c-9316-ef10085cda0a\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-z4kzq"
Nov 03 20:17:40.902364 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:40.902329    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e185c296-cd84-4b0c-9316-ef10085cda0a-istio-token\") pod \"e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-z4kzq\" (UID: \"e185c296-cd84-4b0c-9316-ef10085cda0a\") " pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-z4kzq"
Nov 03 20:17:41.033620 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:41.033526    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-z4kzq"
Nov 03 20:17:41.359988 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:41.359895    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-z4kzq"]
Nov 03 20:17:41.378855 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:41.376902    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:17:41.378855 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:41.377004    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:17:41.378855 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:41.377051    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:17:41.430878 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:41.429571    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-z4kzq" event={"ID":"e185c296-cd84-4b0c-9316-ef10085cda0a","Type":"ContainerStarted","Data":"9a68094e8967403ffb6b2b568f761790b1fad27597c978030ff8c5badc3d1588"}
Nov 03 20:17:42.528231 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:42.528184    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-z4kzq" event={"ID":"e185c296-cd84-4b0c-9316-ef10085cda0a","Type":"ContainerStarted","Data":"3a3fe6ad7dbd552ebf8f796694c6eddbbde0140bca4dd3da4fe06ba8b279f1bc"}
Nov 03 20:17:42.614861 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:42.614794    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-z4kzq" podStartSLOduration=2.61476132 podStartE2EDuration="2.61476132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:17:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:17:42.598263933 +0000 UTC m=+796.750925833" watchObservedRunningTime="2026-11-03 20:17:42.61476132 +0000 UTC m=+796.767423208"
Nov 03 20:17:43.034516 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:43.034427    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-z4kzq"
Nov 03 20:17:43.035850 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:43.035818    5570 patch_prober.go:29] interesting pod/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-z4kzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Nov 03 20:17:43.036029 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:43.035902    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-z4kzq" podUID="e185c296-cd84-4b0c-9316-ef10085cda0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Nov 03 20:17:44.034798 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:44.034186    5570 patch_prober.go:29] interesting pod/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-z4kzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Nov 03 20:17:44.034798 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:44.034277    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-z4kzq" podUID="e185c296-cd84-4b0c-9316-ef10085cda0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Nov 03 20:17:45.034914 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:45.034870    5570 patch_prober.go:29] interesting pod/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-z4kzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Nov 03 20:17:45.035333 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:45.034941    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-z4kzq" podUID="e185c296-cd84-4b0c-9316-ef10085cda0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Nov 03 20:17:45.463432 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:45.463384    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71203a3a-hx4zg"]
Nov 03 20:17:45.468520 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:45.468495    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71203a3a-hx4zg"
Nov 03 20:17:45.476962 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:45.476934    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71203a3a-hx4zg"]
Nov 03 20:17:45.486738 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:45.486696    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:17:45.486957 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:45.486793    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l7s5w\""
Nov 03 20:17:45.605478 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:45.605439    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7f12100-03b3-44fe-ac21-e8504a730164-maas-api-xks-ca\") pod \"maas-api-revoke-sub-71203a3a-hx4zg\" (UID: \"c7f12100-03b3-44fe-ac21-e8504a730164\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71203a3a-hx4zg"
Nov 03 20:17:45.605642 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:45.605520    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7f12100-03b3-44fe-ac21-e8504a730164-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-71203a3a-hx4zg\" (UID: \"c7f12100-03b3-44fe-ac21-e8504a730164\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71203a3a-hx4zg"
Nov 03 20:17:45.605710 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:45.605632    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzgmq\" (UniqueName: \"kubernetes.io/projected/c7f12100-03b3-44fe-ac21-e8504a730164-kube-api-access-hzgmq\") pod \"maas-api-revoke-sub-71203a3a-hx4zg\" (UID: \"c7f12100-03b3-44fe-ac21-e8504a730164\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71203a3a-hx4zg"
Nov 03 20:17:45.708053 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:45.708008    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7f12100-03b3-44fe-ac21-e8504a730164-maas-api-xks-ca\") pod \"maas-api-revoke-sub-71203a3a-hx4zg\" (UID: \"c7f12100-03b3-44fe-ac21-e8504a730164\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71203a3a-hx4zg"
Nov 03 20:17:45.708257 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:45.708081    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7f12100-03b3-44fe-ac21-e8504a730164-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-71203a3a-hx4zg\" (UID: \"c7f12100-03b3-44fe-ac21-e8504a730164\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71203a3a-hx4zg"
Nov 03 20:17:45.708257 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:45.708178    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzgmq\" (UniqueName: \"kubernetes.io/projected/c7f12100-03b3-44fe-ac21-e8504a730164-kube-api-access-hzgmq\") pod \"maas-api-revoke-sub-71203a3a-hx4zg\" (UID: \"c7f12100-03b3-44fe-ac21-e8504a730164\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71203a3a-hx4zg"
Nov 03 20:17:45.709474 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:45.709447    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7f12100-03b3-44fe-ac21-e8504a730164-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-71203a3a-hx4zg\" (UID: \"c7f12100-03b3-44fe-ac21-e8504a730164\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71203a3a-hx4zg"
Nov 03 20:17:45.710912 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:45.710892    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7f12100-03b3-44fe-ac21-e8504a730164-maas-api-xks-ca\") pod \"maas-api-revoke-sub-71203a3a-hx4zg\" (UID: \"c7f12100-03b3-44fe-ac21-e8504a730164\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71203a3a-hx4zg"
Nov 03 20:17:45.728152 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:45.728072    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzgmq\" (UniqueName: \"kubernetes.io/projected/c7f12100-03b3-44fe-ac21-e8504a730164-kube-api-access-hzgmq\") pod \"maas-api-revoke-sub-71203a3a-hx4zg\" (UID: \"c7f12100-03b3-44fe-ac21-e8504a730164\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71203a3a-hx4zg"
Nov 03 20:17:45.789096 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:45.789052    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71203a3a-hx4zg"
Nov 03 20:17:46.034589 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:46.034503    5570 patch_prober.go:29] interesting pod/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-z4kzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Nov 03 20:17:46.034589 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:46.034559    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-z4kzq" podUID="e185c296-cd84-4b0c-9316-ef10085cda0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Nov 03 20:17:46.121507 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:46.121476    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71203a3a-hx4zg"]
Nov 03 20:17:46.123704 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:17:46.123676    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7f12100_03b3_44fe_ac21_e8504a730164.slice/crio-777bf0cc866cf4d74f6a376037899394627b2905a9d2f313534b97df093808a5 WatchSource:0}: Error finding container 777bf0cc866cf4d74f6a376037899394627b2905a9d2f313534b97df093808a5: Status 404 returned error can't find the container with id 777bf0cc866cf4d74f6a376037899394627b2905a9d2f313534b97df093808a5
Nov 03 20:17:46.580395 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:46.580365    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71203a3a-hx4zg" event={"ID":"c7f12100-03b3-44fe-ac21-e8504a730164","Type":"ContainerStarted","Data":"777bf0cc866cf4d74f6a376037899394627b2905a9d2f313534b97df093808a5"}
Nov 03 20:17:47.034681 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:47.034648    5570 patch_prober.go:29] interesting pod/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-z4kzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Nov 03 20:17:47.034894 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:47.034713    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-z4kzq" podUID="e185c296-cd84-4b0c-9316-ef10085cda0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Nov 03 20:17:47.587916 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:47.587861    5570 generic.go:356] "Generic (PLEG): container finished" podID="c7f12100-03b3-44fe-ac21-e8504a730164" containerID="1e965ca4919ebe84922b40a1c631ea5e6cf63909c32fda4a403b7f217fa3185b" exitCode=0
Nov 03 20:17:47.588370 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:47.587941    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71203a3a-hx4zg" event={"ID":"c7f12100-03b3-44fe-ac21-e8504a730164","Type":"ContainerDied","Data":"1e965ca4919ebe84922b40a1c631ea5e6cf63909c32fda4a403b7f217fa3185b"}
Nov 03 20:17:48.289760 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:48.289729    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-z4kzq"
Nov 03 20:17:48.290202 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:48.290166    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-z4kzq"
Nov 03 20:17:48.291398 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:48.291373    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-z4kzq"
Nov 03 20:17:48.767636 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:48.767610    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71203a3a-hx4zg"
Nov 03 20:17:48.875547 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:48.875448    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c7f12100-03b3-44fe-ac21-e8504a730164-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7f12100-03b3-44fe-ac21-e8504a730164-maas-api-xks-ca\") pod \"c7f12100-03b3-44fe-ac21-e8504a730164\" (UID: \"c7f12100-03b3-44fe-ac21-e8504a730164\") "
Nov 03 20:17:48.875728 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:48.875657    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7f12100-03b3-44fe-ac21-e8504a730164-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7f12100-03b3-44fe-ac21-e8504a730164-maas-api-openshift-ca\") pod \"c7f12100-03b3-44fe-ac21-e8504a730164\" (UID: \"c7f12100-03b3-44fe-ac21-e8504a730164\") "
Nov 03 20:17:48.875728 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:48.875699    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7f12100-03b3-44fe-ac21-e8504a730164-kube-api-access-hzgmq\" (UniqueName: \"kubernetes.io/projected/c7f12100-03b3-44fe-ac21-e8504a730164-kube-api-access-hzgmq\") pod \"c7f12100-03b3-44fe-ac21-e8504a730164\" (UID: \"c7f12100-03b3-44fe-ac21-e8504a730164\") "
Nov 03 20:17:48.876156 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:48.876121    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7f12100-03b3-44fe-ac21-e8504a730164-maas-api-openshift-ca" pod "c7f12100-03b3-44fe-ac21-e8504a730164" (UID: "c7f12100-03b3-44fe-ac21-e8504a730164"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:17:48.876331 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:48.876309    5570 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7f12100-03b3-44fe-ac21-e8504a730164-maas-api-openshift-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:17:48.878251 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:48.878219    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7f12100-03b3-44fe-ac21-e8504a730164-maas-api-xks-ca" pod "c7f12100-03b3-44fe-ac21-e8504a730164" (UID: "c7f12100-03b3-44fe-ac21-e8504a730164"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:17:48.878348 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:48.878304    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7f12100-03b3-44fe-ac21-e8504a730164-kube-api-access-hzgmq" pod "c7f12100-03b3-44fe-ac21-e8504a730164" (UID: "c7f12100-03b3-44fe-ac21-e8504a730164"). InnerVolumeSpecName "kube-api-access-hzgmq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:17:48.977346 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:48.977311    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzgmq\" (UniqueName: \"kubernetes.io/projected/c7f12100-03b3-44fe-ac21-e8504a730164-kube-api-access-hzgmq\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:17:48.977346 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:48.977342    5570 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7f12100-03b3-44fe-ac21-e8504a730164-maas-api-xks-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:17:49.602464 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:49.602433    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71203a3a-hx4zg"
Nov 03 20:17:49.602655 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:49.602464    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71203a3a-hx4zg" event={"ID":"c7f12100-03b3-44fe-ac21-e8504a730164","Type":"ContainerDied","Data":"777bf0cc866cf4d74f6a376037899394627b2905a9d2f313534b97df093808a5"}
Nov 03 20:17:49.602655 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:49.602564    5570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="777bf0cc866cf4d74f6a376037899394627b2905a9d2f313534b97df093808a5"
Nov 03 20:17:52.813801 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:52.808942    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/premium-w3-482722-kserve-689dc7bd5d-x8rjp" podUID="369045c3-4d6a-49a7-9bc6-f2939883bee8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:17:52.902026 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:52.895901    5570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-6fc748/premium-w1-6fc748-kserve-7474db9cd7-nf2xk" podUID="830e062e-83fc-4de7-91a7-5534dac1d46b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:17:53.208925 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:53.208869    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/premium-w4-e80203-kserve-5f9f445456-bms9d" podUID="6ad76efc-20c2-4951-b22a-c5a39cd39ed4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:17:53.262932 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:53.262872    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh" podUID="2d3647f2-89e9-4908-a8ee-d84e527abeb0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:17:53.290583 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:53.290537    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/premium-w2-9973d9-kserve-5c6dd4669c-5gv8x" podUID="cefc89ad-7ae1-433b-979d-6601678a6d98" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:17:53.355798 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:53.355486    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz" podUID="3a715ef1-38e9-4386-9eec-e10e8df11a10" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": context deadline exceeded"
Nov 03 20:17:53.578500 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:53.577945    5570 patch_prober.go:29] interesting pod/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmqs6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:17:53.578500 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:53.578021    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmqs6" podUID="516f80b2-26e4-44d2-b97a-6e740d0b53e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:17:53.596003 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:53.595891    5570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-e80203/premium-w4-e80203-kserve-5f9f445456-bms9d" podUID="6ad76efc-20c2-4951-b22a-c5a39cd39ed4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:17:53.640169 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:53.640130    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/simulator-w0-e40816-kserve-688f79796f-2tvw5" podUID="c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Nov 03 20:17:53.811972 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:53.811917    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/premium-w3-482722-kserve-689dc7bd5d-x8rjp" podUID="369045c3-4d6a-49a7-9bc6-f2939883bee8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:17:53.813079 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:53.813055    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-482722/premium-w3-482722-kserve-689dc7bd5d-x8rjp"
Nov 03 20:17:54.237325 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:54.236580    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/premium-w4-e80203-kserve-5f9f445456-bms9d" podUID="6ad76efc-20c2-4951-b22a-c5a39cd39ed4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:17:54.237325 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:54.237281    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-e80203/premium-w4-e80203-kserve-5f9f445456-bms9d"
Nov 03 20:17:54.295798 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:54.292649    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/premium-w2-9973d9-kserve-5c6dd4669c-5gv8x" podUID="cefc89ad-7ae1-433b-979d-6601678a6d98" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:17:54.295798 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:54.293467    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9973d9/premium-w2-9973d9-kserve-5c6dd4669c-5gv8x"
Nov 03 20:17:54.398794 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:54.398732    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz" podUID="3a715ef1-38e9-4386-9eec-e10e8df11a10" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:17:54.399321 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:54.399296    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz"
Nov 03 20:17:54.446793 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:54.439742    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/premium-w1-6fc748-kserve-7474db9cd7-nf2xk" podUID="830e062e-83fc-4de7-91a7-5534dac1d46b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Nov 03 20:17:54.450592 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:54.450322    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9973d9/premium-w2-9973d9-kserve-5c6dd4669c-5gv8x"
Nov 03 20:17:54.450592 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:54.450504    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-e80203/premium-w4-e80203-kserve-5f9f445456-bms9d"
Nov 03 20:17:54.472211 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:54.472184    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz"
Nov 03 20:17:54.473493 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:54.473420    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-482722/premium-w3-482722-kserve-689dc7bd5d-x8rjp"
Nov 03 20:17:57.229537 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:17:57.229500    5570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8bwhz" containerName="extractor"
Nov 03 20:17:57.706984 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:57.706950    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca60b04a-rnj6h"]
Nov 03 20:17:57.707641 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:57.707619    5570 cpu_manager.go:414] "containerMap: removing container" podUID="c7f12100-03b3-44fe-ac21-e8504a730164" containerName="revoke-keys"
Nov 03 20:17:57.707751 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:57.707642    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7f12100-03b3-44fe-ac21-e8504a730164" containerName="revoke-keys"
Nov 03 20:17:57.707875 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:57.707756    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7f12100-03b3-44fe-ac21-e8504a730164" containerName="revoke-keys"
Nov 03 20:17:57.712316 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:57.712293    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca60b04a-rnj6h"
Nov 03 20:17:57.727428 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:57.727395    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca60b04a-rnj6h"]
Nov 03 20:17:57.737630 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:57.737597    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:17:57.737822 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:57.737662    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l7s5w\""
Nov 03 20:17:57.765675 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:57.765643    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10096163-2b34-48f3-95fa-cdd9798a9fdf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ca60b04a-rnj6h\" (UID: \"10096163-2b34-48f3-95fa-cdd9798a9fdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca60b04a-rnj6h"
Nov 03 20:17:57.765895 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:57.765688    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kttk4\" (UniqueName: \"kubernetes.io/projected/10096163-2b34-48f3-95fa-cdd9798a9fdf-kube-api-access-kttk4\") pod \"maas-api-revoke-sub-ca60b04a-rnj6h\" (UID: \"10096163-2b34-48f3-95fa-cdd9798a9fdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca60b04a-rnj6h"
Nov 03 20:17:57.765974 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:57.765952    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10096163-2b34-48f3-95fa-cdd9798a9fdf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ca60b04a-rnj6h\" (UID: \"10096163-2b34-48f3-95fa-cdd9798a9fdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca60b04a-rnj6h"
Nov 03 20:17:57.867380 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:57.867341    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10096163-2b34-48f3-95fa-cdd9798a9fdf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ca60b04a-rnj6h\" (UID: \"10096163-2b34-48f3-95fa-cdd9798a9fdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca60b04a-rnj6h"
Nov 03 20:17:57.867592 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:57.867445    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10096163-2b34-48f3-95fa-cdd9798a9fdf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ca60b04a-rnj6h\" (UID: \"10096163-2b34-48f3-95fa-cdd9798a9fdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca60b04a-rnj6h"
Nov 03 20:17:57.867592 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:57.867478    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kttk4\" (UniqueName: \"kubernetes.io/projected/10096163-2b34-48f3-95fa-cdd9798a9fdf-kube-api-access-kttk4\") pod \"maas-api-revoke-sub-ca60b04a-rnj6h\" (UID: \"10096163-2b34-48f3-95fa-cdd9798a9fdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca60b04a-rnj6h"
Nov 03 20:17:57.868133 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:57.868103    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10096163-2b34-48f3-95fa-cdd9798a9fdf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ca60b04a-rnj6h\" (UID: \"10096163-2b34-48f3-95fa-cdd9798a9fdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca60b04a-rnj6h"
Nov 03 20:17:57.869878 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:57.869860    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10096163-2b34-48f3-95fa-cdd9798a9fdf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ca60b04a-rnj6h\" (UID: \"10096163-2b34-48f3-95fa-cdd9798a9fdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca60b04a-rnj6h"
Nov 03 20:17:57.888299 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:57.888268    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kttk4\" (UniqueName: \"kubernetes.io/projected/10096163-2b34-48f3-95fa-cdd9798a9fdf-kube-api-access-kttk4\") pod \"maas-api-revoke-sub-ca60b04a-rnj6h\" (UID: \"10096163-2b34-48f3-95fa-cdd9798a9fdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca60b04a-rnj6h"
Nov 03 20:17:58.026739 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:58.026657    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca60b04a-rnj6h"
Nov 03 20:17:58.328983 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:58.327751    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca60b04a-rnj6h"]
Nov 03 20:17:58.328983 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:17:58.328113    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10096163_2b34_48f3_95fa_cdd9798a9fdf.slice/crio-0233e5a71a9d02598e3e380c1987181b2cb3705a0c810999b11c1ea077fbc687 WatchSource:0}: Error finding container 0233e5a71a9d02598e3e380c1987181b2cb3705a0c810999b11c1ea077fbc687: Status 404 returned error can't find the container with id 0233e5a71a9d02598e3e380c1987181b2cb3705a0c810999b11c1ea077fbc687
Nov 03 20:17:58.742755 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:58.742717    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca60b04a-rnj6h" event={"ID":"10096163-2b34-48f3-95fa-cdd9798a9fdf","Type":"ContainerStarted","Data":"f8fe2380352955f74cba24b5ca77339070e0316b4fc02555293f3c33c84b0856"}
Nov 03 20:17:58.742976 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:58.742869    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca60b04a-rnj6h" event={"ID":"10096163-2b34-48f3-95fa-cdd9798a9fdf","Type":"ContainerStarted","Data":"0233e5a71a9d02598e3e380c1987181b2cb3705a0c810999b11c1ea077fbc687"}
Nov 03 20:17:58.785910 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:58.785853    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca60b04a-rnj6h" podStartSLOduration=1.7858381269999999 podStartE2EDuration="1.785838127s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:17:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:17:58.782666945 +0000 UTC m=+812.935328832" watchObservedRunningTime="2026-11-03 20:17:58.785838127 +0000 UTC m=+812.938500014"
Nov 03 20:17:59.750121 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:59.750085    5570 generic.go:356] "Generic (PLEG): container finished" podID="10096163-2b34-48f3-95fa-cdd9798a9fdf" containerID="f8fe2380352955f74cba24b5ca77339070e0316b4fc02555293f3c33c84b0856" exitCode=0
Nov 03 20:17:59.750528 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:17:59.750175    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca60b04a-rnj6h" event={"ID":"10096163-2b34-48f3-95fa-cdd9798a9fdf","Type":"ContainerDied","Data":"f8fe2380352955f74cba24b5ca77339070e0316b4fc02555293f3c33c84b0856"}
Nov 03 20:18:00.828919 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:00.828880    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-149c62d4-x6xrj"]
Nov 03 20:18:00.868148 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:00.867174    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-149c62d4-x6xrj"]
Nov 03 20:18:00.868148 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:00.867299    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-149c62d4-x6xrj"
Nov 03 20:18:00.975804 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:00.963043    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdnn6\" (UniqueName: \"kubernetes.io/projected/08701155-f481-4997-b146-ebc997e838c5-kube-api-access-sdnn6\") pod \"maas-api-revoke-sub-149c62d4-x6xrj\" (UID: \"08701155-f481-4997-b146-ebc997e838c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-149c62d4-x6xrj"
Nov 03 20:18:00.975804 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:00.963109    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08701155-f481-4997-b146-ebc997e838c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-149c62d4-x6xrj\" (UID: \"08701155-f481-4997-b146-ebc997e838c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-149c62d4-x6xrj"
Nov 03 20:18:00.975804 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:00.963172    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08701155-f481-4997-b146-ebc997e838c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-149c62d4-x6xrj\" (UID: \"08701155-f481-4997-b146-ebc997e838c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-149c62d4-x6xrj"
Nov 03 20:18:01.064859 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:01.064751    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdnn6\" (UniqueName: \"kubernetes.io/projected/08701155-f481-4997-b146-ebc997e838c5-kube-api-access-sdnn6\") pod \"maas-api-revoke-sub-149c62d4-x6xrj\" (UID: \"08701155-f481-4997-b146-ebc997e838c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-149c62d4-x6xrj"
Nov 03 20:18:01.065132 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:01.064850    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08701155-f481-4997-b146-ebc997e838c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-149c62d4-x6xrj\" (UID: \"08701155-f481-4997-b146-ebc997e838c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-149c62d4-x6xrj"
Nov 03 20:18:01.065226 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:01.065140    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08701155-f481-4997-b146-ebc997e838c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-149c62d4-x6xrj\" (UID: \"08701155-f481-4997-b146-ebc997e838c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-149c62d4-x6xrj"
Nov 03 20:18:01.067254 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:01.067112    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08701155-f481-4997-b146-ebc997e838c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-149c62d4-x6xrj\" (UID: \"08701155-f481-4997-b146-ebc997e838c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-149c62d4-x6xrj"
Nov 03 20:18:01.072055 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:01.072030    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08701155-f481-4997-b146-ebc997e838c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-149c62d4-x6xrj\" (UID: \"08701155-f481-4997-b146-ebc997e838c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-149c62d4-x6xrj"
Nov 03 20:18:01.091634 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:01.091531    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdnn6\" (UniqueName: \"kubernetes.io/projected/08701155-f481-4997-b146-ebc997e838c5-kube-api-access-sdnn6\") pod \"maas-api-revoke-sub-149c62d4-x6xrj\" (UID: \"08701155-f481-4997-b146-ebc997e838c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-149c62d4-x6xrj"
Nov 03 20:18:01.149531 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:01.149499    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca60b04a-rnj6h"
Nov 03 20:18:01.267954 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:01.267916    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/10096163-2b34-48f3-95fa-cdd9798a9fdf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10096163-2b34-48f3-95fa-cdd9798a9fdf-maas-api-xks-ca\") pod \"10096163-2b34-48f3-95fa-cdd9798a9fdf\" (UID: \"10096163-2b34-48f3-95fa-cdd9798a9fdf\") "
Nov 03 20:18:01.268145 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:01.267995    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10096163-2b34-48f3-95fa-cdd9798a9fdf-kube-api-access-kttk4\" (UniqueName: \"kubernetes.io/projected/10096163-2b34-48f3-95fa-cdd9798a9fdf-kube-api-access-kttk4\") pod \"10096163-2b34-48f3-95fa-cdd9798a9fdf\" (UID: \"10096163-2b34-48f3-95fa-cdd9798a9fdf\") "
Nov 03 20:18:01.268203 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:01.268150    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10096163-2b34-48f3-95fa-cdd9798a9fdf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10096163-2b34-48f3-95fa-cdd9798a9fdf-maas-api-openshift-ca\") pod \"10096163-2b34-48f3-95fa-cdd9798a9fdf\" (UID: \"10096163-2b34-48f3-95fa-cdd9798a9fdf\") "
Nov 03 20:18:01.268592 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:01.268555    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10096163-2b34-48f3-95fa-cdd9798a9fdf-maas-api-openshift-ca" pod "10096163-2b34-48f3-95fa-cdd9798a9fdf" (UID: "10096163-2b34-48f3-95fa-cdd9798a9fdf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:18:01.270063 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:01.270036    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10096163-2b34-48f3-95fa-cdd9798a9fdf-maas-api-xks-ca" pod "10096163-2b34-48f3-95fa-cdd9798a9fdf" (UID: "10096163-2b34-48f3-95fa-cdd9798a9fdf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:18:01.270170 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:01.270130    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10096163-2b34-48f3-95fa-cdd9798a9fdf-kube-api-access-kttk4" pod "10096163-2b34-48f3-95fa-cdd9798a9fdf" (UID: "10096163-2b34-48f3-95fa-cdd9798a9fdf"). InnerVolumeSpecName "kube-api-access-kttk4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:18:01.360868 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:01.360767    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-149c62d4-x6xrj"
Nov 03 20:18:01.369898 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:01.369869    5570 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10096163-2b34-48f3-95fa-cdd9798a9fdf-maas-api-xks-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:18:01.370006 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:01.369905    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kttk4\" (UniqueName: \"kubernetes.io/projected/10096163-2b34-48f3-95fa-cdd9798a9fdf-kube-api-access-kttk4\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:18:01.370006 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:01.369921    5570 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10096163-2b34-48f3-95fa-cdd9798a9fdf-maas-api-openshift-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:18:01.666540 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:01.666507    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-149c62d4-x6xrj"]
Nov 03 20:18:01.667844 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:18:01.667814    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08701155_f481_4997_b146_ebc997e838c5.slice/crio-1245f3432c305af8f4711c6cc5a7bfdf54cb16b13a7df254f8176547e82847d2 WatchSource:0}: Error finding container 1245f3432c305af8f4711c6cc5a7bfdf54cb16b13a7df254f8176547e82847d2: Status 404 returned error can't find the container with id 1245f3432c305af8f4711c6cc5a7bfdf54cb16b13a7df254f8176547e82847d2
Nov 03 20:18:01.827604 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:01.827547    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca60b04a-rnj6h"
Nov 03 20:18:01.827604 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:01.827582    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca60b04a-rnj6h" event={"ID":"10096163-2b34-48f3-95fa-cdd9798a9fdf","Type":"ContainerDied","Data":"0233e5a71a9d02598e3e380c1987181b2cb3705a0c810999b11c1ea077fbc687"}
Nov 03 20:18:01.827888 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:01.827633    5570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0233e5a71a9d02598e3e380c1987181b2cb3705a0c810999b11c1ea077fbc687"
Nov 03 20:18:01.829260 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:01.829232    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-149c62d4-x6xrj" event={"ID":"08701155-f481-4997-b146-ebc997e838c5","Type":"ContainerStarted","Data":"1245f3432c305af8f4711c6cc5a7bfdf54cb16b13a7df254f8176547e82847d2"}
Nov 03 20:18:02.836396 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:02.836314    5570 generic.go:356] "Generic (PLEG): container finished" podID="08701155-f481-4997-b146-ebc997e838c5" containerID="4ae68a259a5b7161d3d33ad6695f2104469a7b23847066d880e5d5985a4fc31b" exitCode=0
Nov 03 20:18:02.836753 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:02.836403    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-149c62d4-x6xrj" event={"ID":"08701155-f481-4997-b146-ebc997e838c5","Type":"ContainerDied","Data":"4ae68a259a5b7161d3d33ad6695f2104469a7b23847066d880e5d5985a4fc31b"}
Nov 03 20:18:03.993667 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:03.993638    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-149c62d4-x6xrj"
Nov 03 20:18:04.112216 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:04.112126    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08701155-f481-4997-b146-ebc997e838c5-kube-api-access-sdnn6\" (UniqueName: \"kubernetes.io/projected/08701155-f481-4997-b146-ebc997e838c5-kube-api-access-sdnn6\") pod \"08701155-f481-4997-b146-ebc997e838c5\" (UID: \"08701155-f481-4997-b146-ebc997e838c5\") "
Nov 03 20:18:04.112374 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:04.112227    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08701155-f481-4997-b146-ebc997e838c5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08701155-f481-4997-b146-ebc997e838c5-maas-api-openshift-ca\") pod \"08701155-f481-4997-b146-ebc997e838c5\" (UID: \"08701155-f481-4997-b146-ebc997e838c5\") "
Nov 03 20:18:04.112374 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:04.112299    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/08701155-f481-4997-b146-ebc997e838c5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08701155-f481-4997-b146-ebc997e838c5-maas-api-xks-ca\") pod \"08701155-f481-4997-b146-ebc997e838c5\" (UID: \"08701155-f481-4997-b146-ebc997e838c5\") "
Nov 03 20:18:04.112619 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:04.112586    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08701155-f481-4997-b146-ebc997e838c5-maas-api-openshift-ca" pod "08701155-f481-4997-b146-ebc997e838c5" (UID: "08701155-f481-4997-b146-ebc997e838c5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:18:04.113175 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:04.113149    5570 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08701155-f481-4997-b146-ebc997e838c5-maas-api-openshift-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:18:04.114397 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:04.114370    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08701155-f481-4997-b146-ebc997e838c5-maas-api-xks-ca" pod "08701155-f481-4997-b146-ebc997e838c5" (UID: "08701155-f481-4997-b146-ebc997e838c5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:18:04.114495 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:04.114407    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08701155-f481-4997-b146-ebc997e838c5-kube-api-access-sdnn6" pod "08701155-f481-4997-b146-ebc997e838c5" (UID: "08701155-f481-4997-b146-ebc997e838c5"). InnerVolumeSpecName "kube-api-access-sdnn6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:18:04.214501 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:04.214470    5570 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08701155-f481-4997-b146-ebc997e838c5-maas-api-xks-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:18:04.214501 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:04.214498    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdnn6\" (UniqueName: \"kubernetes.io/projected/08701155-f481-4997-b146-ebc997e838c5-kube-api-access-sdnn6\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:18:04.849183 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:04.849154    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-149c62d4-x6xrj"
Nov 03 20:18:04.849183 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:04.849168    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-149c62d4-x6xrj" event={"ID":"08701155-f481-4997-b146-ebc997e838c5","Type":"ContainerDied","Data":"1245f3432c305af8f4711c6cc5a7bfdf54cb16b13a7df254f8176547e82847d2"}
Nov 03 20:18:04.849402 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:04.849216    5570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1245f3432c305af8f4711c6cc5a7bfdf54cb16b13a7df254f8176547e82847d2"
Nov 03 20:18:05.646785 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:05.646743    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g69f8"]
Nov 03 20:18:07.281859 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:07.269287    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh" podUID="2d3647f2-89e9-4908-a8ee-d84e527abeb0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:18:07.368822 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:07.368083    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/premium-w1-6fc748-kserve-7474db9cd7-nf2xk" podUID="830e062e-83fc-4de7-91a7-5534dac1d46b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:18:07.469504 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:07.464815    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/premium-w4-e80203-kserve-5f9f445456-bms9d" podUID="6ad76efc-20c2-4951-b22a-c5a39cd39ed4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:18:07.640858 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:07.639885    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/simulator-w0-e40816-kserve-688f79796f-2tvw5" podUID="c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:18:07.701814 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:07.695885    5570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-9e1bcb/premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh" podUID="2d3647f2-89e9-4908-a8ee-d84e527abeb0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": context deadline exceeded"
Nov 03 20:18:08.270925 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:08.270869    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh" podUID="2d3647f2-89e9-4908-a8ee-d84e527abeb0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Nov 03 20:18:08.272347 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:08.272322    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-9e1bcb/premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh"
Nov 03 20:18:08.352924 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:08.352868    5570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-e40816/simulator-w0-e40816-kserve-688f79796f-2tvw5" podUID="c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:18:08.368937 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:08.368887    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/premium-w1-6fc748-kserve-7474db9cd7-nf2xk" podUID="830e062e-83fc-4de7-91a7-5534dac1d46b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:18:08.369572 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:08.369551    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6fc748/premium-w1-6fc748-kserve-7474db9cd7-nf2xk"
Nov 03 20:18:08.485974 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:08.485917    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/premium-w4-e80203-kserve-5f9f445456-bms9d" podUID="6ad76efc-20c2-4951-b22a-c5a39cd39ed4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:18:08.486671 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:08.486646    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-e80203/premium-w4-e80203-kserve-5f9f445456-bms9d"
Nov 03 20:18:08.523862 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:08.497572    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/premium-w3-482722-kserve-689dc7bd5d-x8rjp" podUID="369045c3-4d6a-49a7-9bc6-f2939883bee8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Nov 03 20:18:08.531801 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:08.497606    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz" podUID="3a715ef1-38e9-4386-9eec-e10e8df11a10" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": context deadline exceeded"
Nov 03 20:18:08.608799 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:08.607909    5570 patch_prober.go:29] interesting pod/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmqs6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:18:08.608799 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:08.607984    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmqs6" podUID="516f80b2-26e4-44d2-b97a-6e740d0b53e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:18:08.660847 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:08.655290    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/simulator-w0-e40816-kserve-688f79796f-2tvw5" podUID="c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:18:08.660847 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:08.655729    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e40816/simulator-w0-e40816-kserve-688f79796f-2tvw5"
Nov 03 20:18:08.880269 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:08.880178    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-e80203/premium-w4-e80203-kserve-5f9f445456-bms9d"
Nov 03 20:18:09.031878 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:09.030887    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6fc748/premium-w1-6fc748-kserve-7474db9cd7-nf2xk"
Nov 03 20:18:09.035177 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:09.034535    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-9e1bcb/premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh"
Nov 03 20:18:09.047491 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:09.044822    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e40816/simulator-w0-e40816-kserve-688f79796f-2tvw5"
Nov 03 20:18:10.635557 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:10.635515    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-wv4cn"]
Nov 03 20:18:10.636199 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:10.636176    5570 cpu_manager.go:414] "containerMap: removing container" podUID="10096163-2b34-48f3-95fa-cdd9798a9fdf" containerName="revoke-keys"
Nov 03 20:18:10.636199 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:10.636198    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10096163-2b34-48f3-95fa-cdd9798a9fdf" containerName="revoke-keys"
Nov 03 20:18:10.636342 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:10.636234    5570 cpu_manager.go:414] "containerMap: removing container" podUID="08701155-f481-4997-b146-ebc997e838c5" containerName="revoke-keys"
Nov 03 20:18:10.636342 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:10.636241    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08701155-f481-4997-b146-ebc997e838c5" containerName="revoke-keys"
Nov 03 20:18:10.636342 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:10.636324    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="08701155-f481-4997-b146-ebc997e838c5" containerName="revoke-keys"
Nov 03 20:18:10.636342 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:10.636334    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="10096163-2b34-48f3-95fa-cdd9798a9fdf" containerName="revoke-keys"
Nov 03 20:18:10.640650 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:10.640623    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-wv4cn"
Nov 03 20:18:10.664955 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:10.664920    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-wv4cn"]
Nov 03 20:18:10.745535 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:10.745498    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1e2eadf-ede0-4ea2-9eea-f25a39779795-istio-envoy\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-wv4cn\" (UID: \"c1e2eadf-ede0-4ea2-9eea-f25a39779795\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-wv4cn"
Nov 03 20:18:10.745535 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:10.745539    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1e2eadf-ede0-4ea2-9eea-f25a39779795-istio-token\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-wv4cn\" (UID: \"c1e2eadf-ede0-4ea2-9eea-f25a39779795\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-wv4cn"
Nov 03 20:18:10.745763 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:10.745632    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1e2eadf-ede0-4ea2-9eea-f25a39779795-workload-socket\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-wv4cn\" (UID: \"c1e2eadf-ede0-4ea2-9eea-f25a39779795\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-wv4cn"
Nov 03 20:18:10.745763 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:10.745652    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1e2eadf-ede0-4ea2-9eea-f25a39779795-workload-certs\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-wv4cn\" (UID: \"c1e2eadf-ede0-4ea2-9eea-f25a39779795\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-wv4cn"
Nov 03 20:18:10.745763 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:10.745674    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6kv7\" (UniqueName: \"kubernetes.io/projected/c1e2eadf-ede0-4ea2-9eea-f25a39779795-kube-api-access-g6kv7\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-wv4cn\" (UID: \"c1e2eadf-ede0-4ea2-9eea-f25a39779795\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-wv4cn"
Nov 03 20:18:10.745763 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:10.745729    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1e2eadf-ede0-4ea2-9eea-f25a39779795-istio-podinfo\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-wv4cn\" (UID: \"c1e2eadf-ede0-4ea2-9eea-f25a39779795\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-wv4cn"
Nov 03 20:18:10.745922 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:10.745837    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1e2eadf-ede0-4ea2-9eea-f25a39779795-credential-socket\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-wv4cn\" (UID: \"c1e2eadf-ede0-4ea2-9eea-f25a39779795\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-wv4cn"
Nov 03 20:18:10.745980 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:10.745956    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1e2eadf-ede0-4ea2-9eea-f25a39779795-istiod-ca-cert\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-wv4cn\" (UID: \"c1e2eadf-ede0-4ea2-9eea-f25a39779795\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-wv4cn"
Nov 03 20:18:10.746086 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:10.746067    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1e2eadf-ede0-4ea2-9eea-f25a39779795-istio-data\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-wv4cn\" (UID: \"c1e2eadf-ede0-4ea2-9eea-f25a39779795\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-wv4cn"
Nov 03 20:18:10.847412 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:10.847371    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1e2eadf-ede0-4ea2-9eea-f25a39779795-istio-envoy\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-wv4cn\" (UID: \"c1e2eadf-ede0-4ea2-9eea-f25a39779795\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-wv4cn"
Nov 03 20:18:10.847412 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:10.847411    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1e2eadf-ede0-4ea2-9eea-f25a39779795-istio-token\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-wv4cn\" (UID: \"c1e2eadf-ede0-4ea2-9eea-f25a39779795\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-wv4cn"
Nov 03 20:18:10.847636 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:10.847464    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1e2eadf-ede0-4ea2-9eea-f25a39779795-workload-socket\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-wv4cn\" (UID: \"c1e2eadf-ede0-4ea2-9eea-f25a39779795\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-wv4cn"
Nov 03 20:18:10.847636 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:10.847481    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1e2eadf-ede0-4ea2-9eea-f25a39779795-workload-certs\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-wv4cn\" (UID: \"c1e2eadf-ede0-4ea2-9eea-f25a39779795\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-wv4cn"
Nov 03 20:18:10.847636 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:10.847502    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6kv7\" (UniqueName: \"kubernetes.io/projected/c1e2eadf-ede0-4ea2-9eea-f25a39779795-kube-api-access-g6kv7\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-wv4cn\" (UID: \"c1e2eadf-ede0-4ea2-9eea-f25a39779795\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-wv4cn"
Nov 03 20:18:10.847636 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:10.847552    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1e2eadf-ede0-4ea2-9eea-f25a39779795-istio-podinfo\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-wv4cn\" (UID: \"c1e2eadf-ede0-4ea2-9eea-f25a39779795\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-wv4cn"
Nov 03 20:18:10.847636 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:10.847600    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1e2eadf-ede0-4ea2-9eea-f25a39779795-credential-socket\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-wv4cn\" (UID: \"c1e2eadf-ede0-4ea2-9eea-f25a39779795\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-wv4cn"
Nov 03 20:18:10.847941 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:10.847683    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1e2eadf-ede0-4ea2-9eea-f25a39779795-istiod-ca-cert\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-wv4cn\" (UID: \"c1e2eadf-ede0-4ea2-9eea-f25a39779795\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-wv4cn"
Nov 03 20:18:10.847941 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:10.847763    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1e2eadf-ede0-4ea2-9eea-f25a39779795-istio-data\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-wv4cn\" (UID: \"c1e2eadf-ede0-4ea2-9eea-f25a39779795\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-wv4cn"
Nov 03 20:18:10.848065 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:10.847954    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1e2eadf-ede0-4ea2-9eea-f25a39779795-workload-socket\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-wv4cn\" (UID: \"c1e2eadf-ede0-4ea2-9eea-f25a39779795\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-wv4cn"
Nov 03 20:18:10.848065 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:10.848008    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1e2eadf-ede0-4ea2-9eea-f25a39779795-workload-certs\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-wv4cn\" (UID: \"c1e2eadf-ede0-4ea2-9eea-f25a39779795\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-wv4cn"
Nov 03 20:18:10.848065 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:10.848058    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1e2eadf-ede0-4ea2-9eea-f25a39779795-credential-socket\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-wv4cn\" (UID: \"c1e2eadf-ede0-4ea2-9eea-f25a39779795\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-wv4cn"
Nov 03 20:18:10.848193 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:10.848165    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1e2eadf-ede0-4ea2-9eea-f25a39779795-istio-data\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-wv4cn\" (UID: \"c1e2eadf-ede0-4ea2-9eea-f25a39779795\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-wv4cn"
Nov 03 20:18:10.848758 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:10.848734    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1e2eadf-ede0-4ea2-9eea-f25a39779795-istiod-ca-cert\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-wv4cn\" (UID: \"c1e2eadf-ede0-4ea2-9eea-f25a39779795\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-wv4cn"
Nov 03 20:18:10.849904 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:10.849885    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1e2eadf-ede0-4ea2-9eea-f25a39779795-istio-envoy\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-wv4cn\" (UID: \"c1e2eadf-ede0-4ea2-9eea-f25a39779795\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-wv4cn"
Nov 03 20:18:10.850229 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:10.850207    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1e2eadf-ede0-4ea2-9eea-f25a39779795-istio-podinfo\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-wv4cn\" (UID: \"c1e2eadf-ede0-4ea2-9eea-f25a39779795\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-wv4cn"
Nov 03 20:18:10.867635 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:10.867607    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6kv7\" (UniqueName: \"kubernetes.io/projected/c1e2eadf-ede0-4ea2-9eea-f25a39779795-kube-api-access-g6kv7\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-wv4cn\" (UID: \"c1e2eadf-ede0-4ea2-9eea-f25a39779795\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-wv4cn"
Nov 03 20:18:10.867635 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:10.867626    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1e2eadf-ede0-4ea2-9eea-f25a39779795-istio-token\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-wv4cn\" (UID: \"c1e2eadf-ede0-4ea2-9eea-f25a39779795\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-wv4cn"
Nov 03 20:18:10.963191 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:10.963152    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-wv4cn"
Nov 03 20:18:11.325832 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:11.323758    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-wv4cn"]
Nov 03 20:18:11.339271 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:18:11.336406    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1e2eadf_ede0_4ea2_9eea_f25a39779795.slice/crio-a39a32213538df3773348b493f42a48dd7b4cba9e251838b18ed3c4dce7c42bb WatchSource:0}: Error finding container a39a32213538df3773348b493f42a48dd7b4cba9e251838b18ed3c4dce7c42bb: Status 404 returned error can't find the container with id a39a32213538df3773348b493f42a48dd7b4cba9e251838b18ed3c4dce7c42bb
Nov 03 20:18:11.370862 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:11.370761    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:18:11.371107 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:11.370890    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:18:11.371107 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:11.370971    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:18:11.371744 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:11.371722    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-fqjnh"]
Nov 03 20:18:11.382401 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:11.382371    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-fqjnh"]
Nov 03 20:18:11.382536 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:11.382496    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-fqjnh"
Nov 03 20:18:11.471135 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:11.471098    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51c1be09-aa2e-47bd-a863-97f7daf94ddb-istio-podinfo\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-fqjnh\" (UID: \"51c1be09-aa2e-47bd-a863-97f7daf94ddb\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-fqjnh"
Nov 03 20:18:11.471318 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:11.471170    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51c1be09-aa2e-47bd-a863-97f7daf94ddb-istio-envoy\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-fqjnh\" (UID: \"51c1be09-aa2e-47bd-a863-97f7daf94ddb\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-fqjnh"
Nov 03 20:18:11.471396 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:11.471318    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51c1be09-aa2e-47bd-a863-97f7daf94ddb-istiod-ca-cert\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-fqjnh\" (UID: \"51c1be09-aa2e-47bd-a863-97f7daf94ddb\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-fqjnh"
Nov 03 20:18:11.471457 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:11.471394    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51c1be09-aa2e-47bd-a863-97f7daf94ddb-istio-data\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-fqjnh\" (UID: \"51c1be09-aa2e-47bd-a863-97f7daf94ddb\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-fqjnh"
Nov 03 20:18:11.471457 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:11.471418    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51c1be09-aa2e-47bd-a863-97f7daf94ddb-workload-certs\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-fqjnh\" (UID: \"51c1be09-aa2e-47bd-a863-97f7daf94ddb\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-fqjnh"
Nov 03 20:18:11.471558 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:11.471461    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51c1be09-aa2e-47bd-a863-97f7daf94ddb-istio-token\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-fqjnh\" (UID: \"51c1be09-aa2e-47bd-a863-97f7daf94ddb\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-fqjnh"
Nov 03 20:18:11.471558 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:11.471533    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51c1be09-aa2e-47bd-a863-97f7daf94ddb-credential-socket\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-fqjnh\" (UID: \"51c1be09-aa2e-47bd-a863-97f7daf94ddb\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-fqjnh"
Nov 03 20:18:11.471690 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:11.471657    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sc86d\" (UniqueName: \"kubernetes.io/projected/51c1be09-aa2e-47bd-a863-97f7daf94ddb-kube-api-access-sc86d\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-fqjnh\" (UID: \"51c1be09-aa2e-47bd-a863-97f7daf94ddb\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-fqjnh"
Nov 03 20:18:11.471848 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:11.471823    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51c1be09-aa2e-47bd-a863-97f7daf94ddb-workload-socket\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-fqjnh\" (UID: \"51c1be09-aa2e-47bd-a863-97f7daf94ddb\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-fqjnh"
Nov 03 20:18:11.572946 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:11.572855    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51c1be09-aa2e-47bd-a863-97f7daf94ddb-istio-podinfo\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-fqjnh\" (UID: \"51c1be09-aa2e-47bd-a863-97f7daf94ddb\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-fqjnh"
Nov 03 20:18:11.573163 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:11.572952    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51c1be09-aa2e-47bd-a863-97f7daf94ddb-istio-envoy\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-fqjnh\" (UID: \"51c1be09-aa2e-47bd-a863-97f7daf94ddb\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-fqjnh"
Nov 03 20:18:11.573163 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:11.572998    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51c1be09-aa2e-47bd-a863-97f7daf94ddb-istiod-ca-cert\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-fqjnh\" (UID: \"51c1be09-aa2e-47bd-a863-97f7daf94ddb\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-fqjnh"
Nov 03 20:18:11.573163 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:11.573068    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51c1be09-aa2e-47bd-a863-97f7daf94ddb-istio-data\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-fqjnh\" (UID: \"51c1be09-aa2e-47bd-a863-97f7daf94ddb\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-fqjnh"
Nov 03 20:18:11.573163 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:11.573100    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51c1be09-aa2e-47bd-a863-97f7daf94ddb-workload-certs\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-fqjnh\" (UID: \"51c1be09-aa2e-47bd-a863-97f7daf94ddb\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-fqjnh"
Nov 03 20:18:11.573163 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:11.573152    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51c1be09-aa2e-47bd-a863-97f7daf94ddb-istio-token\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-fqjnh\" (UID: \"51c1be09-aa2e-47bd-a863-97f7daf94ddb\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-fqjnh"
Nov 03 20:18:11.573461 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:11.573182    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51c1be09-aa2e-47bd-a863-97f7daf94ddb-credential-socket\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-fqjnh\" (UID: \"51c1be09-aa2e-47bd-a863-97f7daf94ddb\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-fqjnh"
Nov 03 20:18:11.573461 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:11.573255    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sc86d\" (UniqueName: \"kubernetes.io/projected/51c1be09-aa2e-47bd-a863-97f7daf94ddb-kube-api-access-sc86d\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-fqjnh\" (UID: \"51c1be09-aa2e-47bd-a863-97f7daf94ddb\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-fqjnh"
Nov 03 20:18:11.573461 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:11.573387    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51c1be09-aa2e-47bd-a863-97f7daf94ddb-workload-socket\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-fqjnh\" (UID: \"51c1be09-aa2e-47bd-a863-97f7daf94ddb\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-fqjnh"
Nov 03 20:18:11.573703 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:11.573676    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51c1be09-aa2e-47bd-a863-97f7daf94ddb-workload-certs\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-fqjnh\" (UID: \"51c1be09-aa2e-47bd-a863-97f7daf94ddb\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-fqjnh"
Nov 03 20:18:11.573769 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:11.573712    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51c1be09-aa2e-47bd-a863-97f7daf94ddb-istio-data\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-fqjnh\" (UID: \"51c1be09-aa2e-47bd-a863-97f7daf94ddb\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-fqjnh"
Nov 03 20:18:11.574011 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:11.573984    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51c1be09-aa2e-47bd-a863-97f7daf94ddb-credential-socket\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-fqjnh\" (UID: \"51c1be09-aa2e-47bd-a863-97f7daf94ddb\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-fqjnh"
Nov 03 20:18:11.574140 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:11.574028    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51c1be09-aa2e-47bd-a863-97f7daf94ddb-workload-socket\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-fqjnh\" (UID: \"51c1be09-aa2e-47bd-a863-97f7daf94ddb\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-fqjnh"
Nov 03 20:18:11.574201 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:11.574175    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51c1be09-aa2e-47bd-a863-97f7daf94ddb-istiod-ca-cert\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-fqjnh\" (UID: \"51c1be09-aa2e-47bd-a863-97f7daf94ddb\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-fqjnh"
Nov 03 20:18:11.576251 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:11.576226    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51c1be09-aa2e-47bd-a863-97f7daf94ddb-istio-podinfo\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-fqjnh\" (UID: \"51c1be09-aa2e-47bd-a863-97f7daf94ddb\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-fqjnh"
Nov 03 20:18:11.577864 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:11.577845    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51c1be09-aa2e-47bd-a863-97f7daf94ddb-istio-envoy\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-fqjnh\" (UID: \"51c1be09-aa2e-47bd-a863-97f7daf94ddb\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-fqjnh"
Nov 03 20:18:11.593964 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:11.593937    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51c1be09-aa2e-47bd-a863-97f7daf94ddb-istio-token\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-fqjnh\" (UID: \"51c1be09-aa2e-47bd-a863-97f7daf94ddb\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-fqjnh"
Nov 03 20:18:11.594116 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:11.593997    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sc86d\" (UniqueName: \"kubernetes.io/projected/51c1be09-aa2e-47bd-a863-97f7daf94ddb-kube-api-access-sc86d\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-fqjnh\" (UID: \"51c1be09-aa2e-47bd-a863-97f7daf94ddb\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-fqjnh"
Nov 03 20:18:11.704638 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:11.704599    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-fqjnh"
Nov 03 20:18:11.880904 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:11.878568    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2e12a2f-4h7wg"]
Nov 03 20:18:11.889137 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:11.889107    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2e12a2f-4h7wg"]
Nov 03 20:18:11.889380 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:11.889229    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2e12a2f-4h7wg"
Nov 03 20:18:11.905806 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:11.905722    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:18:11.905806 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:11.905725    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l7s5w\""
Nov 03 20:18:11.979885 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:11.979837    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68bp5\" (UniqueName: \"kubernetes.io/projected/2513f556-dddd-489a-b544-08dc7884aec3-kube-api-access-68bp5\") pod \"maas-api-revoke-sub-d2e12a2f-4h7wg\" (UID: \"2513f556-dddd-489a-b544-08dc7884aec3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2e12a2f-4h7wg"
Nov 03 20:18:11.980076 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:11.979992    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2513f556-dddd-489a-b544-08dc7884aec3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2e12a2f-4h7wg\" (UID: \"2513f556-dddd-489a-b544-08dc7884aec3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2e12a2f-4h7wg"
Nov 03 20:18:11.980144 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:11.980085    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2513f556-dddd-489a-b544-08dc7884aec3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2e12a2f-4h7wg\" (UID: \"2513f556-dddd-489a-b544-08dc7884aec3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2e12a2f-4h7wg"
Nov 03 20:18:12.025138 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:12.025102    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-fqjnh"]
Nov 03 20:18:12.026390 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:18:12.026366    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51c1be09_aa2e_47bd_a863_97f7daf94ddb.slice/crio-5fa4559cb10ba37ca774bae9818852783302bb23de4a2c2d4fd364e33441cbff WatchSource:0}: Error finding container 5fa4559cb10ba37ca774bae9818852783302bb23de4a2c2d4fd364e33441cbff: Status 404 returned error can't find the container with id 5fa4559cb10ba37ca774bae9818852783302bb23de4a2c2d4fd364e33441cbff
Nov 03 20:18:12.029074 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:12.029041    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:18:12.029166 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:12.029108    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:18:12.029166 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:12.029141    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:18:12.082912 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:12.082262    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68bp5\" (UniqueName: \"kubernetes.io/projected/2513f556-dddd-489a-b544-08dc7884aec3-kube-api-access-68bp5\") pod \"maas-api-revoke-sub-d2e12a2f-4h7wg\" (UID: \"2513f556-dddd-489a-b544-08dc7884aec3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2e12a2f-4h7wg"
Nov 03 20:18:12.083363 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:12.083253    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2513f556-dddd-489a-b544-08dc7884aec3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2e12a2f-4h7wg\" (UID: \"2513f556-dddd-489a-b544-08dc7884aec3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2e12a2f-4h7wg"
Nov 03 20:18:12.083363 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:12.083324    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2513f556-dddd-489a-b544-08dc7884aec3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2e12a2f-4h7wg\" (UID: \"2513f556-dddd-489a-b544-08dc7884aec3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2e12a2f-4h7wg"
Nov 03 20:18:12.084364 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:12.084341    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2513f556-dddd-489a-b544-08dc7884aec3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2e12a2f-4h7wg\" (UID: \"2513f556-dddd-489a-b544-08dc7884aec3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2e12a2f-4h7wg"
Nov 03 20:18:12.085492 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:12.085465    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-wv4cn" event={"ID":"c1e2eadf-ede0-4ea2-9eea-f25a39779795","Type":"ContainerStarted","Data":"5c6d90f6dff94d54503259d98624356b90f1622e4c025457fbf45267a1efac96"}
Nov 03 20:18:12.085586 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:12.085541    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-wv4cn" event={"ID":"c1e2eadf-ede0-4ea2-9eea-f25a39779795","Type":"ContainerStarted","Data":"a39a32213538df3773348b493f42a48dd7b4cba9e251838b18ed3c4dce7c42bb"}
Nov 03 20:18:12.087495 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:12.087470    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-fqjnh" event={"ID":"51c1be09-aa2e-47bd-a863-97f7daf94ddb","Type":"ContainerStarted","Data":"5fa4559cb10ba37ca774bae9818852783302bb23de4a2c2d4fd364e33441cbff"}
Nov 03 20:18:12.090817 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:12.090796    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2513f556-dddd-489a-b544-08dc7884aec3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2e12a2f-4h7wg\" (UID: \"2513f556-dddd-489a-b544-08dc7884aec3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2e12a2f-4h7wg"
Nov 03 20:18:12.102879 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:12.102854    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68bp5\" (UniqueName: \"kubernetes.io/projected/2513f556-dddd-489a-b544-08dc7884aec3-kube-api-access-68bp5\") pod \"maas-api-revoke-sub-d2e12a2f-4h7wg\" (UID: \"2513f556-dddd-489a-b544-08dc7884aec3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2e12a2f-4h7wg"
Nov 03 20:18:12.129420 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:12.129372    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-wv4cn" podStartSLOduration=2.129357685 podStartE2EDuration="2.129357685s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:18:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:18:12.126919686 +0000 UTC m=+826.279581574" watchObservedRunningTime="2026-11-03 20:18:12.129357685 +0000 UTC m=+826.282019571"
Nov 03 20:18:12.211646 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:12.211613    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2e12a2f-4h7wg"
Nov 03 20:18:12.566797 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:12.566733    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2e12a2f-4h7wg"]
Nov 03 20:18:12.572410 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:18:12.572299    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2513f556_dddd_489a_b544_08dc7884aec3.slice/crio-ee8f66e1cf4334f5b6616bc6eed3284a152e8d6d0528307c65e30e660075b79f WatchSource:0}: Error finding container ee8f66e1cf4334f5b6616bc6eed3284a152e8d6d0528307c65e30e660075b79f: Status 404 returned error can't find the container with id ee8f66e1cf4334f5b6616bc6eed3284a152e8d6d0528307c65e30e660075b79f
Nov 03 20:18:12.970798 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:12.969152    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-wv4cn"
Nov 03 20:18:12.970798 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:12.970588    5570 patch_prober.go:29] interesting pod/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-wv4cn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Nov 03 20:18:12.970798 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:12.970641    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-wv4cn" podUID="c1e2eadf-ede0-4ea2-9eea-f25a39779795" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Nov 03 20:18:13.106547 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:13.105951    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2e12a2f-4h7wg" event={"ID":"2513f556-dddd-489a-b544-08dc7884aec3","Type":"ContainerStarted","Data":"dab00e9a83a21bf12d6f4e1e1b9de560477adf7fcd0778cda63edba11fcccdb2"}
Nov 03 20:18:13.106547 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:13.106015    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2e12a2f-4h7wg" event={"ID":"2513f556-dddd-489a-b544-08dc7884aec3","Type":"ContainerStarted","Data":"ee8f66e1cf4334f5b6616bc6eed3284a152e8d6d0528307c65e30e660075b79f"}
Nov 03 20:18:13.109827 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:13.109741    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-fqjnh" event={"ID":"51c1be09-aa2e-47bd-a863-97f7daf94ddb","Type":"ContainerStarted","Data":"8c8ff7ee0e824dae78ba0ae02863133dd5b8d0ddb46392cb8f4984679b96ed86"}
Nov 03 20:18:13.193156 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:13.188714    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2e12a2f-4h7wg" podStartSLOduration=2.18869188 podStartE2EDuration="2.18869188s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:18:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:18:13.152709556 +0000 UTC m=+827.305371444" watchObservedRunningTime="2026-11-03 20:18:13.18869188 +0000 UTC m=+827.341353773"
Nov 03 20:18:13.212143 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:13.210728    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-fqjnh" podStartSLOduration=2.210708693 podStartE2EDuration="2.210708693s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:18:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:18:13.206832101 +0000 UTC m=+827.359493989" watchObservedRunningTime="2026-11-03 20:18:13.210708693 +0000 UTC m=+827.363370580"
Nov 03 20:18:13.705295 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:13.705252    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-fqjnh"
Nov 03 20:18:13.706567 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:13.706533    5570 patch_prober.go:29] interesting pod/e2e-worker-w3-482722-openshift-default-7645487df-fqjnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Nov 03 20:18:13.706715 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:13.706587    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-fqjnh" podUID="51c1be09-aa2e-47bd-a863-97f7daf94ddb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Nov 03 20:18:13.808832 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:13.807155    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5dd10855-b4dqz"]
Nov 03 20:18:13.813616 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:13.812847    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dd10855-b4dqz"
Nov 03 20:18:13.826828 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:13.826789    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5dd10855-b4dqz"]
Nov 03 20:18:13.835810 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:13.835760    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3137dfc5-1780-4326-a6d5-469e1e780699-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5dd10855-b4dqz\" (UID: \"3137dfc5-1780-4326-a6d5-469e1e780699\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dd10855-b4dqz"
Nov 03 20:18:13.835984 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:13.835956    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bgsg\" (UniqueName: \"kubernetes.io/projected/3137dfc5-1780-4326-a6d5-469e1e780699-kube-api-access-9bgsg\") pod \"maas-api-revoke-sub-5dd10855-b4dqz\" (UID: \"3137dfc5-1780-4326-a6d5-469e1e780699\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dd10855-b4dqz"
Nov 03 20:18:13.836100 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:13.836079    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3137dfc5-1780-4326-a6d5-469e1e780699-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5dd10855-b4dqz\" (UID: \"3137dfc5-1780-4326-a6d5-469e1e780699\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dd10855-b4dqz"
Nov 03 20:18:13.984859 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:13.969990    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3137dfc5-1780-4326-a6d5-469e1e780699-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5dd10855-b4dqz\" (UID: \"3137dfc5-1780-4326-a6d5-469e1e780699\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dd10855-b4dqz"
Nov 03 20:18:13.984859 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:13.970136    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bgsg\" (UniqueName: \"kubernetes.io/projected/3137dfc5-1780-4326-a6d5-469e1e780699-kube-api-access-9bgsg\") pod \"maas-api-revoke-sub-5dd10855-b4dqz\" (UID: \"3137dfc5-1780-4326-a6d5-469e1e780699\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dd10855-b4dqz"
Nov 03 20:18:13.984859 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:13.970242    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3137dfc5-1780-4326-a6d5-469e1e780699-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5dd10855-b4dqz\" (UID: \"3137dfc5-1780-4326-a6d5-469e1e780699\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dd10855-b4dqz"
Nov 03 20:18:13.984859 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:13.980369    5570 patch_prober.go:29] interesting pod/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-wv4cn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Nov 03 20:18:13.984859 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:13.980426    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-wv4cn" podUID="c1e2eadf-ede0-4ea2-9eea-f25a39779795" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Nov 03 20:18:13.988271 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:13.988238    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3137dfc5-1780-4326-a6d5-469e1e780699-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5dd10855-b4dqz\" (UID: \"3137dfc5-1780-4326-a6d5-469e1e780699\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dd10855-b4dqz"
Nov 03 20:18:13.992550 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:13.992517    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3137dfc5-1780-4326-a6d5-469e1e780699-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5dd10855-b4dqz\" (UID: \"3137dfc5-1780-4326-a6d5-469e1e780699\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dd10855-b4dqz"
Nov 03 20:18:14.012168 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:14.012131    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bgsg\" (UniqueName: \"kubernetes.io/projected/3137dfc5-1780-4326-a6d5-469e1e780699-kube-api-access-9bgsg\") pod \"maas-api-revoke-sub-5dd10855-b4dqz\" (UID: \"3137dfc5-1780-4326-a6d5-469e1e780699\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dd10855-b4dqz"
Nov 03 20:18:14.124339 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:14.124301    5570 generic.go:356] "Generic (PLEG): container finished" podID="2513f556-dddd-489a-b544-08dc7884aec3" containerID="dab00e9a83a21bf12d6f4e1e1b9de560477adf7fcd0778cda63edba11fcccdb2" exitCode=0
Nov 03 20:18:14.124726 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:14.124700    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2e12a2f-4h7wg" event={"ID":"2513f556-dddd-489a-b544-08dc7884aec3","Type":"ContainerDied","Data":"dab00e9a83a21bf12d6f4e1e1b9de560477adf7fcd0778cda63edba11fcccdb2"}
Nov 03 20:18:14.208269 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:14.208229    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dd10855-b4dqz"
Nov 03 20:18:14.587863 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:14.587425    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5dd10855-b4dqz"]
Nov 03 20:18:14.593332 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:18:14.593283    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3137dfc5_1780_4326_a6d5_469e1e780699.slice/crio-aee536a66cc55fb561d6404537884c78e29a8936b1e696927ba5fe0025f6988d WatchSource:0}: Error finding container aee536a66cc55fb561d6404537884c78e29a8936b1e696927ba5fe0025f6988d: Status 404 returned error can't find the container with id aee536a66cc55fb561d6404537884c78e29a8936b1e696927ba5fe0025f6988d
Nov 03 20:18:14.705385 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:14.705352    5570 patch_prober.go:29] interesting pod/e2e-worker-w3-482722-openshift-default-7645487df-fqjnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Nov 03 20:18:14.705570 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:14.705402    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-fqjnh" podUID="51c1be09-aa2e-47bd-a863-97f7daf94ddb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Nov 03 20:18:14.965851 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:14.965213    5570 patch_prober.go:29] interesting pod/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-wv4cn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Nov 03 20:18:14.965851 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:14.965277    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-wv4cn" podUID="c1e2eadf-ede0-4ea2-9eea-f25a39779795" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Nov 03 20:18:15.133596 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:15.133558    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dd10855-b4dqz" event={"ID":"3137dfc5-1780-4326-a6d5-469e1e780699","Type":"ContainerStarted","Data":"f655aee66fa3761cc6fa99517c64c01f20c20d8a62136f29963855f1572557f5"}
Nov 03 20:18:15.134177 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:15.134154    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dd10855-b4dqz" event={"ID":"3137dfc5-1780-4326-a6d5-469e1e780699","Type":"ContainerStarted","Data":"aee536a66cc55fb561d6404537884c78e29a8936b1e696927ba5fe0025f6988d"}
Nov 03 20:18:15.631978 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:15.631950    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2e12a2f-4h7wg"
Nov 03 20:18:15.705405 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:15.705370    5570 patch_prober.go:29] interesting pod/e2e-worker-w3-482722-openshift-default-7645487df-fqjnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Nov 03 20:18:15.705597 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:15.705429    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-fqjnh" podUID="51c1be09-aa2e-47bd-a863-97f7daf94ddb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Nov 03 20:18:15.716521 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:15.716456    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dd10855-b4dqz" podStartSLOduration=2.716432522 podStartE2EDuration="2.716432522s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:18:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:18:15.199995619 +0000 UTC m=+829.352657537" watchObservedRunningTime="2026-11-03 20:18:15.716432522 +0000 UTC m=+829.869094511"
Nov 03 20:18:15.798267 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:15.797475    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2513f556-dddd-489a-b544-08dc7884aec3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2513f556-dddd-489a-b544-08dc7884aec3-maas-api-openshift-ca\") pod \"2513f556-dddd-489a-b544-08dc7884aec3\" (UID: \"2513f556-dddd-489a-b544-08dc7884aec3\") "
Nov 03 20:18:15.798267 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:15.797585    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2513f556-dddd-489a-b544-08dc7884aec3-kube-api-access-68bp5\" (UniqueName: \"kubernetes.io/projected/2513f556-dddd-489a-b544-08dc7884aec3-kube-api-access-68bp5\") pod \"2513f556-dddd-489a-b544-08dc7884aec3\" (UID: \"2513f556-dddd-489a-b544-08dc7884aec3\") "
Nov 03 20:18:15.798267 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:15.797636    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2513f556-dddd-489a-b544-08dc7884aec3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2513f556-dddd-489a-b544-08dc7884aec3-maas-api-xks-ca\") pod \"2513f556-dddd-489a-b544-08dc7884aec3\" (UID: \"2513f556-dddd-489a-b544-08dc7884aec3\") "
Nov 03 20:18:15.800548 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:15.800511    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2513f556-dddd-489a-b544-08dc7884aec3-maas-api-openshift-ca" pod "2513f556-dddd-489a-b544-08dc7884aec3" (UID: "2513f556-dddd-489a-b544-08dc7884aec3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:18:15.804810 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:15.804458    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2513f556-dddd-489a-b544-08dc7884aec3-maas-api-xks-ca" pod "2513f556-dddd-489a-b544-08dc7884aec3" (UID: "2513f556-dddd-489a-b544-08dc7884aec3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:18:15.807832 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:15.806925    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2513f556-dddd-489a-b544-08dc7884aec3-kube-api-access-68bp5" pod "2513f556-dddd-489a-b544-08dc7884aec3" (UID: "2513f556-dddd-489a-b544-08dc7884aec3"). InnerVolumeSpecName "kube-api-access-68bp5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:18:15.900830 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:15.900795    5570 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2513f556-dddd-489a-b544-08dc7884aec3-maas-api-xks-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:18:15.900830 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:15.900831    5570 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2513f556-dddd-489a-b544-08dc7884aec3-maas-api-openshift-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:18:15.901081 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:15.900842    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68bp5\" (UniqueName: \"kubernetes.io/projected/2513f556-dddd-489a-b544-08dc7884aec3-kube-api-access-68bp5\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:18:15.991280 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:15.989886    5570 patch_prober.go:29] interesting pod/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-wv4cn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Nov 03 20:18:15.991280 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:15.989942    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-wv4cn" podUID="c1e2eadf-ede0-4ea2-9eea-f25a39779795" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Nov 03 20:18:16.191662 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:16.191616    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2e12a2f-4h7wg"
Nov 03 20:18:16.192116 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:16.191677    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2e12a2f-4h7wg" event={"ID":"2513f556-dddd-489a-b544-08dc7884aec3","Type":"ContainerDied","Data":"ee8f66e1cf4334f5b6616bc6eed3284a152e8d6d0528307c65e30e660075b79f"}
Nov 03 20:18:16.192116 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:16.191725    5570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ee8f66e1cf4334f5b6616bc6eed3284a152e8d6d0528307c65e30e660075b79f"
Nov 03 20:18:16.194137 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:16.194106    5570 generic.go:356] "Generic (PLEG): container finished" podID="3137dfc5-1780-4326-a6d5-469e1e780699" containerID="f655aee66fa3761cc6fa99517c64c01f20c20d8a62136f29963855f1572557f5" exitCode=0
Nov 03 20:18:16.194290 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:16.194197    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dd10855-b4dqz" event={"ID":"3137dfc5-1780-4326-a6d5-469e1e780699","Type":"ContainerDied","Data":"f655aee66fa3761cc6fa99517c64c01f20c20d8a62136f29963855f1572557f5"}
Nov 03 20:18:16.705474 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:16.705441    5570 patch_prober.go:29] interesting pod/e2e-worker-w3-482722-openshift-default-7645487df-fqjnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Nov 03 20:18:16.705670 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:16.705508    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-fqjnh" podUID="51c1be09-aa2e-47bd-a863-97f7daf94ddb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Nov 03 20:18:16.968302 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:16.968200    5570 patch_prober.go:29] interesting pod/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-wv4cn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Nov 03 20:18:16.968592 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:16.968561    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-wv4cn" podUID="c1e2eadf-ede0-4ea2-9eea-f25a39779795" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Nov 03 20:18:17.436378 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:17.436349    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dd10855-b4dqz"
Nov 03 20:18:17.495046 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:17.494941    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3137dfc5-1780-4326-a6d5-469e1e780699-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3137dfc5-1780-4326-a6d5-469e1e780699-maas-api-xks-ca\") pod \"3137dfc5-1780-4326-a6d5-469e1e780699\" (UID: \"3137dfc5-1780-4326-a6d5-469e1e780699\") "
Nov 03 20:18:17.495304 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:17.495173    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3137dfc5-1780-4326-a6d5-469e1e780699-kube-api-access-9bgsg\" (UniqueName: \"kubernetes.io/projected/3137dfc5-1780-4326-a6d5-469e1e780699-kube-api-access-9bgsg\") pod \"3137dfc5-1780-4326-a6d5-469e1e780699\" (UID: \"3137dfc5-1780-4326-a6d5-469e1e780699\") "
Nov 03 20:18:17.495399 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:17.495310    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3137dfc5-1780-4326-a6d5-469e1e780699-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3137dfc5-1780-4326-a6d5-469e1e780699-maas-api-openshift-ca\") pod \"3137dfc5-1780-4326-a6d5-469e1e780699\" (UID: \"3137dfc5-1780-4326-a6d5-469e1e780699\") "
Nov 03 20:18:17.496545 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:17.496234    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3137dfc5-1780-4326-a6d5-469e1e780699-maas-api-openshift-ca" pod "3137dfc5-1780-4326-a6d5-469e1e780699" (UID: "3137dfc5-1780-4326-a6d5-469e1e780699"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:18:17.499613 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:17.499579    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3137dfc5-1780-4326-a6d5-469e1e780699-maas-api-xks-ca" pod "3137dfc5-1780-4326-a6d5-469e1e780699" (UID: "3137dfc5-1780-4326-a6d5-469e1e780699"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:18:17.502006 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:17.499753    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3137dfc5-1780-4326-a6d5-469e1e780699-kube-api-access-9bgsg" pod "3137dfc5-1780-4326-a6d5-469e1e780699" (UID: "3137dfc5-1780-4326-a6d5-469e1e780699"). InnerVolumeSpecName "kube-api-access-9bgsg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:18:17.597248 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:17.597144    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9bgsg\" (UniqueName: \"kubernetes.io/projected/3137dfc5-1780-4326-a6d5-469e1e780699-kube-api-access-9bgsg\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:18:17.597248 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:17.597181    5570 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3137dfc5-1780-4326-a6d5-469e1e780699-maas-api-openshift-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:18:17.597248 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:17.597191    5570 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3137dfc5-1780-4326-a6d5-469e1e780699-maas-api-xks-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:18:17.705413 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:17.705373    5570 patch_prober.go:29] interesting pod/e2e-worker-w3-482722-openshift-default-7645487df-fqjnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Nov 03 20:18:17.705590 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:17.705434    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-fqjnh" podUID="51c1be09-aa2e-47bd-a863-97f7daf94ddb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Nov 03 20:18:17.966052 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:17.964497    5570 patch_prober.go:29] interesting pod/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-wv4cn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Nov 03 20:18:17.966052 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:17.964557    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-wv4cn" podUID="c1e2eadf-ede0-4ea2-9eea-f25a39779795" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Nov 03 20:18:18.211207 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:18.211174    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dd10855-b4dqz"
Nov 03 20:18:18.211207 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:18.211196    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dd10855-b4dqz" event={"ID":"3137dfc5-1780-4326-a6d5-469e1e780699","Type":"ContainerDied","Data":"aee536a66cc55fb561d6404537884c78e29a8936b1e696927ba5fe0025f6988d"}
Nov 03 20:18:18.211425 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:18.211257    5570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aee536a66cc55fb561d6404537884c78e29a8936b1e696927ba5fe0025f6988d"
Nov 03 20:18:18.705199 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:18.705164    5570 patch_prober.go:29] interesting pod/e2e-worker-w3-482722-openshift-default-7645487df-fqjnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Nov 03 20:18:18.705710 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:18.705232    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-fqjnh" podUID="51c1be09-aa2e-47bd-a863-97f7daf94ddb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Nov 03 20:18:18.969159 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:18.969075    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-wv4cn"
Nov 03 20:18:18.969524 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:18.969499    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-wv4cn"
Nov 03 20:18:18.970413 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:18.970393    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-wv4cn"
Nov 03 20:18:19.709825 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:19.709788    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-fqjnh"
Nov 03 20:18:19.710221 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:19.710182    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-fqjnh"
Nov 03 20:18:19.710984 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:19.710967    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-fqjnh"
Nov 03 20:18:23.443976 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:23.443925    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/premium-w2-9973d9-kserve-5c6dd4669c-5gv8x" podUID="cefc89ad-7ae1-433b-979d-6601678a6d98" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:18:23.540800 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:23.529390    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz" podUID="3a715ef1-38e9-4386-9eec-e10e8df11a10" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:18:23.540800 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:23.530206    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/premium-w3-482722-kserve-689dc7bd5d-x8rjp" podUID="369045c3-4d6a-49a7-9bc6-f2939883bee8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:18:23.626800 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:23.625925    5570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-e80203/premium-w4-e80203-kserve-5f9f445456-bms9d" podUID="6ad76efc-20c2-4951-b22a-c5a39cd39ed4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:18:23.927861 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:23.926871    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/premium-w4-e80203-kserve-5f9f445456-bms9d" podUID="6ad76efc-20c2-4951-b22a-c5a39cd39ed4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:18:24.055799 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:24.054886    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/simulator-w0-e40816-kserve-688f79796f-2tvw5" podUID="c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:18:24.055799 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:24.055287    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh" podUID="2d3647f2-89e9-4908-a8ee-d84e527abeb0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Nov 03 20:18:24.055799 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:24.055566    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/premium-w1-6fc748-kserve-7474db9cd7-nf2xk" podUID="830e062e-83fc-4de7-91a7-5534dac1d46b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Nov 03 20:18:24.456799 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:24.444856    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/premium-w2-9973d9-kserve-5c6dd4669c-5gv8x" podUID="cefc89ad-7ae1-433b-979d-6601678a6d98" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:18:24.456799 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:24.445605    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9973d9/premium-w2-9973d9-kserve-5c6dd4669c-5gv8x"
Nov 03 20:18:24.536011 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:24.532936    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz" podUID="3a715ef1-38e9-4386-9eec-e10e8df11a10" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:18:24.536011 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:24.533766    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz"
Nov 03 20:18:24.536011 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:24.534927    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/premium-w3-482722-kserve-689dc7bd5d-x8rjp" podUID="369045c3-4d6a-49a7-9bc6-f2939883bee8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Nov 03 20:18:24.536011 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:24.535259    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-482722/premium-w3-482722-kserve-689dc7bd5d-x8rjp"
Nov 03 20:18:24.864854 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:24.840873    5570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-9973d9/premium-w2-9973d9-kserve-5c6dd4669c-5gv8x" podUID="cefc89ad-7ae1-433b-979d-6601678a6d98" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": context deadline exceeded"
Nov 03 20:18:24.931913 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:24.931864    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-527bbe8c-9ktsh"]
Nov 03 20:18:24.957432 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:24.941662    5570 cpu_manager.go:414] "containerMap: removing container" podUID="2513f556-dddd-489a-b544-08dc7884aec3" containerName="revoke-keys"
Nov 03 20:18:24.960801 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:24.957664    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2513f556-dddd-489a-b544-08dc7884aec3" containerName="revoke-keys"
Nov 03 20:18:24.983808 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:24.980034    5570 cpu_manager.go:414] "containerMap: removing container" podUID="3137dfc5-1780-4326-a6d5-469e1e780699" containerName="revoke-keys"
Nov 03 20:18:24.994867 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:24.994827    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3137dfc5-1780-4326-a6d5-469e1e780699" containerName="revoke-keys"
Nov 03 20:18:24.998807 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:24.995331    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="2513f556-dddd-489a-b544-08dc7884aec3" containerName="revoke-keys"
Nov 03 20:18:25.007803 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:25.003837    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="3137dfc5-1780-4326-a6d5-469e1e780699" containerName="revoke-keys"
Nov 03 20:18:25.023803 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:25.012942    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/premium-w4-e80203-kserve-5f9f445456-bms9d" podUID="6ad76efc-20c2-4951-b22a-c5a39cd39ed4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Nov 03 20:18:25.069126 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:25.057831    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-527bbe8c-9ktsh"]
Nov 03 20:18:25.069126 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:25.058006    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-e80203/premium-w4-e80203-kserve-5f9f445456-bms9d"
Nov 03 20:18:25.069126 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:25.058074    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/premium-w1-6fc748-kserve-7474db9cd7-nf2xk" podUID="830e062e-83fc-4de7-91a7-5534dac1d46b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:18:25.069126 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:25.058505    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-527bbe8c-9ktsh"
Nov 03 20:18:25.105835 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:25.083932    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh" podUID="2d3647f2-89e9-4908-a8ee-d84e527abeb0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:18:25.105835 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:25.086984    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/simulator-w0-e40816-kserve-688f79796f-2tvw5" podUID="c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Nov 03 20:18:25.130891 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:25.124585    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6fc748/premium-w1-6fc748-kserve-7474db9cd7-nf2xk"
Nov 03 20:18:25.130891 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:25.124678    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-9e1bcb/premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh"
Nov 03 20:18:25.130891 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:25.124716    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e40816/simulator-w0-e40816-kserve-688f79796f-2tvw5"
Nov 03 20:18:25.329828 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:25.293948    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l7s5w\""
Nov 03 20:18:25.329828 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:25.294740    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:18:25.331903 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:25.330400    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxmks\" (UniqueName: \"kubernetes.io/projected/22541c4e-43ae-4075-ab06-365e275ba881-kube-api-access-kxmks\") pod \"maas-api-revoke-sub-527bbe8c-9ktsh\" (UID: \"22541c4e-43ae-4075-ab06-365e275ba881\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-527bbe8c-9ktsh"
Nov 03 20:18:25.331903 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:25.330564    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22541c4e-43ae-4075-ab06-365e275ba881-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-527bbe8c-9ktsh\" (UID: \"22541c4e-43ae-4075-ab06-365e275ba881\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-527bbe8c-9ktsh"
Nov 03 20:18:25.331903 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:25.331134    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22541c4e-43ae-4075-ab06-365e275ba881-maas-api-xks-ca\") pod \"maas-api-revoke-sub-527bbe8c-9ktsh\" (UID: \"22541c4e-43ae-4075-ab06-365e275ba881\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-527bbe8c-9ktsh"
Nov 03 20:18:25.626853 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:25.507670    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22541c4e-43ae-4075-ab06-365e275ba881-maas-api-xks-ca\") pod \"maas-api-revoke-sub-527bbe8c-9ktsh\" (UID: \"22541c4e-43ae-4075-ab06-365e275ba881\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-527bbe8c-9ktsh"
Nov 03 20:18:25.626853 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:25.507746    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxmks\" (UniqueName: \"kubernetes.io/projected/22541c4e-43ae-4075-ab06-365e275ba881-kube-api-access-kxmks\") pod \"maas-api-revoke-sub-527bbe8c-9ktsh\" (UID: \"22541c4e-43ae-4075-ab06-365e275ba881\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-527bbe8c-9ktsh"
Nov 03 20:18:25.626853 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:25.507844    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22541c4e-43ae-4075-ab06-365e275ba881-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-527bbe8c-9ktsh\" (UID: \"22541c4e-43ae-4075-ab06-365e275ba881\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-527bbe8c-9ktsh"
Nov 03 20:18:25.626853 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:25.508995    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22541c4e-43ae-4075-ab06-365e275ba881-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-527bbe8c-9ktsh\" (UID: \"22541c4e-43ae-4075-ab06-365e275ba881\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-527bbe8c-9ktsh"
Nov 03 20:18:25.626853 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:25.514688    5570 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz" podUID="3a715ef1-38e9-4386-9eec-e10e8df11a10" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:18:25.626853 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:25.599257    5570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-482722/premium-w3-482722-kserve-689dc7bd5d-x8rjp" podUID="369045c3-4d6a-49a7-9bc6-f2939883bee8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:18:25.667866 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:25.627859    5570 patch_prober.go:29] interesting pod/e2e-worker-w2-9973d9-openshift-default-ffb6475-nz52s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:18:25.667866 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:25.627940    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-nz52s" podUID="974badc1-41ad-461c-8719-e1f3f836ccbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:18:25.667866 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:25.630024    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/premium-w2-9973d9-kserve-5c6dd4669c-5gv8x" podUID="cefc89ad-7ae1-433b-979d-6601678a6d98" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Nov 03 20:18:25.667866 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:25.630391    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz" podUID="3a715ef1-38e9-4386-9eec-e10e8df11a10" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": context deadline exceeded"
Nov 03 20:18:25.667866 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:25.630665    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/premium-w3-482722-kserve-689dc7bd5d-x8rjp" podUID="369045c3-4d6a-49a7-9bc6-f2939883bee8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Nov 03 20:18:26.062449 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:26.061923    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/premium-w1-6fc748-kserve-7474db9cd7-nf2xk" podUID="830e062e-83fc-4de7-91a7-5534dac1d46b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Nov 03 20:18:26.087785 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:26.087730    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh" podUID="2d3647f2-89e9-4908-a8ee-d84e527abeb0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Nov 03 20:18:26.191798 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:26.184702    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/premium-w4-e80203-kserve-5f9f445456-bms9d" podUID="6ad76efc-20c2-4951-b22a-c5a39cd39ed4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Nov 03 20:18:26.191798 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:26.186035    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/simulator-w0-e40816-kserve-688f79796f-2tvw5" podUID="c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:18:26.502108 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:26.500270    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22541c4e-43ae-4075-ab06-365e275ba881-maas-api-xks-ca\") pod \"maas-api-revoke-sub-527bbe8c-9ktsh\" (UID: \"22541c4e-43ae-4075-ab06-365e275ba881\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-527bbe8c-9ktsh"
Nov 03 20:18:26.502108 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:26.501169    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxmks\" (UniqueName: \"kubernetes.io/projected/22541c4e-43ae-4075-ab06-365e275ba881-kube-api-access-kxmks\") pod \"maas-api-revoke-sub-527bbe8c-9ktsh\" (UID: \"22541c4e-43ae-4075-ab06-365e275ba881\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-527bbe8c-9ktsh"
Nov 03 20:18:26.661637 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:26.661585    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/premium-w3-482722-kserve-689dc7bd5d-x8rjp" podUID="369045c3-4d6a-49a7-9bc6-f2939883bee8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:18:26.665841 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:26.665078    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/premium-w2-9973d9-kserve-5c6dd4669c-5gv8x" podUID="cefc89ad-7ae1-433b-979d-6601678a6d98" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Nov 03 20:18:26.682357 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:26.681174    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz" podUID="3a715ef1-38e9-4386-9eec-e10e8df11a10" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": context deadline exceeded"
Nov 03 20:18:26.758981 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:26.758881    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l7s5w\""
Nov 03 20:18:26.827861 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:26.827827    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-527bbe8c-9ktsh"
Nov 03 20:18:26.868058 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:26.866803    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6fc748/premium-w1-6fc748-kserve-7474db9cd7-nf2xk"
Nov 03 20:18:26.868058 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:26.867210    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-e80203/premium-w4-e80203-kserve-5f9f445456-bms9d"
Nov 03 20:18:26.868058 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:26.867255    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz"
Nov 03 20:18:26.873198 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:26.872894    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-482722/premium-w3-482722-kserve-689dc7bd5d-x8rjp"
Nov 03 20:18:26.931449 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:26.923738    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmkq7"]
Nov 03 20:18:26.934898 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:26.934862    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmkq7"]
Nov 03 20:18:26.942318 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:26.942276    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmkq7"
Nov 03 20:18:27.039505 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:27.039393    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/157f80a0-a704-4bed-b544-205c3f2a52b3-credential-socket\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmkq7\" (UID: \"157f80a0-a704-4bed-b544-205c3f2a52b3\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmkq7"
Nov 03 20:18:27.039866 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:27.039834    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/157f80a0-a704-4bed-b544-205c3f2a52b3-istio-podinfo\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmkq7\" (UID: \"157f80a0-a704-4bed-b544-205c3f2a52b3\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmkq7"
Nov 03 20:18:27.040068 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:27.040035    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/157f80a0-a704-4bed-b544-205c3f2a52b3-workload-certs\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmkq7\" (UID: \"157f80a0-a704-4bed-b544-205c3f2a52b3\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmkq7"
Nov 03 20:18:27.041706 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:27.041666    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/157f80a0-a704-4bed-b544-205c3f2a52b3-workload-socket\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmkq7\" (UID: \"157f80a0-a704-4bed-b544-205c3f2a52b3\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmkq7"
Nov 03 20:18:27.041899 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:27.041881    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8npn\" (UniqueName: \"kubernetes.io/projected/157f80a0-a704-4bed-b544-205c3f2a52b3-kube-api-access-z8npn\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmkq7\" (UID: \"157f80a0-a704-4bed-b544-205c3f2a52b3\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmkq7"
Nov 03 20:18:27.042344 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:27.042290    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/157f80a0-a704-4bed-b544-205c3f2a52b3-istio-envoy\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmkq7\" (UID: \"157f80a0-a704-4bed-b544-205c3f2a52b3\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmkq7"
Nov 03 20:18:27.042570 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:27.042536    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/157f80a0-a704-4bed-b544-205c3f2a52b3-istio-data\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmkq7\" (UID: \"157f80a0-a704-4bed-b544-205c3f2a52b3\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmkq7"
Nov 03 20:18:27.042760 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:27.042741    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/157f80a0-a704-4bed-b544-205c3f2a52b3-istio-token\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmkq7\" (UID: \"157f80a0-a704-4bed-b544-205c3f2a52b3\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmkq7"
Nov 03 20:18:27.043272 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:27.043251    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/157f80a0-a704-4bed-b544-205c3f2a52b3-istiod-ca-cert\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmkq7\" (UID: \"157f80a0-a704-4bed-b544-205c3f2a52b3\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmkq7"
Nov 03 20:18:27.057008 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:27.056984    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-9e1bcb/premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh"
Nov 03 20:18:27.101128 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:27.100680    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e40816/simulator-w0-e40816-kserve-688f79796f-2tvw5"
Nov 03 20:18:27.145616 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:27.145576    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/157f80a0-a704-4bed-b544-205c3f2a52b3-istio-envoy\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmkq7\" (UID: \"157f80a0-a704-4bed-b544-205c3f2a52b3\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmkq7"
Nov 03 20:18:27.145861 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:27.145839    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/157f80a0-a704-4bed-b544-205c3f2a52b3-istio-data\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmkq7\" (UID: \"157f80a0-a704-4bed-b544-205c3f2a52b3\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmkq7"
Nov 03 20:18:27.146002 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:27.145987    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/157f80a0-a704-4bed-b544-205c3f2a52b3-istio-token\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmkq7\" (UID: \"157f80a0-a704-4bed-b544-205c3f2a52b3\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmkq7"
Nov 03 20:18:27.146195 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:27.146176    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/157f80a0-a704-4bed-b544-205c3f2a52b3-istiod-ca-cert\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmkq7\" (UID: \"157f80a0-a704-4bed-b544-205c3f2a52b3\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmkq7"
Nov 03 20:18:27.146371 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:27.146355    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/157f80a0-a704-4bed-b544-205c3f2a52b3-credential-socket\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmkq7\" (UID: \"157f80a0-a704-4bed-b544-205c3f2a52b3\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmkq7"
Nov 03 20:18:27.146532 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:27.146517    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/157f80a0-a704-4bed-b544-205c3f2a52b3-istio-podinfo\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmkq7\" (UID: \"157f80a0-a704-4bed-b544-205c3f2a52b3\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmkq7"
Nov 03 20:18:27.146635 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:27.146620    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/157f80a0-a704-4bed-b544-205c3f2a52b3-workload-certs\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmkq7\" (UID: \"157f80a0-a704-4bed-b544-205c3f2a52b3\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmkq7"
Nov 03 20:18:27.146950 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:27.146931    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/157f80a0-a704-4bed-b544-205c3f2a52b3-workload-socket\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmkq7\" (UID: \"157f80a0-a704-4bed-b544-205c3f2a52b3\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmkq7"
Nov 03 20:18:27.147093 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:27.147065    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8npn\" (UniqueName: \"kubernetes.io/projected/157f80a0-a704-4bed-b544-205c3f2a52b3-kube-api-access-z8npn\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmkq7\" (UID: \"157f80a0-a704-4bed-b544-205c3f2a52b3\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmkq7"
Nov 03 20:18:27.147562 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:27.147534    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9973d9/premium-w2-9973d9-kserve-5c6dd4669c-5gv8x"
Nov 03 20:18:27.149794 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:27.148652    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/157f80a0-a704-4bed-b544-205c3f2a52b3-istiod-ca-cert\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmkq7\" (UID: \"157f80a0-a704-4bed-b544-205c3f2a52b3\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmkq7"
Nov 03 20:18:27.149794 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:27.149487    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/157f80a0-a704-4bed-b544-205c3f2a52b3-credential-socket\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmkq7\" (UID: \"157f80a0-a704-4bed-b544-205c3f2a52b3\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmkq7"
Nov 03 20:18:27.152371 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:27.150934    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/157f80a0-a704-4bed-b544-205c3f2a52b3-istio-envoy\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmkq7\" (UID: \"157f80a0-a704-4bed-b544-205c3f2a52b3\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmkq7"
Nov 03 20:18:27.152371 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:27.151196    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/157f80a0-a704-4bed-b544-205c3f2a52b3-istio-data\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmkq7\" (UID: \"157f80a0-a704-4bed-b544-205c3f2a52b3\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmkq7"
Nov 03 20:18:27.152371 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:27.152203    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/157f80a0-a704-4bed-b544-205c3f2a52b3-workload-socket\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmkq7\" (UID: \"157f80a0-a704-4bed-b544-205c3f2a52b3\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmkq7"
Nov 03 20:18:27.152564 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:27.152405    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/157f80a0-a704-4bed-b544-205c3f2a52b3-istio-podinfo\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmkq7\" (UID: \"157f80a0-a704-4bed-b544-205c3f2a52b3\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmkq7"
Nov 03 20:18:27.152564 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:27.152441    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/157f80a0-a704-4bed-b544-205c3f2a52b3-workload-certs\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmkq7\" (UID: \"157f80a0-a704-4bed-b544-205c3f2a52b3\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmkq7"
Nov 03 20:18:27.178559 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:27.178528    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8npn\" (UniqueName: \"kubernetes.io/projected/157f80a0-a704-4bed-b544-205c3f2a52b3-kube-api-access-z8npn\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmkq7\" (UID: \"157f80a0-a704-4bed-b544-205c3f2a52b3\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmkq7"
Nov 03 20:18:27.179246 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:27.179222    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/157f80a0-a704-4bed-b544-205c3f2a52b3-istio-token\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmkq7\" (UID: \"157f80a0-a704-4bed-b544-205c3f2a52b3\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmkq7"
Nov 03 20:18:27.263119 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:18:27.262741    5570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8bwhz" containerName="extractor"
Nov 03 20:18:27.334758 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:18:27.334712    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22541c4e_43ae_4075_ab06_365e275ba881.slice/crio-5f60972aec32b478fd20ac2f265ad296c7784b7fa7c0c37b75da1f5038b61499 WatchSource:0}: Error finding container 5f60972aec32b478fd20ac2f265ad296c7784b7fa7c0c37b75da1f5038b61499: Status 404 returned error can't find the container with id 5f60972aec32b478fd20ac2f265ad296c7784b7fa7c0c37b75da1f5038b61499
Nov 03 20:18:27.335122 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:27.335095    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-527bbe8c-9ktsh"]
Nov 03 20:18:27.455349 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:27.455311    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmkq7"
Nov 03 20:18:27.763465 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:27.763427    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-657vf"]
Nov 03 20:18:27.807062 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:27.807019    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-657vf"]
Nov 03 20:18:27.807388 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:27.807361    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-657vf"
Nov 03 20:18:27.808846 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:27.808317    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f089dd20-d6a6-4c63-88a5-3fbe9a2ff315-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-657vf\" (UID: \"f089dd20-d6a6-4c63-88a5-3fbe9a2ff315\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-657vf"
Nov 03 20:18:27.808846 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:27.808377    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f089dd20-d6a6-4c63-88a5-3fbe9a2ff315-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-657vf\" (UID: \"f089dd20-d6a6-4c63-88a5-3fbe9a2ff315\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-657vf"
Nov 03 20:18:27.808846 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:27.808465    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f089dd20-d6a6-4c63-88a5-3fbe9a2ff315-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-657vf\" (UID: \"f089dd20-d6a6-4c63-88a5-3fbe9a2ff315\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-657vf"
Nov 03 20:18:27.808846 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:27.808494    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f089dd20-d6a6-4c63-88a5-3fbe9a2ff315-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-657vf\" (UID: \"f089dd20-d6a6-4c63-88a5-3fbe9a2ff315\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-657vf"
Nov 03 20:18:27.808846 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:27.808618    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f089dd20-d6a6-4c63-88a5-3fbe9a2ff315-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-657vf\" (UID: \"f089dd20-d6a6-4c63-88a5-3fbe9a2ff315\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-657vf"
Nov 03 20:18:27.808846 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:27.808711    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dghvh\" (UniqueName: \"kubernetes.io/projected/f089dd20-d6a6-4c63-88a5-3fbe9a2ff315-kube-api-access-dghvh\") pod \"maas-default-gateway-openshift-default-694db55c87-657vf\" (UID: \"f089dd20-d6a6-4c63-88a5-3fbe9a2ff315\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-657vf"
Nov 03 20:18:27.808846 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:27.808758    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f089dd20-d6a6-4c63-88a5-3fbe9a2ff315-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-657vf\" (UID: \"f089dd20-d6a6-4c63-88a5-3fbe9a2ff315\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-657vf"
Nov 03 20:18:27.808846 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:27.808800    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f089dd20-d6a6-4c63-88a5-3fbe9a2ff315-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-657vf\" (UID: \"f089dd20-d6a6-4c63-88a5-3fbe9a2ff315\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-657vf"
Nov 03 20:18:27.808846 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:27.808847    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f089dd20-d6a6-4c63-88a5-3fbe9a2ff315-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-657vf\" (UID: \"f089dd20-d6a6-4c63-88a5-3fbe9a2ff315\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-657vf"
Nov 03 20:18:27.815807 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:27.812647    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmkq7"]
Nov 03 20:18:27.820258 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:27.816578    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-vdrfs"]
Nov 03 20:18:27.824445 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:27.824409    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:18:27.825371 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:27.825341    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:18:27.825503 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:27.825404    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:18:27.827101 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:27.827071    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-vdrfs"]
Nov 03 20:18:27.827353 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:27.827332    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-vdrfs"
Nov 03 20:18:27.890564 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:27.890530    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-527bbe8c-9ktsh" event={"ID":"22541c4e-43ae-4075-ab06-365e275ba881","Type":"ContainerStarted","Data":"e118c468a5c7f954ef0bc4918703c3d9a614fd8305322d46265b4ab2c5152692"}
Nov 03 20:18:27.890734 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:27.890589    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-527bbe8c-9ktsh" event={"ID":"22541c4e-43ae-4075-ab06-365e275ba881","Type":"ContainerStarted","Data":"5f60972aec32b478fd20ac2f265ad296c7784b7fa7c0c37b75da1f5038b61499"}
Nov 03 20:18:27.893248 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:27.893215    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmkq7" event={"ID":"157f80a0-a704-4bed-b544-205c3f2a52b3","Type":"ContainerStarted","Data":"8be46425a20c8ac76a082b07be9e9ff588a7cb4c60dcb55d700db5efaca41da9"}
Nov 03 20:18:27.909935 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:27.909902    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f089dd20-d6a6-4c63-88a5-3fbe9a2ff315-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-657vf\" (UID: \"f089dd20-d6a6-4c63-88a5-3fbe9a2ff315\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-657vf"
Nov 03 20:18:27.910138 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:27.909954    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b70a87b-76a6-40e6-9ced-037c3ceee5cf-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-vdrfs\" (UID: \"1b70a87b-76a6-40e6-9ced-037c3ceee5cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-vdrfs"
Nov 03 20:18:27.910138 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:27.910027    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dghvh\" (UniqueName: \"kubernetes.io/projected/f089dd20-d6a6-4c63-88a5-3fbe9a2ff315-kube-api-access-dghvh\") pod \"maas-default-gateway-openshift-default-694db55c87-657vf\" (UID: \"f089dd20-d6a6-4c63-88a5-3fbe9a2ff315\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-657vf"
Nov 03 20:18:27.910138 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:27.910054    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b70a87b-76a6-40e6-9ced-037c3ceee5cf-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-vdrfs\" (UID: \"1b70a87b-76a6-40e6-9ced-037c3ceee5cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-vdrfs"
Nov 03 20:18:27.910289 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:27.910260    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f089dd20-d6a6-4c63-88a5-3fbe9a2ff315-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-657vf\" (UID: \"f089dd20-d6a6-4c63-88a5-3fbe9a2ff315\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-657vf"
Nov 03 20:18:27.910342 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:27.910297    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f089dd20-d6a6-4c63-88a5-3fbe9a2ff315-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-657vf\" (UID: \"f089dd20-d6a6-4c63-88a5-3fbe9a2ff315\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-657vf"
Nov 03 20:18:27.910342 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:27.910327    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b70a87b-76a6-40e6-9ced-037c3ceee5cf-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-vdrfs\" (UID: \"1b70a87b-76a6-40e6-9ced-037c3ceee5cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-vdrfs"
Nov 03 20:18:27.910471 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:27.910441    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b70a87b-76a6-40e6-9ced-037c3ceee5cf-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-vdrfs\" (UID: \"1b70a87b-76a6-40e6-9ced-037c3ceee5cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-vdrfs"
Nov 03 20:18:27.910525 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:27.910489    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjs6p\" (UniqueName: \"kubernetes.io/projected/1b70a87b-76a6-40e6-9ced-037c3ceee5cf-kube-api-access-pjs6p\") pod \"maas-default-gateway-openshift-default-694db55c87-vdrfs\" (UID: \"1b70a87b-76a6-40e6-9ced-037c3ceee5cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-vdrfs"
Nov 03 20:18:27.910608 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:27.910585    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b70a87b-76a6-40e6-9ced-037c3ceee5cf-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-vdrfs\" (UID: \"1b70a87b-76a6-40e6-9ced-037c3ceee5cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-vdrfs"
Nov 03 20:18:27.910665 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:27.910641    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f089dd20-d6a6-4c63-88a5-3fbe9a2ff315-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-657vf\" (UID: \"f089dd20-d6a6-4c63-88a5-3fbe9a2ff315\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-657vf"
Nov 03 20:18:27.910720 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:27.910679    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b70a87b-76a6-40e6-9ced-037c3ceee5cf-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-vdrfs\" (UID: \"1b70a87b-76a6-40e6-9ced-037c3ceee5cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-vdrfs"
Nov 03 20:18:27.910767 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:27.910717    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f089dd20-d6a6-4c63-88a5-3fbe9a2ff315-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-657vf\" (UID: \"f089dd20-d6a6-4c63-88a5-3fbe9a2ff315\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-657vf"
Nov 03 20:18:27.910767 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:27.910732    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f089dd20-d6a6-4c63-88a5-3fbe9a2ff315-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-657vf\" (UID: \"f089dd20-d6a6-4c63-88a5-3fbe9a2ff315\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-657vf"
Nov 03 20:18:27.910900 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:27.910800    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b70a87b-76a6-40e6-9ced-037c3ceee5cf-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-vdrfs\" (UID: \"1b70a87b-76a6-40e6-9ced-037c3ceee5cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-vdrfs"
Nov 03 20:18:27.910900 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:27.910853    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f089dd20-d6a6-4c63-88a5-3fbe9a2ff315-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-657vf\" (UID: \"f089dd20-d6a6-4c63-88a5-3fbe9a2ff315\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-657vf"
Nov 03 20:18:27.911005 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:27.910991    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f089dd20-d6a6-4c63-88a5-3fbe9a2ff315-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-657vf\" (UID: \"f089dd20-d6a6-4c63-88a5-3fbe9a2ff315\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-657vf"
Nov 03 20:18:27.911082 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:27.911064    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f089dd20-d6a6-4c63-88a5-3fbe9a2ff315-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-657vf\" (UID: \"f089dd20-d6a6-4c63-88a5-3fbe9a2ff315\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-657vf"
Nov 03 20:18:27.911143 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:27.911091    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f089dd20-d6a6-4c63-88a5-3fbe9a2ff315-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-657vf\" (UID: \"f089dd20-d6a6-4c63-88a5-3fbe9a2ff315\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-657vf"
Nov 03 20:18:27.911189 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:27.911168    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f089dd20-d6a6-4c63-88a5-3fbe9a2ff315-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-657vf\" (UID: \"f089dd20-d6a6-4c63-88a5-3fbe9a2ff315\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-657vf"
Nov 03 20:18:27.911189 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:27.911178    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b70a87b-76a6-40e6-9ced-037c3ceee5cf-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-vdrfs\" (UID: \"1b70a87b-76a6-40e6-9ced-037c3ceee5cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-vdrfs"
Nov 03 20:18:27.911265 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:27.911246    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f089dd20-d6a6-4c63-88a5-3fbe9a2ff315-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-657vf\" (UID: \"f089dd20-d6a6-4c63-88a5-3fbe9a2ff315\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-657vf"
Nov 03 20:18:27.911389 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:27.911368    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f089dd20-d6a6-4c63-88a5-3fbe9a2ff315-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-657vf\" (UID: \"f089dd20-d6a6-4c63-88a5-3fbe9a2ff315\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-657vf"
Nov 03 20:18:27.913354 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:27.913329    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f089dd20-d6a6-4c63-88a5-3fbe9a2ff315-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-657vf\" (UID: \"f089dd20-d6a6-4c63-88a5-3fbe9a2ff315\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-657vf"
Nov 03 20:18:27.913745 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:27.913724    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f089dd20-d6a6-4c63-88a5-3fbe9a2ff315-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-657vf\" (UID: \"f089dd20-d6a6-4c63-88a5-3fbe9a2ff315\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-657vf"
Nov 03 20:18:27.932831 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:27.932743    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f089dd20-d6a6-4c63-88a5-3fbe9a2ff315-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-657vf\" (UID: \"f089dd20-d6a6-4c63-88a5-3fbe9a2ff315\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-657vf"
Nov 03 20:18:27.932987 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:27.932901    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dghvh\" (UniqueName: \"kubernetes.io/projected/f089dd20-d6a6-4c63-88a5-3fbe9a2ff315-kube-api-access-dghvh\") pod \"maas-default-gateway-openshift-default-694db55c87-657vf\" (UID: \"f089dd20-d6a6-4c63-88a5-3fbe9a2ff315\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-657vf"
Nov 03 20:18:27.942143 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:27.941262    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-527bbe8c-9ktsh" podStartSLOduration=3.941240982 podStartE2EDuration="3.941240982s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:18:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:18:27.93649047 +0000 UTC m=+842.089152359" watchObservedRunningTime="2026-11-03 20:18:27.941240982 +0000 UTC m=+842.093902870"
Nov 03 20:18:28.012017 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:28.011914    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b70a87b-76a6-40e6-9ced-037c3ceee5cf-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-vdrfs\" (UID: \"1b70a87b-76a6-40e6-9ced-037c3ceee5cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-vdrfs"
Nov 03 20:18:28.012250 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:28.012224    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b70a87b-76a6-40e6-9ced-037c3ceee5cf-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-vdrfs\" (UID: \"1b70a87b-76a6-40e6-9ced-037c3ceee5cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-vdrfs"
Nov 03 20:18:28.012571 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:28.012546    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b70a87b-76a6-40e6-9ced-037c3ceee5cf-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-vdrfs\" (UID: \"1b70a87b-76a6-40e6-9ced-037c3ceee5cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-vdrfs"
Nov 03 20:18:28.012835 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:28.012813    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b70a87b-76a6-40e6-9ced-037c3ceee5cf-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-vdrfs\" (UID: \"1b70a87b-76a6-40e6-9ced-037c3ceee5cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-vdrfs"
Nov 03 20:18:28.013286 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:28.013111    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b70a87b-76a6-40e6-9ced-037c3ceee5cf-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-vdrfs\" (UID: \"1b70a87b-76a6-40e6-9ced-037c3ceee5cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-vdrfs"
Nov 03 20:18:28.013286 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:28.013143    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b70a87b-76a6-40e6-9ced-037c3ceee5cf-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-vdrfs\" (UID: \"1b70a87b-76a6-40e6-9ced-037c3ceee5cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-vdrfs"
Nov 03 20:18:28.013286 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:28.013174    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b70a87b-76a6-40e6-9ced-037c3ceee5cf-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-vdrfs\" (UID: \"1b70a87b-76a6-40e6-9ced-037c3ceee5cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-vdrfs"
Nov 03 20:18:28.013286 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:28.013202    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b70a87b-76a6-40e6-9ced-037c3ceee5cf-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-vdrfs\" (UID: \"1b70a87b-76a6-40e6-9ced-037c3ceee5cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-vdrfs"
Nov 03 20:18:28.013286 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:28.013227    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjs6p\" (UniqueName: \"kubernetes.io/projected/1b70a87b-76a6-40e6-9ced-037c3ceee5cf-kube-api-access-pjs6p\") pod \"maas-default-gateway-openshift-default-694db55c87-vdrfs\" (UID: \"1b70a87b-76a6-40e6-9ced-037c3ceee5cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-vdrfs"
Nov 03 20:18:28.013286 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:28.013265    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b70a87b-76a6-40e6-9ced-037c3ceee5cf-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-vdrfs\" (UID: \"1b70a87b-76a6-40e6-9ced-037c3ceee5cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-vdrfs"
Nov 03 20:18:28.013286 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:28.013370    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b70a87b-76a6-40e6-9ced-037c3ceee5cf-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-vdrfs\" (UID: \"1b70a87b-76a6-40e6-9ced-037c3ceee5cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-vdrfs"
Nov 03 20:18:28.014019 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:28.013636    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b70a87b-76a6-40e6-9ced-037c3ceee5cf-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-vdrfs\" (UID: \"1b70a87b-76a6-40e6-9ced-037c3ceee5cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-vdrfs"
Nov 03 20:18:28.014019 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:28.013637    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b70a87b-76a6-40e6-9ced-037c3ceee5cf-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-vdrfs\" (UID: \"1b70a87b-76a6-40e6-9ced-037c3ceee5cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-vdrfs"
Nov 03 20:18:28.014019 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:28.013914    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b70a87b-76a6-40e6-9ced-037c3ceee5cf-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-vdrfs\" (UID: \"1b70a87b-76a6-40e6-9ced-037c3ceee5cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-vdrfs"
Nov 03 20:18:28.015932 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:28.015909    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b70a87b-76a6-40e6-9ced-037c3ceee5cf-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-vdrfs\" (UID: \"1b70a87b-76a6-40e6-9ced-037c3ceee5cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-vdrfs"
Nov 03 20:18:28.016096 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:28.016077    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b70a87b-76a6-40e6-9ced-037c3ceee5cf-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-vdrfs\" (UID: \"1b70a87b-76a6-40e6-9ced-037c3ceee5cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-vdrfs"
Nov 03 20:18:28.033680 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:28.033648    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjs6p\" (UniqueName: \"kubernetes.io/projected/1b70a87b-76a6-40e6-9ced-037c3ceee5cf-kube-api-access-pjs6p\") pod \"maas-default-gateway-openshift-default-694db55c87-vdrfs\" (UID: \"1b70a87b-76a6-40e6-9ced-037c3ceee5cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-vdrfs"
Nov 03 20:18:28.033835 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:28.033661    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b70a87b-76a6-40e6-9ced-037c3ceee5cf-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-vdrfs\" (UID: \"1b70a87b-76a6-40e6-9ced-037c3ceee5cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-vdrfs"
Nov 03 20:18:28.137756 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:28.137701    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-657vf"
Nov 03 20:18:28.152373 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:28.152282    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-vdrfs"
Nov 03 20:18:28.488488 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:28.488461    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-657vf"]
Nov 03 20:18:28.502021 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:28.501990    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-vdrfs"]
Nov 03 20:18:28.503919 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:18:28.503886    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf089dd20_d6a6_4c63_88a5_3fbe9a2ff315.slice/crio-b125be287f3a9650ac805e0eb7ecc6adf820540d2a413cf2c082b0c0d2f3a568 WatchSource:0}: Error finding container b125be287f3a9650ac805e0eb7ecc6adf820540d2a413cf2c082b0c0d2f3a568: Status 404 returned error can't find the container with id b125be287f3a9650ac805e0eb7ecc6adf820540d2a413cf2c082b0c0d2f3a568
Nov 03 20:18:28.504201 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:18:28.504170    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b70a87b_76a6_40e6_9ced_037c3ceee5cf.slice/crio-17b2caebb6f0f8981effc65e9fc788c3d109bad836b965938e22b17da918cf61 WatchSource:0}: Error finding container 17b2caebb6f0f8981effc65e9fc788c3d109bad836b965938e22b17da918cf61: Status 404 returned error can't find the container with id 17b2caebb6f0f8981effc65e9fc788c3d109bad836b965938e22b17da918cf61
Nov 03 20:18:28.506942 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:28.506907    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:18:28.507046 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:28.506988    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:18:28.507046 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:28.506917    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:18:28.507046 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:28.507022    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:18:28.507209 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:28.507075    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:18:28.507209 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:28.507125    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:18:28.905401 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:28.905300    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmkq7" event={"ID":"157f80a0-a704-4bed-b544-205c3f2a52b3","Type":"ContainerStarted","Data":"73be03f9649446acd3f30a87e3b60fbb7889d073564dcf2859c9ed6c410dcaf5"}
Nov 03 20:18:28.907730 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:28.907692    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-657vf" event={"ID":"f089dd20-d6a6-4c63-88a5-3fbe9a2ff315","Type":"ContainerStarted","Data":"e598e97468cbcd07e4b5d768c526d95cf1deafd0e08e7ea71195cf09dd643d38"}
Nov 03 20:18:28.907910 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:28.907745    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-657vf" event={"ID":"f089dd20-d6a6-4c63-88a5-3fbe9a2ff315","Type":"ContainerStarted","Data":"b125be287f3a9650ac805e0eb7ecc6adf820540d2a413cf2c082b0c0d2f3a568"}
Nov 03 20:18:28.910097 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:28.910055    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-vdrfs" event={"ID":"1b70a87b-76a6-40e6-9ced-037c3ceee5cf","Type":"ContainerStarted","Data":"cfc34fea0388713e474e0bfd29b896ec0c51595167214dd09881337a3d7882e6"}
Nov 03 20:18:28.910256 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:28.910131    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-vdrfs" event={"ID":"1b70a87b-76a6-40e6-9ced-037c3ceee5cf","Type":"ContainerStarted","Data":"17b2caebb6f0f8981effc65e9fc788c3d109bad836b965938e22b17da918cf61"}
Nov 03 20:18:28.912134 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:28.912102    5570 generic.go:356] "Generic (PLEG): container finished" podID="22541c4e-43ae-4075-ab06-365e275ba881" containerID="e118c468a5c7f954ef0bc4918703c3d9a614fd8305322d46265b4ab2c5152692" exitCode=0
Nov 03 20:18:28.912329 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:28.912150    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-527bbe8c-9ktsh" event={"ID":"22541c4e-43ae-4075-ab06-365e275ba881","Type":"ContainerDied","Data":"e118c468a5c7f954ef0bc4918703c3d9a614fd8305322d46265b4ab2c5152692"}
Nov 03 20:18:28.964597 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:28.964527    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmkq7" podStartSLOduration=2.964503449 podStartE2EDuration="2.964503449s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:18:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:18:28.950391142 +0000 UTC m=+843.103053031" watchObservedRunningTime="2026-11-03 20:18:28.964503449 +0000 UTC m=+843.117165337"
Nov 03 20:18:28.997902 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:28.997840    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-657vf" podStartSLOduration=1.997826314 podStartE2EDuration="1.997826314s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:18:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:18:28.995125269 +0000 UTC m=+843.147787157" watchObservedRunningTime="2026-11-03 20:18:28.997826314 +0000 UTC m=+843.150488200"
Nov 03 20:18:29.096949 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:29.096798    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-vdrfs" podStartSLOduration=2.096753243 podStartE2EDuration="2.096753243s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:18:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:18:29.095700275 +0000 UTC m=+843.248362188" watchObservedRunningTime="2026-11-03 20:18:29.096753243 +0000 UTC m=+843.249415141"
Nov 03 20:18:29.145799 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:29.143835    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-657vf"
Nov 03 20:18:29.145799 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:29.143980    5570 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-657vf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Nov 03 20:18:29.145799 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:29.144023    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-657vf" podUID="f089dd20-d6a6-4c63-88a5-3fbe9a2ff315" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Nov 03 20:18:29.168944 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:29.167905    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-vdrfs"
Nov 03 20:18:29.168944 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:29.168102    5570 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-vdrfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Nov 03 20:18:29.168944 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:29.168151    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-vdrfs" podUID="1b70a87b-76a6-40e6-9ced-037c3ceee5cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Nov 03 20:18:29.466048 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:29.464978    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmkq7"
Nov 03 20:18:29.488799 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:29.474903    5570 patch_prober.go:29] interesting pod/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmkq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Nov 03 20:18:29.488799 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:29.474976    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmkq7" podUID="157f80a0-a704-4bed-b544-205c3f2a52b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Nov 03 20:18:30.141281 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:30.141233    5570 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-657vf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Nov 03 20:18:30.170383 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:30.170341    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-657vf" podUID="f089dd20-d6a6-4c63-88a5-3fbe9a2ff315" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Nov 03 20:18:30.170630 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:30.170265    5570 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-vdrfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Nov 03 20:18:30.174858 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:30.170791    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-vdrfs" podUID="1b70a87b-76a6-40e6-9ced-037c3ceee5cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Nov 03 20:18:30.424823 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:30.424761    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-527bbe8c-9ktsh"
Nov 03 20:18:30.463795 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:30.463561    5570 patch_prober.go:29] interesting pod/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmkq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Nov 03 20:18:30.463795 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:30.463645    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmkq7" podUID="157f80a0-a704-4bed-b544-205c3f2a52b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Nov 03 20:18:30.622430 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:30.622388    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22541c4e-43ae-4075-ab06-365e275ba881-kube-api-access-kxmks\" (UniqueName: \"kubernetes.io/projected/22541c4e-43ae-4075-ab06-365e275ba881-kube-api-access-kxmks\") pod \"22541c4e-43ae-4075-ab06-365e275ba881\" (UID: \"22541c4e-43ae-4075-ab06-365e275ba881\") "
Nov 03 20:18:30.622907 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:30.622871    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/22541c4e-43ae-4075-ab06-365e275ba881-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22541c4e-43ae-4075-ab06-365e275ba881-maas-api-xks-ca\") pod \"22541c4e-43ae-4075-ab06-365e275ba881\" (UID: \"22541c4e-43ae-4075-ab06-365e275ba881\") "
Nov 03 20:18:30.623228 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:30.623206    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22541c4e-43ae-4075-ab06-365e275ba881-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22541c4e-43ae-4075-ab06-365e275ba881-maas-api-openshift-ca\") pod \"22541c4e-43ae-4075-ab06-365e275ba881\" (UID: \"22541c4e-43ae-4075-ab06-365e275ba881\") "
Nov 03 20:18:30.623519 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:30.623494    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22541c4e-43ae-4075-ab06-365e275ba881-maas-api-openshift-ca" pod "22541c4e-43ae-4075-ab06-365e275ba881" (UID: "22541c4e-43ae-4075-ab06-365e275ba881"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:18:30.624043 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:30.624020    5570 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22541c4e-43ae-4075-ab06-365e275ba881-maas-api-openshift-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:18:30.625701 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:30.625013    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22541c4e-43ae-4075-ab06-365e275ba881-kube-api-access-kxmks" pod "22541c4e-43ae-4075-ab06-365e275ba881" (UID: "22541c4e-43ae-4075-ab06-365e275ba881"). InnerVolumeSpecName "kube-api-access-kxmks". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:18:30.625701 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:30.625090    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22541c4e-43ae-4075-ab06-365e275ba881-maas-api-xks-ca" pod "22541c4e-43ae-4075-ab06-365e275ba881" (UID: "22541c4e-43ae-4075-ab06-365e275ba881"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:18:30.725177 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:30.725083    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kxmks\" (UniqueName: \"kubernetes.io/projected/22541c4e-43ae-4075-ab06-365e275ba881-kube-api-access-kxmks\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:18:30.725177 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:30.725116    5570 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22541c4e-43ae-4075-ab06-365e275ba881-maas-api-xks-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:18:31.053854 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:31.037925    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-527bbe8c-9ktsh"
Nov 03 20:18:31.061037 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:31.052847    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-527bbe8c-9ktsh" event={"ID":"22541c4e-43ae-4075-ab06-365e275ba881","Type":"ContainerDied","Data":"5f60972aec32b478fd20ac2f265ad296c7784b7fa7c0c37b75da1f5038b61499"}
Nov 03 20:18:31.061315 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:31.061294    5570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5f60972aec32b478fd20ac2f265ad296c7784b7fa7c0c37b75da1f5038b61499"
Nov 03 20:18:31.146802 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:31.146176    5570 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-657vf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Nov 03 20:18:31.146802 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:31.146272    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-657vf" podUID="f089dd20-d6a6-4c63-88a5-3fbe9a2ff315" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Nov 03 20:18:31.155206 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:31.155173    5570 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-vdrfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Nov 03 20:18:31.155506 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:31.155466    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-vdrfs" podUID="1b70a87b-76a6-40e6-9ced-037c3ceee5cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Nov 03 20:18:31.470798 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:31.470254    5570 patch_prober.go:29] interesting pod/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmkq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Nov 03 20:18:31.470798 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:31.470334    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmkq7" podUID="157f80a0-a704-4bed-b544-205c3f2a52b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Nov 03 20:18:32.168471 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:32.168430    5570 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-vdrfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Nov 03 20:18:32.169124 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:32.169075    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-vdrfs" podUID="1b70a87b-76a6-40e6-9ced-037c3ceee5cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Nov 03 20:18:32.169883 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:32.169860    5570 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-657vf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Nov 03 20:18:32.170020 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:32.170001    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-657vf" podUID="f089dd20-d6a6-4c63-88a5-3fbe9a2ff315" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Nov 03 20:18:32.459798 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:32.458205    5570 patch_prober.go:29] interesting pod/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmkq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Nov 03 20:18:32.459798 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:32.458276    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmkq7" podUID="157f80a0-a704-4bed-b544-205c3f2a52b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Nov 03 20:18:33.150799 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:33.147332    5570 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-657vf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Nov 03 20:18:33.150799 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:33.147392    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-657vf" podUID="f089dd20-d6a6-4c63-88a5-3fbe9a2ff315" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Nov 03 20:18:33.171797 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:33.160125    5570 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-vdrfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Nov 03 20:18:33.171797 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:33.160187    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-vdrfs" podUID="1b70a87b-76a6-40e6-9ced-037c3ceee5cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Nov 03 20:18:33.490798 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:33.471596    5570 patch_prober.go:29] interesting pod/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmkq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Nov 03 20:18:33.490798 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:33.471664    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmkq7" podUID="157f80a0-a704-4bed-b544-205c3f2a52b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Nov 03 20:18:34.141801 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:34.138873    5570 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-657vf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Nov 03 20:18:34.141801 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:34.138935    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-657vf" podUID="f089dd20-d6a6-4c63-88a5-3fbe9a2ff315" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Nov 03 20:18:34.162813 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:34.161056    5570 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-vdrfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Nov 03 20:18:34.162813 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:34.161127    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-vdrfs" podUID="1b70a87b-76a6-40e6-9ced-037c3ceee5cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Nov 03 20:18:34.460501 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:34.456645    5570 patch_prober.go:29] interesting pod/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmkq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Nov 03 20:18:34.460501 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:34.456768    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmkq7" podUID="157f80a0-a704-4bed-b544-205c3f2a52b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Nov 03 20:18:35.140922 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:35.140878    5570 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-657vf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Nov 03 20:18:35.141319 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:35.140950    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-657vf" podUID="f089dd20-d6a6-4c63-88a5-3fbe9a2ff315" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Nov 03 20:18:35.152978 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:35.152941    5570 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-vdrfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Nov 03 20:18:35.153183 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:35.153005    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-vdrfs" podUID="1b70a87b-76a6-40e6-9ced-037c3ceee5cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Nov 03 20:18:35.462232 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:35.462202    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmkq7"
Nov 03 20:18:35.462657 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:35.462604    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmkq7"
Nov 03 20:18:35.463504 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:35.463484    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmkq7"
Nov 03 20:18:36.144800 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:36.141099    5570 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-657vf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Nov 03 20:18:36.144800 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:36.141173    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-657vf" podUID="f089dd20-d6a6-4c63-88a5-3fbe9a2ff315" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Nov 03 20:18:36.186798 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:36.183820    5570 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-vdrfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Nov 03 20:18:36.186798 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:36.184226    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-vdrfs" podUID="1b70a87b-76a6-40e6-9ced-037c3ceee5cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Nov 03 20:18:37.686967 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:37.686913    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-657vf"
Nov 03 20:18:37.689090 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:37.687623    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-657vf"
Nov 03 20:18:37.689090 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:37.688612    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-657vf"
Nov 03 20:18:37.868808 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:37.857591    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-vdrfs"
Nov 03 20:18:37.868808 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:37.858395    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-vdrfs"
Nov 03 20:18:37.868808 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:37.859892    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-vdrfs"
Nov 03 20:18:39.474923 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:39.474878    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g69f8"]
Nov 03 20:18:41.162798 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:41.161990    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-qskf7"]
Nov 03 20:18:41.163335 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:41.162972    5570 cpu_manager.go:414] "containerMap: removing container" podUID="22541c4e-43ae-4075-ab06-365e275ba881" containerName="revoke-keys"
Nov 03 20:18:41.163335 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:41.162995    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22541c4e-43ae-4075-ab06-365e275ba881" containerName="revoke-keys"
Nov 03 20:18:41.163335 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:41.163152    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="22541c4e-43ae-4075-ab06-365e275ba881" containerName="revoke-keys"
Nov 03 20:18:41.651800 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:41.634875    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/premium-w2-9973d9-kserve-5c6dd4669c-5gv8x" podUID="cefc89ad-7ae1-433b-979d-6601678a6d98" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:18:41.680017 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:41.679980    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-qskf7"]
Nov 03 20:18:41.680371 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:41.680353    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-qskf7"
Nov 03 20:18:41.681911 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:41.681886    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-xt4z5"]
Nov 03 20:18:41.699761 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:41.697249    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/premium-w4-e80203-kserve-5f9f445456-bms9d" podUID="6ad76efc-20c2-4951-b22a-c5a39cd39ed4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:18:41.718805 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:41.706876    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/premium-w3-482722-kserve-689dc7bd5d-x8rjp" podUID="369045c3-4d6a-49a7-9bc6-f2939883bee8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:18:41.733683 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:41.733646    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-xt4z5"]
Nov 03 20:18:41.736143 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:41.736114    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-xt4z5"
Nov 03 20:18:41.780019 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:41.774725    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/469fc281-f7ee-4584-b3b6-85c6a669f806-workload-certs\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-qskf7\" (UID: \"469fc281-f7ee-4584-b3b6-85c6a669f806\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-qskf7"
Nov 03 20:18:41.780019 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:41.774922    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/469fc281-f7ee-4584-b3b6-85c6a669f806-istio-podinfo\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-qskf7\" (UID: \"469fc281-f7ee-4584-b3b6-85c6a669f806\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-qskf7"
Nov 03 20:18:41.780019 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:41.774993    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/469fc281-f7ee-4584-b3b6-85c6a669f806-istio-envoy\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-qskf7\" (UID: \"469fc281-f7ee-4584-b3b6-85c6a669f806\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-qskf7"
Nov 03 20:18:41.780019 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:41.775025    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/469fc281-f7ee-4584-b3b6-85c6a669f806-istio-data\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-qskf7\" (UID: \"469fc281-f7ee-4584-b3b6-85c6a669f806\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-qskf7"
Nov 03 20:18:41.780019 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:41.775072    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pwb2\" (UniqueName: \"kubernetes.io/projected/469fc281-f7ee-4584-b3b6-85c6a669f806-kube-api-access-6pwb2\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-qskf7\" (UID: \"469fc281-f7ee-4584-b3b6-85c6a669f806\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-qskf7"
Nov 03 20:18:41.780019 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:41.775127    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/469fc281-f7ee-4584-b3b6-85c6a669f806-istiod-ca-cert\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-qskf7\" (UID: \"469fc281-f7ee-4584-b3b6-85c6a669f806\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-qskf7"
Nov 03 20:18:41.780019 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:41.775187    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/469fc281-f7ee-4584-b3b6-85c6a669f806-workload-socket\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-qskf7\" (UID: \"469fc281-f7ee-4584-b3b6-85c6a669f806\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-qskf7"
Nov 03 20:18:41.780019 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:41.775219    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/469fc281-f7ee-4584-b3b6-85c6a669f806-istio-token\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-qskf7\" (UID: \"469fc281-f7ee-4584-b3b6-85c6a669f806\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-qskf7"
Nov 03 20:18:41.780019 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:41.775305    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/469fc281-f7ee-4584-b3b6-85c6a669f806-credential-socket\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-qskf7\" (UID: \"469fc281-f7ee-4584-b3b6-85c6a669f806\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-qskf7"
Nov 03 20:18:41.838992 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:41.824395    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz" podUID="3a715ef1-38e9-4386-9eec-e10e8df11a10" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:18:41.885843 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:41.884060    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e026bfa1-f899-46e8-9309-be16eea226bb-istio-envoy\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-xt4z5\" (UID: \"e026bfa1-f899-46e8-9309-be16eea226bb\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-xt4z5"
Nov 03 20:18:41.885843 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:41.884119    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/469fc281-f7ee-4584-b3b6-85c6a669f806-workload-socket\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-qskf7\" (UID: \"469fc281-f7ee-4584-b3b6-85c6a669f806\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-qskf7"
Nov 03 20:18:41.885843 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:41.884149    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e026bfa1-f899-46e8-9309-be16eea226bb-workload-socket\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-xt4z5\" (UID: \"e026bfa1-f899-46e8-9309-be16eea226bb\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-xt4z5"
Nov 03 20:18:41.885843 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:41.884184    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/469fc281-f7ee-4584-b3b6-85c6a669f806-istio-token\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-qskf7\" (UID: \"469fc281-f7ee-4584-b3b6-85c6a669f806\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-qskf7"
Nov 03 20:18:41.885843 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:41.884223    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e026bfa1-f899-46e8-9309-be16eea226bb-istio-podinfo\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-xt4z5\" (UID: \"e026bfa1-f899-46e8-9309-be16eea226bb\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-xt4z5"
Nov 03 20:18:41.885843 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:41.884291    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/469fc281-f7ee-4584-b3b6-85c6a669f806-credential-socket\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-qskf7\" (UID: \"469fc281-f7ee-4584-b3b6-85c6a669f806\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-qskf7"
Nov 03 20:18:41.885843 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:41.884399    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/469fc281-f7ee-4584-b3b6-85c6a669f806-workload-certs\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-qskf7\" (UID: \"469fc281-f7ee-4584-b3b6-85c6a669f806\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-qskf7"
Nov 03 20:18:41.885843 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:41.884436    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e026bfa1-f899-46e8-9309-be16eea226bb-istio-data\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-xt4z5\" (UID: \"e026bfa1-f899-46e8-9309-be16eea226bb\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-xt4z5"
Nov 03 20:18:41.885843 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:41.884514    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e026bfa1-f899-46e8-9309-be16eea226bb-workload-certs\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-xt4z5\" (UID: \"e026bfa1-f899-46e8-9309-be16eea226bb\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-xt4z5"
Nov 03 20:18:41.885843 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:41.884581    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5zsf\" (UniqueName: \"kubernetes.io/projected/e026bfa1-f899-46e8-9309-be16eea226bb-kube-api-access-m5zsf\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-xt4z5\" (UID: \"e026bfa1-f899-46e8-9309-be16eea226bb\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-xt4z5"
Nov 03 20:18:41.885843 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:41.884648    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/469fc281-f7ee-4584-b3b6-85c6a669f806-istio-podinfo\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-qskf7\" (UID: \"469fc281-f7ee-4584-b3b6-85c6a669f806\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-qskf7"
Nov 03 20:18:41.885843 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:41.884676    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e026bfa1-f899-46e8-9309-be16eea226bb-istiod-ca-cert\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-xt4z5\" (UID: \"e026bfa1-f899-46e8-9309-be16eea226bb\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-xt4z5"
Nov 03 20:18:41.885843 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:41.884747    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/469fc281-f7ee-4584-b3b6-85c6a669f806-istio-envoy\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-qskf7\" (UID: \"469fc281-f7ee-4584-b3b6-85c6a669f806\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-qskf7"
Nov 03 20:18:41.885843 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:41.884799    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/469fc281-f7ee-4584-b3b6-85c6a669f806-istio-data\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-qskf7\" (UID: \"469fc281-f7ee-4584-b3b6-85c6a669f806\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-qskf7"
Nov 03 20:18:41.885843 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:41.884826    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e026bfa1-f899-46e8-9309-be16eea226bb-credential-socket\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-xt4z5\" (UID: \"e026bfa1-f899-46e8-9309-be16eea226bb\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-xt4z5"
Nov 03 20:18:41.885843 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:41.884876    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pwb2\" (UniqueName: \"kubernetes.io/projected/469fc281-f7ee-4584-b3b6-85c6a669f806-kube-api-access-6pwb2\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-qskf7\" (UID: \"469fc281-f7ee-4584-b3b6-85c6a669f806\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-qskf7"
Nov 03 20:18:41.885843 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:41.884923    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e026bfa1-f899-46e8-9309-be16eea226bb-istio-token\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-xt4z5\" (UID: \"e026bfa1-f899-46e8-9309-be16eea226bb\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-xt4z5"
Nov 03 20:18:41.885843 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:41.884964    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/469fc281-f7ee-4584-b3b6-85c6a669f806-istiod-ca-cert\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-qskf7\" (UID: \"469fc281-f7ee-4584-b3b6-85c6a669f806\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-qskf7"
Nov 03 20:18:41.963121 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:41.934689    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/469fc281-f7ee-4584-b3b6-85c6a669f806-istiod-ca-cert\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-qskf7\" (UID: \"469fc281-f7ee-4584-b3b6-85c6a669f806\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-qskf7"
Nov 03 20:18:41.963121 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:41.945669    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/469fc281-f7ee-4584-b3b6-85c6a669f806-workload-socket\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-qskf7\" (UID: \"469fc281-f7ee-4584-b3b6-85c6a669f806\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-qskf7"
Nov 03 20:18:41.963121 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:41.953662    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/469fc281-f7ee-4584-b3b6-85c6a669f806-credential-socket\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-qskf7\" (UID: \"469fc281-f7ee-4584-b3b6-85c6a669f806\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-qskf7"
Nov 03 20:18:41.963121 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:41.954072    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/469fc281-f7ee-4584-b3b6-85c6a669f806-workload-certs\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-qskf7\" (UID: \"469fc281-f7ee-4584-b3b6-85c6a669f806\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-qskf7"
Nov 03 20:18:42.000290 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:41.982718    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/469fc281-f7ee-4584-b3b6-85c6a669f806-istio-token\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-qskf7\" (UID: \"469fc281-f7ee-4584-b3b6-85c6a669f806\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-qskf7"
Nov 03 20:18:42.000290 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:41.989257    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/469fc281-f7ee-4584-b3b6-85c6a669f806-istio-data\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-qskf7\" (UID: \"469fc281-f7ee-4584-b3b6-85c6a669f806\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-qskf7"
Nov 03 20:18:42.041729 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:42.036263    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e026bfa1-f899-46e8-9309-be16eea226bb-istiod-ca-cert\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-xt4z5\" (UID: \"e026bfa1-f899-46e8-9309-be16eea226bb\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-xt4z5"
Nov 03 20:18:42.041729 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:42.036354    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e026bfa1-f899-46e8-9309-be16eea226bb-credential-socket\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-xt4z5\" (UID: \"e026bfa1-f899-46e8-9309-be16eea226bb\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-xt4z5"
Nov 03 20:18:42.041729 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:42.036420    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e026bfa1-f899-46e8-9309-be16eea226bb-istio-token\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-xt4z5\" (UID: \"e026bfa1-f899-46e8-9309-be16eea226bb\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-xt4z5"
Nov 03 20:18:42.041729 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:42.036488    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e026bfa1-f899-46e8-9309-be16eea226bb-istio-envoy\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-xt4z5\" (UID: \"e026bfa1-f899-46e8-9309-be16eea226bb\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-xt4z5"
Nov 03 20:18:42.041729 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:42.036523    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e026bfa1-f899-46e8-9309-be16eea226bb-workload-socket\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-xt4z5\" (UID: \"e026bfa1-f899-46e8-9309-be16eea226bb\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-xt4z5"
Nov 03 20:18:42.041729 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:42.036568    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e026bfa1-f899-46e8-9309-be16eea226bb-istio-podinfo\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-xt4z5\" (UID: \"e026bfa1-f899-46e8-9309-be16eea226bb\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-xt4z5"
Nov 03 20:18:42.041729 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:42.036717    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e026bfa1-f899-46e8-9309-be16eea226bb-istio-data\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-xt4z5\" (UID: \"e026bfa1-f899-46e8-9309-be16eea226bb\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-xt4z5"
Nov 03 20:18:42.041729 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:42.036815    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e026bfa1-f899-46e8-9309-be16eea226bb-workload-certs\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-xt4z5\" (UID: \"e026bfa1-f899-46e8-9309-be16eea226bb\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-xt4z5"
Nov 03 20:18:42.041729 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:42.036923    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5zsf\" (UniqueName: \"kubernetes.io/projected/e026bfa1-f899-46e8-9309-be16eea226bb-kube-api-access-m5zsf\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-xt4z5\" (UID: \"e026bfa1-f899-46e8-9309-be16eea226bb\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-xt4z5"
Nov 03 20:18:42.041729 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:42.038355    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e026bfa1-f899-46e8-9309-be16eea226bb-istiod-ca-cert\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-xt4z5\" (UID: \"e026bfa1-f899-46e8-9309-be16eea226bb\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-xt4z5"
Nov 03 20:18:42.041729 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:42.038677    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e026bfa1-f899-46e8-9309-be16eea226bb-credential-socket\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-xt4z5\" (UID: \"e026bfa1-f899-46e8-9309-be16eea226bb\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-xt4z5"
Nov 03 20:18:42.041729 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:42.041060    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e026bfa1-f899-46e8-9309-be16eea226bb-workload-socket\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-xt4z5\" (UID: \"e026bfa1-f899-46e8-9309-be16eea226bb\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-xt4z5"
Nov 03 20:18:42.041729 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:42.041259    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e026bfa1-f899-46e8-9309-be16eea226bb-istio-data\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-xt4z5\" (UID: \"e026bfa1-f899-46e8-9309-be16eea226bb\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-xt4z5"
Nov 03 20:18:42.041729 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:42.041343    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e026bfa1-f899-46e8-9309-be16eea226bb-workload-certs\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-xt4z5\" (UID: \"e026bfa1-f899-46e8-9309-be16eea226bb\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-xt4z5"
Nov 03 20:18:42.061980 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:42.061839    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e026bfa1-f899-46e8-9309-be16eea226bb-istio-podinfo\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-xt4z5\" (UID: \"e026bfa1-f899-46e8-9309-be16eea226bb\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-xt4z5"
Nov 03 20:18:42.082999 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:42.082956    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pwb2\" (UniqueName: \"kubernetes.io/projected/469fc281-f7ee-4584-b3b6-85c6a669f806-kube-api-access-6pwb2\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-qskf7\" (UID: \"469fc281-f7ee-4584-b3b6-85c6a669f806\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-qskf7"
Nov 03 20:18:42.086880 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:42.083706    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/469fc281-f7ee-4584-b3b6-85c6a669f806-istio-podinfo\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-qskf7\" (UID: \"469fc281-f7ee-4584-b3b6-85c6a669f806\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-qskf7"
Nov 03 20:18:42.092261 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:42.088059    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e026bfa1-f899-46e8-9309-be16eea226bb-istio-envoy\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-xt4z5\" (UID: \"e026bfa1-f899-46e8-9309-be16eea226bb\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-xt4z5"
Nov 03 20:18:42.092494 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:42.091243    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5zsf\" (UniqueName: \"kubernetes.io/projected/e026bfa1-f899-46e8-9309-be16eea226bb-kube-api-access-m5zsf\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-xt4z5\" (UID: \"e026bfa1-f899-46e8-9309-be16eea226bb\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-xt4z5"
Nov 03 20:18:42.116268 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:42.111454    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e026bfa1-f899-46e8-9309-be16eea226bb-istio-token\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-xt4z5\" (UID: \"e026bfa1-f899-46e8-9309-be16eea226bb\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-xt4z5"
Nov 03 20:18:42.123223 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:42.123093    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/469fc281-f7ee-4584-b3b6-85c6a669f806-istio-envoy\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-qskf7\" (UID: \"469fc281-f7ee-4584-b3b6-85c6a669f806\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-qskf7"
Nov 03 20:18:42.132404 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:42.132164    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-xt4z5"
Nov 03 20:18:42.303580 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:42.303541    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-qskf7"
Nov 03 20:18:42.470043 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:42.470013    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-xt4z5"]
Nov 03 20:18:42.470436 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:18:42.470413    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode026bfa1_f899_46e8_9309_be16eea226bb.slice/crio-66ed62de0da2e8c8f541b3358d6cfe7819efcd760b7e252f5bb2732f5400f29a WatchSource:0}: Error finding container 66ed62de0da2e8c8f541b3358d6cfe7819efcd760b7e252f5bb2732f5400f29a: Status 404 returned error can't find the container with id 66ed62de0da2e8c8f541b3358d6cfe7819efcd760b7e252f5bb2732f5400f29a
Nov 03 20:18:42.473177 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:42.473141    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:18:42.473321 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:42.473211    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:18:42.473321 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:42.473246    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:18:42.502120 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:42.502094    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-xt4z5" event={"ID":"e026bfa1-f899-46e8-9309-be16eea226bb","Type":"ContainerStarted","Data":"66ed62de0da2e8c8f541b3358d6cfe7819efcd760b7e252f5bb2732f5400f29a"}
Nov 03 20:18:42.619199 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:42.614573    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-qskf7"]
Nov 03 20:18:42.622062 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:42.622023    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:18:42.622177 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:42.622115    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:18:42.622235 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:42.622180    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:18:43.510668 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:43.510625    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-xt4z5" event={"ID":"e026bfa1-f899-46e8-9309-be16eea226bb","Type":"ContainerStarted","Data":"b4c63954b88fed3fa4e0f508daa689ef20d457a83e2b12c3f5d3a8c3e34f6308"}
Nov 03 20:18:43.512358 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:43.512326    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-qskf7" event={"ID":"469fc281-f7ee-4584-b3b6-85c6a669f806","Type":"ContainerStarted","Data":"04cc270bc0362b4abb3459a1a7771e91e258cc93bb68adfcdb883fbb3afe99bd"}
Nov 03 20:18:43.512486 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:43.512373    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-qskf7" event={"ID":"469fc281-f7ee-4584-b3b6-85c6a669f806","Type":"ContainerStarted","Data":"cdb58f004d531827d29b96dde040e3906021c454874c789ca6b67e00663dbcfa"}
Nov 03 20:18:43.617886 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:43.617806    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-xt4z5" podStartSLOduration=2.617759748 podStartE2EDuration="2.617759748s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:18:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:18:43.604159531 +0000 UTC m=+857.756821455" watchObservedRunningTime="2026-11-03 20:18:43.617759748 +0000 UTC m=+857.770421634"
Nov 03 20:18:44.132283 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:44.132244    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-xt4z5"
Nov 03 20:18:44.133332 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:44.133307    5570 patch_prober.go:29] interesting pod/e2e-worker-w0-e40816-openshift-default-744c57967c-xt4z5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Nov 03 20:18:44.133423 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:44.133371    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-xt4z5" podUID="e026bfa1-f899-46e8-9309-be16eea226bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Nov 03 20:18:44.304135 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:44.304087    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-qskf7"
Nov 03 20:18:44.305568 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:44.305532    5570 patch_prober.go:29] interesting pod/e2e-worker-w2-9973d9-openshift-default-ffb6475-qskf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Nov 03 20:18:44.305629 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:44.305592    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-qskf7" podUID="469fc281-f7ee-4584-b3b6-85c6a669f806" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Nov 03 20:18:45.132665 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:45.132622    5570 patch_prober.go:29] interesting pod/e2e-worker-w0-e40816-openshift-default-744c57967c-xt4z5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Nov 03 20:18:45.133207 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:45.133182    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-xt4z5" podUID="e026bfa1-f899-46e8-9309-be16eea226bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Nov 03 20:18:45.304389 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:45.304352    5570 patch_prober.go:29] interesting pod/e2e-worker-w2-9973d9-openshift-default-ffb6475-qskf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Nov 03 20:18:45.304571 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:45.304408    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-qskf7" podUID="469fc281-f7ee-4584-b3b6-85c6a669f806" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Nov 03 20:18:46.132565 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:46.132531    5570 patch_prober.go:29] interesting pod/e2e-worker-w0-e40816-openshift-default-744c57967c-xt4z5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Nov 03 20:18:46.132817 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:46.132584    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-xt4z5" podUID="e026bfa1-f899-46e8-9309-be16eea226bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Nov 03 20:18:46.304566 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:46.304531    5570 patch_prober.go:29] interesting pod/e2e-worker-w2-9973d9-openshift-default-ffb6475-qskf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Nov 03 20:18:46.304838 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:46.304814    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-qskf7" podUID="469fc281-f7ee-4584-b3b6-85c6a669f806" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Nov 03 20:18:47.132904 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:47.132865    5570 patch_prober.go:29] interesting pod/e2e-worker-w0-e40816-openshift-default-744c57967c-xt4z5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Nov 03 20:18:47.133301 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:47.132944    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-xt4z5" podUID="e026bfa1-f899-46e8-9309-be16eea226bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Nov 03 20:18:47.304078 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:47.304043    5570 patch_prober.go:29] interesting pod/e2e-worker-w2-9973d9-openshift-default-ffb6475-qskf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Nov 03 20:18:47.304263 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:47.304098    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-qskf7" podUID="469fc281-f7ee-4584-b3b6-85c6a669f806" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Nov 03 20:18:48.132809 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:48.132745    5570 patch_prober.go:29] interesting pod/e2e-worker-w0-e40816-openshift-default-744c57967c-xt4z5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Nov 03 20:18:48.132995 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:48.132850    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-xt4z5" podUID="e026bfa1-f899-46e8-9309-be16eea226bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Nov 03 20:18:48.307799 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:48.306437    5570 patch_prober.go:29] interesting pod/e2e-worker-w2-9973d9-openshift-default-ffb6475-qskf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Nov 03 20:18:48.307799 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:48.306501    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-qskf7" podUID="469fc281-f7ee-4584-b3b6-85c6a669f806" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Nov 03 20:18:49.132357 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:49.132317    5570 patch_prober.go:29] interesting pod/e2e-worker-w0-e40816-openshift-default-744c57967c-xt4z5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Nov 03 20:18:49.132601 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:49.132373    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-xt4z5" podUID="e026bfa1-f899-46e8-9309-be16eea226bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Nov 03 20:18:49.304039 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:49.303999    5570 patch_prober.go:29] interesting pod/e2e-worker-w2-9973d9-openshift-default-ffb6475-qskf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Nov 03 20:18:49.304675 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:49.304629    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-qskf7" podUID="469fc281-f7ee-4584-b3b6-85c6a669f806" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Nov 03 20:18:50.137455 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:50.137418    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-xt4z5"
Nov 03 20:18:50.137823 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:50.137801    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-xt4z5"
Nov 03 20:18:50.138568 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:50.138552    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-xt4z5"
Nov 03 20:18:50.208876 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:50.208808    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-qskf7" podStartSLOduration=9.208787871 podStartE2EDuration="9.208787871s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:18:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:18:43.693191747 +0000 UTC m=+857.845853634" watchObservedRunningTime="2026-11-03 20:18:50.208787871 +0000 UTC m=+864.361449753"
Nov 03 20:18:50.309227 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:50.309197    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-qskf7"
Nov 03 20:18:50.309617 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:50.309522    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-qskf7"
Nov 03 20:18:50.310278 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:50.310256    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-qskf7"
Nov 03 20:18:53.568594 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:53.568555    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-482722/distinct-w3-482722-kserve-d876967c-kd488"]
Nov 03 20:18:53.575151 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:53.575106    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-482722/distinct-w3-482722-kserve-d876967c-kd488"
Nov 03 20:18:53.589810 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:53.589743    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-482722/distinct-w3-482722-kserve-d876967c-kd488"]
Nov 03 20:18:53.598754 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:53.598724    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-482722\"/\"distinct-w3-482722-kserve-self-signed-certs\""
Nov 03 20:18:53.648941 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:53.648900    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a97f9cc-2261-471d-9053-51ee965850bd-tmp-dir\") pod \"distinct-w3-482722-kserve-d876967c-kd488\" (UID: \"9a97f9cc-2261-471d-9053-51ee965850bd\") " pod="e2e-models-e2e-worker-w3-482722/distinct-w3-482722-kserve-d876967c-kd488"
Nov 03 20:18:53.648941 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:53.648946    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a97f9cc-2261-471d-9053-51ee965850bd-dshm\") pod \"distinct-w3-482722-kserve-d876967c-kd488\" (UID: \"9a97f9cc-2261-471d-9053-51ee965850bd\") " pod="e2e-models-e2e-worker-w3-482722/distinct-w3-482722-kserve-d876967c-kd488"
Nov 03 20:18:53.649161 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:53.649083    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a97f9cc-2261-471d-9053-51ee965850bd-model-cache\") pod \"distinct-w3-482722-kserve-d876967c-kd488\" (UID: \"9a97f9cc-2261-471d-9053-51ee965850bd\") " pod="e2e-models-e2e-worker-w3-482722/distinct-w3-482722-kserve-d876967c-kd488"
Nov 03 20:18:53.649209 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:53.649196    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a97f9cc-2261-471d-9053-51ee965850bd-tls-certs\") pod \"distinct-w3-482722-kserve-d876967c-kd488\" (UID: \"9a97f9cc-2261-471d-9053-51ee965850bd\") " pod="e2e-models-e2e-worker-w3-482722/distinct-w3-482722-kserve-d876967c-kd488"
Nov 03 20:18:53.649245 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:53.649229    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a97f9cc-2261-471d-9053-51ee965850bd-home\") pod \"distinct-w3-482722-kserve-d876967c-kd488\" (UID: \"9a97f9cc-2261-471d-9053-51ee965850bd\") " pod="e2e-models-e2e-worker-w3-482722/distinct-w3-482722-kserve-d876967c-kd488"
Nov 03 20:18:53.649393 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:53.649381    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7s8s7\" (UniqueName: \"kubernetes.io/projected/9a97f9cc-2261-471d-9053-51ee965850bd-kube-api-access-7s8s7\") pod \"distinct-w3-482722-kserve-d876967c-kd488\" (UID: \"9a97f9cc-2261-471d-9053-51ee965850bd\") " pod="e2e-models-e2e-worker-w3-482722/distinct-w3-482722-kserve-d876967c-kd488"
Nov 03 20:18:53.760381 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:53.760267    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a97f9cc-2261-471d-9053-51ee965850bd-tmp-dir\") pod \"distinct-w3-482722-kserve-d876967c-kd488\" (UID: \"9a97f9cc-2261-471d-9053-51ee965850bd\") " pod="e2e-models-e2e-worker-w3-482722/distinct-w3-482722-kserve-d876967c-kd488"
Nov 03 20:18:53.760795 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:53.760751    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a97f9cc-2261-471d-9053-51ee965850bd-dshm\") pod \"distinct-w3-482722-kserve-d876967c-kd488\" (UID: \"9a97f9cc-2261-471d-9053-51ee965850bd\") " pod="e2e-models-e2e-worker-w3-482722/distinct-w3-482722-kserve-d876967c-kd488"
Nov 03 20:18:53.761051 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:53.761030    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a97f9cc-2261-471d-9053-51ee965850bd-model-cache\") pod \"distinct-w3-482722-kserve-d876967c-kd488\" (UID: \"9a97f9cc-2261-471d-9053-51ee965850bd\") " pod="e2e-models-e2e-worker-w3-482722/distinct-w3-482722-kserve-d876967c-kd488"
Nov 03 20:18:53.761354 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:53.761333    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a97f9cc-2261-471d-9053-51ee965850bd-tls-certs\") pod \"distinct-w3-482722-kserve-d876967c-kd488\" (UID: \"9a97f9cc-2261-471d-9053-51ee965850bd\") " pod="e2e-models-e2e-worker-w3-482722/distinct-w3-482722-kserve-d876967c-kd488"
Nov 03 20:18:53.761539 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:53.761520    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a97f9cc-2261-471d-9053-51ee965850bd-home\") pod \"distinct-w3-482722-kserve-d876967c-kd488\" (UID: \"9a97f9cc-2261-471d-9053-51ee965850bd\") " pod="e2e-models-e2e-worker-w3-482722/distinct-w3-482722-kserve-d876967c-kd488"
Nov 03 20:18:53.761871 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:53.761851    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7s8s7\" (UniqueName: \"kubernetes.io/projected/9a97f9cc-2261-471d-9053-51ee965850bd-kube-api-access-7s8s7\") pod \"distinct-w3-482722-kserve-d876967c-kd488\" (UID: \"9a97f9cc-2261-471d-9053-51ee965850bd\") " pod="e2e-models-e2e-worker-w3-482722/distinct-w3-482722-kserve-d876967c-kd488"
Nov 03 20:18:53.763849 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:53.763827    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a97f9cc-2261-471d-9053-51ee965850bd-model-cache\") pod \"distinct-w3-482722-kserve-d876967c-kd488\" (UID: \"9a97f9cc-2261-471d-9053-51ee965850bd\") " pod="e2e-models-e2e-worker-w3-482722/distinct-w3-482722-kserve-d876967c-kd488"
Nov 03 20:18:53.764408 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:53.764390    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a97f9cc-2261-471d-9053-51ee965850bd-tmp-dir\") pod \"distinct-w3-482722-kserve-d876967c-kd488\" (UID: \"9a97f9cc-2261-471d-9053-51ee965850bd\") " pod="e2e-models-e2e-worker-w3-482722/distinct-w3-482722-kserve-d876967c-kd488"
Nov 03 20:18:53.799355 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:53.769095    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a97f9cc-2261-471d-9053-51ee965850bd-home\") pod \"distinct-w3-482722-kserve-d876967c-kd488\" (UID: \"9a97f9cc-2261-471d-9053-51ee965850bd\") " pod="e2e-models-e2e-worker-w3-482722/distinct-w3-482722-kserve-d876967c-kd488"
Nov 03 20:18:53.800372 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:53.800297    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a97f9cc-2261-471d-9053-51ee965850bd-tls-certs\") pod \"distinct-w3-482722-kserve-d876967c-kd488\" (UID: \"9a97f9cc-2261-471d-9053-51ee965850bd\") " pod="e2e-models-e2e-worker-w3-482722/distinct-w3-482722-kserve-d876967c-kd488"
Nov 03 20:18:53.804836 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:53.800827    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7s8s7\" (UniqueName: \"kubernetes.io/projected/9a97f9cc-2261-471d-9053-51ee965850bd-kube-api-access-7s8s7\") pod \"distinct-w3-482722-kserve-d876967c-kd488\" (UID: \"9a97f9cc-2261-471d-9053-51ee965850bd\") " pod="e2e-models-e2e-worker-w3-482722/distinct-w3-482722-kserve-d876967c-kd488"
Nov 03 20:18:53.813414 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:53.810131    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a97f9cc-2261-471d-9053-51ee965850bd-dshm\") pod \"distinct-w3-482722-kserve-d876967c-kd488\" (UID: \"9a97f9cc-2261-471d-9053-51ee965850bd\") " pod="e2e-models-e2e-worker-w3-482722/distinct-w3-482722-kserve-d876967c-kd488"
Nov 03 20:18:53.895854 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:53.893053    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-482722/distinct-w3-482722-kserve-d876967c-kd488"
Nov 03 20:18:54.234520 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:54.234485    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-482722/distinct-w3-482722-kserve-d876967c-kd488"]
Nov 03 20:18:54.235272 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:18:54.235228    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9a97f9cc_2261_471d_9053_51ee965850bd.slice/crio-ad770193bae872b6c4e021f6c8f1c6e4b8ecbf8c12db828b3d4ae5b57950572f WatchSource:0}: Error finding container ad770193bae872b6c4e021f6c8f1c6e4b8ecbf8c12db828b3d4ae5b57950572f: Status 404 returned error can't find the container with id ad770193bae872b6c4e021f6c8f1c6e4b8ecbf8c12db828b3d4ae5b57950572f
Nov 03 20:18:54.613104 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:54.613011    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-482722/distinct-w3-482722-kserve-d876967c-kd488" event={"ID":"9a97f9cc-2261-471d-9053-51ee965850bd","Type":"ContainerStarted","Data":"a233a0e6dfafb8dfd9e6fe03c9a65d2eb2db2e0adf497de68e8b2e56a92e53fe"}
Nov 03 20:18:54.613104 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:54.613054    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-482722/distinct-w3-482722-kserve-d876967c-kd488" event={"ID":"9a97f9cc-2261-471d-9053-51ee965850bd","Type":"ContainerStarted","Data":"ad770193bae872b6c4e021f6c8f1c6e4b8ecbf8c12db828b3d4ae5b57950572f"}
Nov 03 20:18:54.894581 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:54.894483    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-482722/distinct-w3-482722-kserve-d876967c-kd488"
Nov 03 20:18:57.447205 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:18:57.447164    5570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8bwhz" containerName="extractor"
Nov 03 20:18:57.639798 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:18:57.639003    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/premium-w2-9973d9-kserve-5c6dd4669c-5gv8x" podUID="cefc89ad-7ae1-433b-979d-6601678a6d98" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:19:03.894127 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:03.894078    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-482722/distinct-w3-482722-kserve-d876967c-kd488"
Nov 03 20:19:03.906814 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:03.906782    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-482722/distinct-w3-482722-kserve-d876967c-kd488"
Nov 03 20:19:03.945946 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:03.945915    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-482722/distinct-w3-482722-kserve-d876967c-kd488"
Nov 03 20:19:03.950116 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:03.950069    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-482722/distinct-w3-482722-kserve-d876967c-kd488" podStartSLOduration=10.95005471 podStartE2EDuration="10.95005471s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:18:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:18:54.664788878 +0000 UTC m=+868.817450760" watchObservedRunningTime="2026-11-03 20:19:03.95005471 +0000 UTC m=+878.102716637"
Nov 03 20:19:05.770862 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:05.770821    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-9e1bcb/distinct-2-w6-9e1bcb-kserve-7d96d9f6f6-4f7fl"]
Nov 03 20:19:05.784713 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:05.784659    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-9e1bcb/distinct-2-w6-9e1bcb-kserve-7d96d9f6f6-4f7fl"]
Nov 03 20:19:05.784950 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:05.784863    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-2-w6-9e1bcb-kserve-7d96d9f6f6-4f7fl"
Nov 03 20:19:05.814834 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:05.814805    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-9e1bcb\"/\"distinct-2-w6-9e1bcb-kserve-self-signed-certs\""
Nov 03 20:19:05.920064 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:05.920016    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56353b0f-08ee-4ada-9dc0-ac5afd5a59c7-model-cache\") pod \"distinct-2-w6-9e1bcb-kserve-7d96d9f6f6-4f7fl\" (UID: \"56353b0f-08ee-4ada-9dc0-ac5afd5a59c7\") " pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-2-w6-9e1bcb-kserve-7d96d9f6f6-4f7fl"
Nov 03 20:19:05.920259 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:05.920173    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56353b0f-08ee-4ada-9dc0-ac5afd5a59c7-home\") pod \"distinct-2-w6-9e1bcb-kserve-7d96d9f6f6-4f7fl\" (UID: \"56353b0f-08ee-4ada-9dc0-ac5afd5a59c7\") " pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-2-w6-9e1bcb-kserve-7d96d9f6f6-4f7fl"
Nov 03 20:19:05.920259 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:05.920213    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56353b0f-08ee-4ada-9dc0-ac5afd5a59c7-tmp-dir\") pod \"distinct-2-w6-9e1bcb-kserve-7d96d9f6f6-4f7fl\" (UID: \"56353b0f-08ee-4ada-9dc0-ac5afd5a59c7\") " pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-2-w6-9e1bcb-kserve-7d96d9f6f6-4f7fl"
Nov 03 20:19:05.920388 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:05.920335    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pg4w4\" (UniqueName: \"kubernetes.io/projected/56353b0f-08ee-4ada-9dc0-ac5afd5a59c7-kube-api-access-pg4w4\") pod \"distinct-2-w6-9e1bcb-kserve-7d96d9f6f6-4f7fl\" (UID: \"56353b0f-08ee-4ada-9dc0-ac5afd5a59c7\") " pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-2-w6-9e1bcb-kserve-7d96d9f6f6-4f7fl"
Nov 03 20:19:05.920388 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:05.920370    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56353b0f-08ee-4ada-9dc0-ac5afd5a59c7-dshm\") pod \"distinct-2-w6-9e1bcb-kserve-7d96d9f6f6-4f7fl\" (UID: \"56353b0f-08ee-4ada-9dc0-ac5afd5a59c7\") " pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-2-w6-9e1bcb-kserve-7d96d9f6f6-4f7fl"
Nov 03 20:19:05.920543 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:05.920522    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56353b0f-08ee-4ada-9dc0-ac5afd5a59c7-tls-certs\") pod \"distinct-2-w6-9e1bcb-kserve-7d96d9f6f6-4f7fl\" (UID: \"56353b0f-08ee-4ada-9dc0-ac5afd5a59c7\") " pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-2-w6-9e1bcb-kserve-7d96d9f6f6-4f7fl"
Nov 03 20:19:06.034883 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:06.022821    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56353b0f-08ee-4ada-9dc0-ac5afd5a59c7-home\") pod \"distinct-2-w6-9e1bcb-kserve-7d96d9f6f6-4f7fl\" (UID: \"56353b0f-08ee-4ada-9dc0-ac5afd5a59c7\") " pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-2-w6-9e1bcb-kserve-7d96d9f6f6-4f7fl"
Nov 03 20:19:06.034883 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:06.022882    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56353b0f-08ee-4ada-9dc0-ac5afd5a59c7-tmp-dir\") pod \"distinct-2-w6-9e1bcb-kserve-7d96d9f6f6-4f7fl\" (UID: \"56353b0f-08ee-4ada-9dc0-ac5afd5a59c7\") " pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-2-w6-9e1bcb-kserve-7d96d9f6f6-4f7fl"
Nov 03 20:19:06.034883 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:06.022970    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pg4w4\" (UniqueName: \"kubernetes.io/projected/56353b0f-08ee-4ada-9dc0-ac5afd5a59c7-kube-api-access-pg4w4\") pod \"distinct-2-w6-9e1bcb-kserve-7d96d9f6f6-4f7fl\" (UID: \"56353b0f-08ee-4ada-9dc0-ac5afd5a59c7\") " pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-2-w6-9e1bcb-kserve-7d96d9f6f6-4f7fl"
Nov 03 20:19:06.034883 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:06.023021    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56353b0f-08ee-4ada-9dc0-ac5afd5a59c7-dshm\") pod \"distinct-2-w6-9e1bcb-kserve-7d96d9f6f6-4f7fl\" (UID: \"56353b0f-08ee-4ada-9dc0-ac5afd5a59c7\") " pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-2-w6-9e1bcb-kserve-7d96d9f6f6-4f7fl"
Nov 03 20:19:06.034883 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:06.023159    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56353b0f-08ee-4ada-9dc0-ac5afd5a59c7-tls-certs\") pod \"distinct-2-w6-9e1bcb-kserve-7d96d9f6f6-4f7fl\" (UID: \"56353b0f-08ee-4ada-9dc0-ac5afd5a59c7\") " pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-2-w6-9e1bcb-kserve-7d96d9f6f6-4f7fl"
Nov 03 20:19:06.034883 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:06.023216    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56353b0f-08ee-4ada-9dc0-ac5afd5a59c7-model-cache\") pod \"distinct-2-w6-9e1bcb-kserve-7d96d9f6f6-4f7fl\" (UID: \"56353b0f-08ee-4ada-9dc0-ac5afd5a59c7\") " pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-2-w6-9e1bcb-kserve-7d96d9f6f6-4f7fl"
Nov 03 20:19:06.034883 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:06.023836    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56353b0f-08ee-4ada-9dc0-ac5afd5a59c7-model-cache\") pod \"distinct-2-w6-9e1bcb-kserve-7d96d9f6f6-4f7fl\" (UID: \"56353b0f-08ee-4ada-9dc0-ac5afd5a59c7\") " pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-2-w6-9e1bcb-kserve-7d96d9f6f6-4f7fl"
Nov 03 20:19:06.034883 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:06.024144    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56353b0f-08ee-4ada-9dc0-ac5afd5a59c7-home\") pod \"distinct-2-w6-9e1bcb-kserve-7d96d9f6f6-4f7fl\" (UID: \"56353b0f-08ee-4ada-9dc0-ac5afd5a59c7\") " pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-2-w6-9e1bcb-kserve-7d96d9f6f6-4f7fl"
Nov 03 20:19:06.034883 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:06.024367    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56353b0f-08ee-4ada-9dc0-ac5afd5a59c7-tmp-dir\") pod \"distinct-2-w6-9e1bcb-kserve-7d96d9f6f6-4f7fl\" (UID: \"56353b0f-08ee-4ada-9dc0-ac5afd5a59c7\") " pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-2-w6-9e1bcb-kserve-7d96d9f6f6-4f7fl"
Nov 03 20:19:06.053650 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:06.053577    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pg4w4\" (UniqueName: \"kubernetes.io/projected/56353b0f-08ee-4ada-9dc0-ac5afd5a59c7-kube-api-access-pg4w4\") pod \"distinct-2-w6-9e1bcb-kserve-7d96d9f6f6-4f7fl\" (UID: \"56353b0f-08ee-4ada-9dc0-ac5afd5a59c7\") " pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-2-w6-9e1bcb-kserve-7d96d9f6f6-4f7fl"
Nov 03 20:19:06.073923 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:06.059064    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56353b0f-08ee-4ada-9dc0-ac5afd5a59c7-dshm\") pod \"distinct-2-w6-9e1bcb-kserve-7d96d9f6f6-4f7fl\" (UID: \"56353b0f-08ee-4ada-9dc0-ac5afd5a59c7\") " pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-2-w6-9e1bcb-kserve-7d96d9f6f6-4f7fl"
Nov 03 20:19:06.073923 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:06.061102    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56353b0f-08ee-4ada-9dc0-ac5afd5a59c7-tls-certs\") pod \"distinct-2-w6-9e1bcb-kserve-7d96d9f6f6-4f7fl\" (UID: \"56353b0f-08ee-4ada-9dc0-ac5afd5a59c7\") " pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-2-w6-9e1bcb-kserve-7d96d9f6f6-4f7fl"
Nov 03 20:19:06.121909 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:06.118309    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-2-w6-9e1bcb-kserve-7d96d9f6f6-4f7fl"
Nov 03 20:19:06.437249 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:06.437220    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-9e1bcb/distinct-2-w6-9e1bcb-kserve-7d96d9f6f6-4f7fl"]
Nov 03 20:19:06.438411 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:19:06.438381    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod56353b0f_08ee_4ada_9dc0_ac5afd5a59c7.slice/crio-58acf036aaca0c18a60e119c07c8c1f96cc7ee096678314226baf6da4026534d WatchSource:0}: Error finding container 58acf036aaca0c18a60e119c07c8c1f96cc7ee096678314226baf6da4026534d: Status 404 returned error can't find the container with id 58acf036aaca0c18a60e119c07c8c1f96cc7ee096678314226baf6da4026534d
Nov 03 20:19:07.063142 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:07.063097    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-2-w6-9e1bcb-kserve-7d96d9f6f6-4f7fl" event={"ID":"56353b0f-08ee-4ada-9dc0-ac5afd5a59c7","Type":"ContainerStarted","Data":"29dc9938d203f96fc96a132d6cbe946dd3dce03c4807b8721a8ff8f7baea05af"}
Nov 03 20:19:07.063142 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:07.063147    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-2-w6-9e1bcb-kserve-7d96d9f6f6-4f7fl" event={"ID":"56353b0f-08ee-4ada-9dc0-ac5afd5a59c7","Type":"ContainerStarted","Data":"58acf036aaca0c18a60e119c07c8c1f96cc7ee096678314226baf6da4026534d"}
Nov 03 20:19:07.106459 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:07.106407    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-2-w6-9e1bcb-kserve-7d96d9f6f6-4f7fl" podStartSLOduration=2.106393645 podStartE2EDuration="2.106393645s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:19:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:19:07.103190372 +0000 UTC m=+881.255852258" watchObservedRunningTime="2026-11-03 20:19:07.106393645 +0000 UTC m=+881.259055529"
Nov 03 20:19:07.118759 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:07.118714    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-2-w6-9e1bcb-kserve-7d96d9f6f6-4f7fl"
Nov 03 20:19:07.767477 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:07.767440    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g69f8"]
Nov 03 20:19:11.673536 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:11.673503    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-wkmg8"]
Nov 03 20:19:11.677735 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:11.677713    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-wkmg8"
Nov 03 20:19:11.715215 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:11.715183    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-wkmg8"]
Nov 03 20:19:11.770226 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:11.770188    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e314286e-4b2d-4fae-bc59-528809a8dbce-istio-token\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-wkmg8\" (UID: \"e314286e-4b2d-4fae-bc59-528809a8dbce\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-wkmg8"
Nov 03 20:19:11.770407 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:11.770240    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e314286e-4b2d-4fae-bc59-528809a8dbce-workload-certs\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-wkmg8\" (UID: \"e314286e-4b2d-4fae-bc59-528809a8dbce\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-wkmg8"
Nov 03 20:19:11.770407 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:11.770356    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e314286e-4b2d-4fae-bc59-528809a8dbce-workload-socket\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-wkmg8\" (UID: \"e314286e-4b2d-4fae-bc59-528809a8dbce\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-wkmg8"
Nov 03 20:19:11.770407 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:11.770391    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qpw8\" (UniqueName: \"kubernetes.io/projected/e314286e-4b2d-4fae-bc59-528809a8dbce-kube-api-access-2qpw8\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-wkmg8\" (UID: \"e314286e-4b2d-4fae-bc59-528809a8dbce\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-wkmg8"
Nov 03 20:19:11.770516 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:11.770452    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e314286e-4b2d-4fae-bc59-528809a8dbce-istiod-ca-cert\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-wkmg8\" (UID: \"e314286e-4b2d-4fae-bc59-528809a8dbce\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-wkmg8"
Nov 03 20:19:11.770516 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:11.770492    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e314286e-4b2d-4fae-bc59-528809a8dbce-istio-envoy\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-wkmg8\" (UID: \"e314286e-4b2d-4fae-bc59-528809a8dbce\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-wkmg8"
Nov 03 20:19:11.770579 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:11.770541    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e314286e-4b2d-4fae-bc59-528809a8dbce-credential-socket\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-wkmg8\" (UID: \"e314286e-4b2d-4fae-bc59-528809a8dbce\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-wkmg8"
Nov 03 20:19:11.770635 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:11.770622    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e314286e-4b2d-4fae-bc59-528809a8dbce-istio-data\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-wkmg8\" (UID: \"e314286e-4b2d-4fae-bc59-528809a8dbce\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-wkmg8"
Nov 03 20:19:11.770670 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:11.770645    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e314286e-4b2d-4fae-bc59-528809a8dbce-istio-podinfo\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-wkmg8\" (UID: \"e314286e-4b2d-4fae-bc59-528809a8dbce\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-wkmg8"
Nov 03 20:19:11.872625 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:11.872589    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e314286e-4b2d-4fae-bc59-528809a8dbce-istio-token\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-wkmg8\" (UID: \"e314286e-4b2d-4fae-bc59-528809a8dbce\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-wkmg8"
Nov 03 20:19:11.872854 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:11.872647    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e314286e-4b2d-4fae-bc59-528809a8dbce-workload-certs\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-wkmg8\" (UID: \"e314286e-4b2d-4fae-bc59-528809a8dbce\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-wkmg8"
Nov 03 20:19:11.872854 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:11.872716    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e314286e-4b2d-4fae-bc59-528809a8dbce-workload-socket\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-wkmg8\" (UID: \"e314286e-4b2d-4fae-bc59-528809a8dbce\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-wkmg8"
Nov 03 20:19:11.872854 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:11.872745    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qpw8\" (UniqueName: \"kubernetes.io/projected/e314286e-4b2d-4fae-bc59-528809a8dbce-kube-api-access-2qpw8\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-wkmg8\" (UID: \"e314286e-4b2d-4fae-bc59-528809a8dbce\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-wkmg8"
Nov 03 20:19:11.872854 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:11.872813    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e314286e-4b2d-4fae-bc59-528809a8dbce-istiod-ca-cert\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-wkmg8\" (UID: \"e314286e-4b2d-4fae-bc59-528809a8dbce\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-wkmg8"
Nov 03 20:19:11.873085 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:11.872864    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e314286e-4b2d-4fae-bc59-528809a8dbce-istio-envoy\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-wkmg8\" (UID: \"e314286e-4b2d-4fae-bc59-528809a8dbce\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-wkmg8"
Nov 03 20:19:11.873085 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:11.872937    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e314286e-4b2d-4fae-bc59-528809a8dbce-credential-socket\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-wkmg8\" (UID: \"e314286e-4b2d-4fae-bc59-528809a8dbce\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-wkmg8"
Nov 03 20:19:11.873085 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:11.872971    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e314286e-4b2d-4fae-bc59-528809a8dbce-istio-data\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-wkmg8\" (UID: \"e314286e-4b2d-4fae-bc59-528809a8dbce\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-wkmg8"
Nov 03 20:19:11.873085 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:11.873001    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e314286e-4b2d-4fae-bc59-528809a8dbce-istio-podinfo\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-wkmg8\" (UID: \"e314286e-4b2d-4fae-bc59-528809a8dbce\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-wkmg8"
Nov 03 20:19:11.873085 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:11.873073    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e314286e-4b2d-4fae-bc59-528809a8dbce-workload-certs\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-wkmg8\" (UID: \"e314286e-4b2d-4fae-bc59-528809a8dbce\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-wkmg8"
Nov 03 20:19:11.873373 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:11.873111    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e314286e-4b2d-4fae-bc59-528809a8dbce-workload-socket\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-wkmg8\" (UID: \"e314286e-4b2d-4fae-bc59-528809a8dbce\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-wkmg8"
Nov 03 20:19:11.873430 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:11.873387    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e314286e-4b2d-4fae-bc59-528809a8dbce-credential-socket\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-wkmg8\" (UID: \"e314286e-4b2d-4fae-bc59-528809a8dbce\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-wkmg8"
Nov 03 20:19:11.873430 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:11.873411    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e314286e-4b2d-4fae-bc59-528809a8dbce-istio-data\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-wkmg8\" (UID: \"e314286e-4b2d-4fae-bc59-528809a8dbce\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-wkmg8"
Nov 03 20:19:11.873664 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:11.873638    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e314286e-4b2d-4fae-bc59-528809a8dbce-istiod-ca-cert\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-wkmg8\" (UID: \"e314286e-4b2d-4fae-bc59-528809a8dbce\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-wkmg8"
Nov 03 20:19:11.875350 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:11.875327    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e314286e-4b2d-4fae-bc59-528809a8dbce-istio-envoy\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-wkmg8\" (UID: \"e314286e-4b2d-4fae-bc59-528809a8dbce\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-wkmg8"
Nov 03 20:19:11.875570 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:11.875553    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e314286e-4b2d-4fae-bc59-528809a8dbce-istio-podinfo\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-wkmg8\" (UID: \"e314286e-4b2d-4fae-bc59-528809a8dbce\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-wkmg8"
Nov 03 20:19:11.892682 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:11.892652    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e314286e-4b2d-4fae-bc59-528809a8dbce-istio-token\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-wkmg8\" (UID: \"e314286e-4b2d-4fae-bc59-528809a8dbce\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-wkmg8"
Nov 03 20:19:11.892878 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:11.892856    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qpw8\" (UniqueName: \"kubernetes.io/projected/e314286e-4b2d-4fae-bc59-528809a8dbce-kube-api-access-2qpw8\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-wkmg8\" (UID: \"e314286e-4b2d-4fae-bc59-528809a8dbce\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-wkmg8"
Nov 03 20:19:11.996886 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:11.996849    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-wkmg8"
Nov 03 20:19:12.317280 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:19:12.314291    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode314286e_4b2d_4fae_bc59_528809a8dbce.slice/crio-1a75e80c761a8d9ef7d2023f9be3da61cca4e9173f219793eab19617ef57d8d7 WatchSource:0}: Error finding container 1a75e80c761a8d9ef7d2023f9be3da61cca4e9173f219793eab19617ef57d8d7: Status 404 returned error can't find the container with id 1a75e80c761a8d9ef7d2023f9be3da61cca4e9173f219793eab19617ef57d8d7
Nov 03 20:19:12.317280 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:12.314536    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-wkmg8"]
Nov 03 20:19:12.317611 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:12.317455    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:19:12.317611 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:12.317521    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:19:12.317611 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:12.317549    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:19:13.168202 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:13.168166    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-wkmg8" event={"ID":"e314286e-4b2d-4fae-bc59-528809a8dbce","Type":"ContainerStarted","Data":"39dce94b7e35773069c096fb3e6dee58c6fc5bf5e34a8cbdbb93fa453abea1cc"}
Nov 03 20:19:13.168611 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:13.168219    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-wkmg8" event={"ID":"e314286e-4b2d-4fae-bc59-528809a8dbce","Type":"ContainerStarted","Data":"1a75e80c761a8d9ef7d2023f9be3da61cca4e9173f219793eab19617ef57d8d7"}
Nov 03 20:19:13.247807 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:13.247530    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-wkmg8" podStartSLOduration=2.247508274 podStartE2EDuration="2.247508274s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:19:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:19:13.226565699 +0000 UTC m=+887.379227584" watchObservedRunningTime="2026-11-03 20:19:13.247508274 +0000 UTC m=+887.400170162"
Nov 03 20:19:13.997264 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:13.997226    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-wkmg8"
Nov 03 20:19:13.999106 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:13.999074    5570 patch_prober.go:29] interesting pod/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-wkmg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Nov 03 20:19:13.999266 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:13.999138    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-wkmg8" podUID="e314286e-4b2d-4fae-bc59-528809a8dbce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Nov 03 20:19:14.997973 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:14.997935    5570 patch_prober.go:29] interesting pod/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-wkmg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Nov 03 20:19:14.998378 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:14.997991    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-wkmg8" podUID="e314286e-4b2d-4fae-bc59-528809a8dbce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Nov 03 20:19:16.005796 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:16.002160    5570 patch_prober.go:29] interesting pod/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-wkmg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Nov 03 20:19:16.005796 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:16.002227    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-wkmg8" podUID="e314286e-4b2d-4fae-bc59-528809a8dbce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Nov 03 20:19:16.119245 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:16.119210    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-2-w6-9e1bcb-kserve-7d96d9f6f6-4f7fl"
Nov 03 20:19:16.131496 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:16.131463    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-2-w6-9e1bcb-kserve-7d96d9f6f6-4f7fl"
Nov 03 20:19:16.205538 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:16.205507    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-2-w6-9e1bcb-kserve-7d96d9f6f6-4f7fl"
Nov 03 20:19:16.998198 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:16.998158    5570 patch_prober.go:29] interesting pod/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-wkmg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Nov 03 20:19:16.998421 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:16.998223    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-wkmg8" podUID="e314286e-4b2d-4fae-bc59-528809a8dbce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Nov 03 20:19:17.999409 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:17.999374    5570 patch_prober.go:29] interesting pod/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-wkmg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Nov 03 20:19:18.000028 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:17.999982    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-wkmg8" podUID="e314286e-4b2d-4fae-bc59-528809a8dbce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Nov 03 20:19:19.409307 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:19.409209    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-wkmg8"
Nov 03 20:19:19.410392 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:19.410362    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-wkmg8"
Nov 03 20:19:19.411402 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:19.411361    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-wkmg8"
Nov 03 20:19:26.685339 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:26.685305    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-5c82t"]
Nov 03 20:19:26.691259 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:26.691231    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-5c82t"
Nov 03 20:19:26.732031 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:26.731996    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-5c82t"]
Nov 03 20:19:26.880795 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:26.858327    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6029825e-e953-4c43-9f4d-d19b5ca38623-credential-socket\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-5c82t\" (UID: \"6029825e-e953-4c43-9f4d-d19b5ca38623\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-5c82t"
Nov 03 20:19:26.916812 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:26.889907    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6029825e-e953-4c43-9f4d-d19b5ca38623-workload-socket\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-5c82t\" (UID: \"6029825e-e953-4c43-9f4d-d19b5ca38623\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-5c82t"
Nov 03 20:19:26.916812 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:26.890087    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6029825e-e953-4c43-9f4d-d19b5ca38623-istio-podinfo\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-5c82t\" (UID: \"6029825e-e953-4c43-9f4d-d19b5ca38623\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-5c82t"
Nov 03 20:19:26.916812 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:26.890139    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6029825e-e953-4c43-9f4d-d19b5ca38623-workload-certs\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-5c82t\" (UID: \"6029825e-e953-4c43-9f4d-d19b5ca38623\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-5c82t"
Nov 03 20:19:26.916812 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:26.890186    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6029825e-e953-4c43-9f4d-d19b5ca38623-istio-envoy\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-5c82t\" (UID: \"6029825e-e953-4c43-9f4d-d19b5ca38623\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-5c82t"
Nov 03 20:19:26.951043 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:26.951007    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6029825e-e953-4c43-9f4d-d19b5ca38623-istio-token\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-5c82t\" (UID: \"6029825e-e953-4c43-9f4d-d19b5ca38623\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-5c82t"
Nov 03 20:19:26.951477 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:26.951455    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ff2pz\" (UniqueName: \"kubernetes.io/projected/6029825e-e953-4c43-9f4d-d19b5ca38623-kube-api-access-ff2pz\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-5c82t\" (UID: \"6029825e-e953-4c43-9f4d-d19b5ca38623\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-5c82t"
Nov 03 20:19:26.951659 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:26.951643    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6029825e-e953-4c43-9f4d-d19b5ca38623-istio-data\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-5c82t\" (UID: \"6029825e-e953-4c43-9f4d-d19b5ca38623\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-5c82t"
Nov 03 20:19:26.951804 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:26.951789    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6029825e-e953-4c43-9f4d-d19b5ca38623-istiod-ca-cert\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-5c82t\" (UID: \"6029825e-e953-4c43-9f4d-d19b5ca38623\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-5c82t"
Nov 03 20:19:27.068901 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:27.068861    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6029825e-e953-4c43-9f4d-d19b5ca38623-istio-data\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-5c82t\" (UID: \"6029825e-e953-4c43-9f4d-d19b5ca38623\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-5c82t"
Nov 03 20:19:27.069189 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:27.069162    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6029825e-e953-4c43-9f4d-d19b5ca38623-istiod-ca-cert\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-5c82t\" (UID: \"6029825e-e953-4c43-9f4d-d19b5ca38623\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-5c82t"
Nov 03 20:19:27.069376 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:27.069360    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6029825e-e953-4c43-9f4d-d19b5ca38623-credential-socket\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-5c82t\" (UID: \"6029825e-e953-4c43-9f4d-d19b5ca38623\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-5c82t"
Nov 03 20:19:27.069513 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:27.069497    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6029825e-e953-4c43-9f4d-d19b5ca38623-workload-socket\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-5c82t\" (UID: \"6029825e-e953-4c43-9f4d-d19b5ca38623\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-5c82t"
Nov 03 20:19:27.070128 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:27.070101    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6029825e-e953-4c43-9f4d-d19b5ca38623-istio-podinfo\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-5c82t\" (UID: \"6029825e-e953-4c43-9f4d-d19b5ca38623\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-5c82t"
Nov 03 20:19:27.070288 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:27.070266    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6029825e-e953-4c43-9f4d-d19b5ca38623-workload-certs\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-5c82t\" (UID: \"6029825e-e953-4c43-9f4d-d19b5ca38623\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-5c82t"
Nov 03 20:19:27.070455 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:27.070437    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6029825e-e953-4c43-9f4d-d19b5ca38623-istio-envoy\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-5c82t\" (UID: \"6029825e-e953-4c43-9f4d-d19b5ca38623\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-5c82t"
Nov 03 20:19:27.070742 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:27.070722    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6029825e-e953-4c43-9f4d-d19b5ca38623-istio-token\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-5c82t\" (UID: \"6029825e-e953-4c43-9f4d-d19b5ca38623\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-5c82t"
Nov 03 20:19:27.070940 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:27.070922    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ff2pz\" (UniqueName: \"kubernetes.io/projected/6029825e-e953-4c43-9f4d-d19b5ca38623-kube-api-access-ff2pz\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-5c82t\" (UID: \"6029825e-e953-4c43-9f4d-d19b5ca38623\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-5c82t"
Nov 03 20:19:27.071367 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:27.071339    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6029825e-e953-4c43-9f4d-d19b5ca38623-istiod-ca-cert\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-5c82t\" (UID: \"6029825e-e953-4c43-9f4d-d19b5ca38623\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-5c82t"
Nov 03 20:19:27.071763 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:27.071735    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6029825e-e953-4c43-9f4d-d19b5ca38623-credential-socket\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-5c82t\" (UID: \"6029825e-e953-4c43-9f4d-d19b5ca38623\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-5c82t"
Nov 03 20:19:27.072095 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:27.072071    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6029825e-e953-4c43-9f4d-d19b5ca38623-workload-socket\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-5c82t\" (UID: \"6029825e-e953-4c43-9f4d-d19b5ca38623\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-5c82t"
Nov 03 20:19:27.072334 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:27.072313    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6029825e-e953-4c43-9f4d-d19b5ca38623-istio-data\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-5c82t\" (UID: \"6029825e-e953-4c43-9f4d-d19b5ca38623\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-5c82t"
Nov 03 20:19:27.090209 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:27.090172    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6029825e-e953-4c43-9f4d-d19b5ca38623-workload-certs\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-5c82t\" (UID: \"6029825e-e953-4c43-9f4d-d19b5ca38623\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-5c82t"
Nov 03 20:19:27.118387 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:27.118351    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6029825e-e953-4c43-9f4d-d19b5ca38623-istio-envoy\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-5c82t\" (UID: \"6029825e-e953-4c43-9f4d-d19b5ca38623\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-5c82t"
Nov 03 20:19:27.148799 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:27.136124    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6029825e-e953-4c43-9f4d-d19b5ca38623-istio-podinfo\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-5c82t\" (UID: \"6029825e-e953-4c43-9f4d-d19b5ca38623\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-5c82t"
Nov 03 20:19:27.153794 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:27.149845    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ff2pz\" (UniqueName: \"kubernetes.io/projected/6029825e-e953-4c43-9f4d-d19b5ca38623-kube-api-access-ff2pz\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-5c82t\" (UID: \"6029825e-e953-4c43-9f4d-d19b5ca38623\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-5c82t"
Nov 03 20:19:27.168804 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:27.150267    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6029825e-e953-4c43-9f4d-d19b5ca38623-istio-token\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-5c82t\" (UID: \"6029825e-e953-4c43-9f4d-d19b5ca38623\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-5c82t"
Nov 03 20:19:27.307444 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:19:27.307349    5570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8bwhz" containerName="extractor"
Nov 03 20:19:27.308014 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:27.307994    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-5c82t"
Nov 03 20:19:28.423809 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:28.410227    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-e80203/distinct-2-w4-e80203-kserve-756f9b9bc6-69gxj"]
Nov 03 20:19:28.501803 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:28.475839    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-e80203/distinct-2-w4-e80203-kserve-756f9b9bc6-69gxj"]
Nov 03 20:19:28.517843 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:28.513874    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-5c82t"]
Nov 03 20:19:28.535615 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:28.483258    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-e80203/distinct-2-w4-e80203-kserve-756f9b9bc6-69gxj"
Nov 03 20:19:28.612947 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:28.610007    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-e80203\"/\"distinct-2-w4-e80203-kserve-self-signed-certs\""
Nov 03 20:19:28.667864 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:28.667706    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwf85\" (UniqueName: \"kubernetes.io/projected/ccf5dc2f-f116-4917-bb1a-b136d8ed17f6-kube-api-access-pwf85\") pod \"distinct-2-w4-e80203-kserve-756f9b9bc6-69gxj\" (UID: \"ccf5dc2f-f116-4917-bb1a-b136d8ed17f6\") " pod="e2e-models-e2e-worker-w4-e80203/distinct-2-w4-e80203-kserve-756f9b9bc6-69gxj"
Nov 03 20:19:28.668309 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:28.668279    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ccf5dc2f-f116-4917-bb1a-b136d8ed17f6-model-cache\") pod \"distinct-2-w4-e80203-kserve-756f9b9bc6-69gxj\" (UID: \"ccf5dc2f-f116-4917-bb1a-b136d8ed17f6\") " pod="e2e-models-e2e-worker-w4-e80203/distinct-2-w4-e80203-kserve-756f9b9bc6-69gxj"
Nov 03 20:19:28.683479 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:28.683387    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ccf5dc2f-f116-4917-bb1a-b136d8ed17f6-tls-certs\") pod \"distinct-2-w4-e80203-kserve-756f9b9bc6-69gxj\" (UID: \"ccf5dc2f-f116-4917-bb1a-b136d8ed17f6\") " pod="e2e-models-e2e-worker-w4-e80203/distinct-2-w4-e80203-kserve-756f9b9bc6-69gxj"
Nov 03 20:19:28.683813 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:28.683760    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ccf5dc2f-f116-4917-bb1a-b136d8ed17f6-dshm\") pod \"distinct-2-w4-e80203-kserve-756f9b9bc6-69gxj\" (UID: \"ccf5dc2f-f116-4917-bb1a-b136d8ed17f6\") " pod="e2e-models-e2e-worker-w4-e80203/distinct-2-w4-e80203-kserve-756f9b9bc6-69gxj"
Nov 03 20:19:28.684146 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:28.684122    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ccf5dc2f-f116-4917-bb1a-b136d8ed17f6-tmp-dir\") pod \"distinct-2-w4-e80203-kserve-756f9b9bc6-69gxj\" (UID: \"ccf5dc2f-f116-4917-bb1a-b136d8ed17f6\") " pod="e2e-models-e2e-worker-w4-e80203/distinct-2-w4-e80203-kserve-756f9b9bc6-69gxj"
Nov 03 20:19:28.684406 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:28.684386    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ccf5dc2f-f116-4917-bb1a-b136d8ed17f6-home\") pod \"distinct-2-w4-e80203-kserve-756f9b9bc6-69gxj\" (UID: \"ccf5dc2f-f116-4917-bb1a-b136d8ed17f6\") " pod="e2e-models-e2e-worker-w4-e80203/distinct-2-w4-e80203-kserve-756f9b9bc6-69gxj"
Nov 03 20:19:28.777458 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:28.777385    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:19:28.781989 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:28.781937    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:19:28.782768 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:28.782733    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:19:28.814227 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:28.813960    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ccf5dc2f-f116-4917-bb1a-b136d8ed17f6-model-cache\") pod \"distinct-2-w4-e80203-kserve-756f9b9bc6-69gxj\" (UID: \"ccf5dc2f-f116-4917-bb1a-b136d8ed17f6\") " pod="e2e-models-e2e-worker-w4-e80203/distinct-2-w4-e80203-kserve-756f9b9bc6-69gxj"
Nov 03 20:19:28.814227 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:28.814036    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ccf5dc2f-f116-4917-bb1a-b136d8ed17f6-tls-certs\") pod \"distinct-2-w4-e80203-kserve-756f9b9bc6-69gxj\" (UID: \"ccf5dc2f-f116-4917-bb1a-b136d8ed17f6\") " pod="e2e-models-e2e-worker-w4-e80203/distinct-2-w4-e80203-kserve-756f9b9bc6-69gxj"
Nov 03 20:19:28.814227 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:28.814191    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ccf5dc2f-f116-4917-bb1a-b136d8ed17f6-dshm\") pod \"distinct-2-w4-e80203-kserve-756f9b9bc6-69gxj\" (UID: \"ccf5dc2f-f116-4917-bb1a-b136d8ed17f6\") " pod="e2e-models-e2e-worker-w4-e80203/distinct-2-w4-e80203-kserve-756f9b9bc6-69gxj"
Nov 03 20:19:28.814516 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:28.814323    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ccf5dc2f-f116-4917-bb1a-b136d8ed17f6-tmp-dir\") pod \"distinct-2-w4-e80203-kserve-756f9b9bc6-69gxj\" (UID: \"ccf5dc2f-f116-4917-bb1a-b136d8ed17f6\") " pod="e2e-models-e2e-worker-w4-e80203/distinct-2-w4-e80203-kserve-756f9b9bc6-69gxj"
Nov 03 20:19:28.814516 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:28.814431    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ccf5dc2f-f116-4917-bb1a-b136d8ed17f6-home\") pod \"distinct-2-w4-e80203-kserve-756f9b9bc6-69gxj\" (UID: \"ccf5dc2f-f116-4917-bb1a-b136d8ed17f6\") " pod="e2e-models-e2e-worker-w4-e80203/distinct-2-w4-e80203-kserve-756f9b9bc6-69gxj"
Nov 03 20:19:28.814627 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:28.814539    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwf85\" (UniqueName: \"kubernetes.io/projected/ccf5dc2f-f116-4917-bb1a-b136d8ed17f6-kube-api-access-pwf85\") pod \"distinct-2-w4-e80203-kserve-756f9b9bc6-69gxj\" (UID: \"ccf5dc2f-f116-4917-bb1a-b136d8ed17f6\") " pod="e2e-models-e2e-worker-w4-e80203/distinct-2-w4-e80203-kserve-756f9b9bc6-69gxj"
Nov 03 20:19:28.832387 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:28.832353    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ccf5dc2f-f116-4917-bb1a-b136d8ed17f6-model-cache\") pod \"distinct-2-w4-e80203-kserve-756f9b9bc6-69gxj\" (UID: \"ccf5dc2f-f116-4917-bb1a-b136d8ed17f6\") " pod="e2e-models-e2e-worker-w4-e80203/distinct-2-w4-e80203-kserve-756f9b9bc6-69gxj"
Nov 03 20:19:28.875801 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:28.871435    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ccf5dc2f-f116-4917-bb1a-b136d8ed17f6-tmp-dir\") pod \"distinct-2-w4-e80203-kserve-756f9b9bc6-69gxj\" (UID: \"ccf5dc2f-f116-4917-bb1a-b136d8ed17f6\") " pod="e2e-models-e2e-worker-w4-e80203/distinct-2-w4-e80203-kserve-756f9b9bc6-69gxj"
Nov 03 20:19:28.907294 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:28.907173    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ccf5dc2f-f116-4917-bb1a-b136d8ed17f6-home\") pod \"distinct-2-w4-e80203-kserve-756f9b9bc6-69gxj\" (UID: \"ccf5dc2f-f116-4917-bb1a-b136d8ed17f6\") " pod="e2e-models-e2e-worker-w4-e80203/distinct-2-w4-e80203-kserve-756f9b9bc6-69gxj"
Nov 03 20:19:28.942904 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:28.942866    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ccf5dc2f-f116-4917-bb1a-b136d8ed17f6-dshm\") pod \"distinct-2-w4-e80203-kserve-756f9b9bc6-69gxj\" (UID: \"ccf5dc2f-f116-4917-bb1a-b136d8ed17f6\") " pod="e2e-models-e2e-worker-w4-e80203/distinct-2-w4-e80203-kserve-756f9b9bc6-69gxj"
Nov 03 20:19:28.963084 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:28.963008    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwf85\" (UniqueName: \"kubernetes.io/projected/ccf5dc2f-f116-4917-bb1a-b136d8ed17f6-kube-api-access-pwf85\") pod \"distinct-2-w4-e80203-kserve-756f9b9bc6-69gxj\" (UID: \"ccf5dc2f-f116-4917-bb1a-b136d8ed17f6\") " pod="e2e-models-e2e-worker-w4-e80203/distinct-2-w4-e80203-kserve-756f9b9bc6-69gxj"
Nov 03 20:19:28.977821 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:28.967904    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ccf5dc2f-f116-4917-bb1a-b136d8ed17f6-tls-certs\") pod \"distinct-2-w4-e80203-kserve-756f9b9bc6-69gxj\" (UID: \"ccf5dc2f-f116-4917-bb1a-b136d8ed17f6\") " pod="e2e-models-e2e-worker-w4-e80203/distinct-2-w4-e80203-kserve-756f9b9bc6-69gxj"
Nov 03 20:19:29.145807 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:29.136162    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh" podUID="2d3647f2-89e9-4908-a8ee-d84e527abeb0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:19:29.165406 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:29.165365    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-e80203/distinct-2-w4-e80203-kserve-756f9b9bc6-69gxj"
Nov 03 20:19:29.638641 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:29.638529    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/premium-w2-9973d9-kserve-5c6dd4669c-5gv8x" podUID="cefc89ad-7ae1-433b-979d-6601678a6d98" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:19:29.701100 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:29.699883    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/premium-w3-482722-kserve-689dc7bd5d-x8rjp" podUID="369045c3-4d6a-49a7-9bc6-f2939883bee8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:19:29.960833 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:29.942992    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/distinct-w3-482722-kserve-d876967c-kd488" podUID="9a97f9cc-2261-471d-9053-51ee965850bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:19:29.960833 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:29.944404    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/premium-w4-e80203-kserve-5f9f445456-bms9d" podUID="6ad76efc-20c2-4951-b22a-c5a39cd39ed4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Nov 03 20:19:29.972455 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:29.971095    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz" podUID="3a715ef1-38e9-4386-9eec-e10e8df11a10" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:19:30.106414 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:30.083405    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/premium-w1-6fc748-kserve-7474db9cd7-nf2xk" podUID="830e062e-83fc-4de7-91a7-5534dac1d46b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:19:30.240850 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:30.240113    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-5c82t" event={"ID":"6029825e-e953-4c43-9f4d-d19b5ca38623","Type":"ContainerStarted","Data":"61df14cbe40ac50825e634951cd7b5b5dbc121f9048dfffa32676dd32719d0b9"}
Nov 03 20:19:30.240850 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:30.240178    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-5c82t" event={"ID":"6029825e-e953-4c43-9f4d-d19b5ca38623","Type":"ContainerStarted","Data":"d039eb5c206efbf9b02b55250cc200257402319e89c62cbae7cb9aa8dd768afa"}
Nov 03 20:19:30.274799 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:30.250504    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-e80203/distinct-2-w4-e80203-kserve-756f9b9bc6-69gxj"]
Nov 03 20:19:30.329800 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:30.324161    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-5c82t"
Nov 03 20:19:30.379839 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:30.364438    5570 patch_prober.go:29] interesting pod/e2e-worker-w3-482722-openshift-default-7645487df-5c82t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Nov 03 20:19:30.379839 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:30.364498    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-5c82t" podUID="6029825e-e953-4c43-9f4d-d19b5ca38623" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Nov 03 20:19:30.418083 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:30.418012    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-5c82t" podStartSLOduration=4.417988288 podStartE2EDuration="4.417988288s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:19:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:19:30.402803779 +0000 UTC m=+904.555465667" watchObservedRunningTime="2026-11-03 20:19:30.417988288 +0000 UTC m=+904.570650177"
Nov 03 20:19:31.222797 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:31.222270    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-e80203/distinct-2-w4-e80203-kserve-756f9b9bc6-69gxj" event={"ID":"ccf5dc2f-f116-4917-bb1a-b136d8ed17f6","Type":"ContainerStarted","Data":"54522cc77363c1530edc0a1e0af7881d460018cab0c8183826ac56ac16a32f79"}
Nov 03 20:19:31.222797 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:31.222322    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-e80203/distinct-2-w4-e80203-kserve-756f9b9bc6-69gxj" event={"ID":"ccf5dc2f-f116-4917-bb1a-b136d8ed17f6","Type":"ContainerStarted","Data":"50bf087bc34fa4520a8a0bb9461a0bea8c2c409be168d1aa2031ece8fc0a95d3"}
Nov 03 20:19:31.279649 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:31.279591    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-e80203/distinct-2-w4-e80203-kserve-756f9b9bc6-69gxj" podStartSLOduration=3.279575657 podStartE2EDuration="3.279575657s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:19:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:19:31.276826659 +0000 UTC m=+905.429488547" watchObservedRunningTime="2026-11-03 20:19:31.279575657 +0000 UTC m=+905.432237543"
Nov 03 20:19:31.310346 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:31.310305    5570 patch_prober.go:29] interesting pod/e2e-worker-w3-482722-openshift-default-7645487df-5c82t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Nov 03 20:19:31.310528 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:31.310368    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-5c82t" podUID="6029825e-e953-4c43-9f4d-d19b5ca38623" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Nov 03 20:19:32.166061 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:32.166027    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-e80203/distinct-2-w4-e80203-kserve-756f9b9bc6-69gxj"
Nov 03 20:19:32.309986 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:32.309947    5570 patch_prober.go:29] interesting pod/e2e-worker-w3-482722-openshift-default-7645487df-5c82t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Nov 03 20:19:32.310420 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:32.310005    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-5c82t" podUID="6029825e-e953-4c43-9f4d-d19b5ca38623" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Nov 03 20:19:33.310021 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:33.309985    5570 patch_prober.go:29] interesting pod/e2e-worker-w3-482722-openshift-default-7645487df-5c82t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Nov 03 20:19:33.310442 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:33.310040    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-5c82t" podUID="6029825e-e953-4c43-9f4d-d19b5ca38623" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Nov 03 20:19:34.309841 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:34.309765    5570 patch_prober.go:29] interesting pod/e2e-worker-w3-482722-openshift-default-7645487df-5c82t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Nov 03 20:19:34.310035 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:34.309858    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-5c82t" podUID="6029825e-e953-4c43-9f4d-d19b5ca38623" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Nov 03 20:19:35.309944 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:35.309907    5570 patch_prober.go:29] interesting pod/e2e-worker-w3-482722-openshift-default-7645487df-5c82t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Nov 03 20:19:35.310362 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:35.309969    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-5c82t" podUID="6029825e-e953-4c43-9f4d-d19b5ca38623" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Nov 03 20:19:36.310104 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:36.310068    5570 patch_prober.go:29] interesting pod/e2e-worker-w3-482722-openshift-default-7645487df-5c82t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Nov 03 20:19:36.310536 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:36.310130    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-5c82t" podUID="6029825e-e953-4c43-9f4d-d19b5ca38623" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Nov 03 20:19:37.315560 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:37.315472    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-5c82t"
Nov 03 20:19:37.315998 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:37.315801    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-5c82t"
Nov 03 20:19:37.316597 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:37.316562    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-5c82t"
Nov 03 20:19:39.166177 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:39.166138    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-e80203/distinct-2-w4-e80203-kserve-756f9b9bc6-69gxj"
Nov 03 20:19:39.180035 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:39.180004    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-e80203/distinct-2-w4-e80203-kserve-756f9b9bc6-69gxj"
Nov 03 20:19:39.322218 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:39.322185    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-e80203/distinct-2-w4-e80203-kserve-756f9b9bc6-69gxj"
Nov 03 20:19:41.295837 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:41.295802    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-e40816/distinct-2-w0-e40816-kserve-79fb46654d-s242x"]
Nov 03 20:19:41.301264 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:41.301237    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e40816/distinct-2-w0-e40816-kserve-79fb46654d-s242x"
Nov 03 20:19:41.323317 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:41.323280    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-e40816/distinct-2-w0-e40816-kserve-79fb46654d-s242x"]
Nov 03 20:19:41.333858 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:41.333821    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-e40816\"/\"distinct-2-w0-e40816-kserve-self-signed-certs\""
Nov 03 20:19:41.421693 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:41.421651    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ad9b048-925c-4d01-a839-a5f25d4dd02d-tmp-dir\") pod \"distinct-2-w0-e40816-kserve-79fb46654d-s242x\" (UID: \"4ad9b048-925c-4d01-a839-a5f25d4dd02d\") " pod="e2e-models-e2e-worker-w0-e40816/distinct-2-w0-e40816-kserve-79fb46654d-s242x"
Nov 03 20:19:41.421896 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:41.421703    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-924nf\" (UniqueName: \"kubernetes.io/projected/4ad9b048-925c-4d01-a839-a5f25d4dd02d-kube-api-access-924nf\") pod \"distinct-2-w0-e40816-kserve-79fb46654d-s242x\" (UID: \"4ad9b048-925c-4d01-a839-a5f25d4dd02d\") " pod="e2e-models-e2e-worker-w0-e40816/distinct-2-w0-e40816-kserve-79fb46654d-s242x"
Nov 03 20:19:41.421896 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:41.421799    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ad9b048-925c-4d01-a839-a5f25d4dd02d-model-cache\") pod \"distinct-2-w0-e40816-kserve-79fb46654d-s242x\" (UID: \"4ad9b048-925c-4d01-a839-a5f25d4dd02d\") " pod="e2e-models-e2e-worker-w0-e40816/distinct-2-w0-e40816-kserve-79fb46654d-s242x"
Nov 03 20:19:41.421896 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:41.421845    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ad9b048-925c-4d01-a839-a5f25d4dd02d-tls-certs\") pod \"distinct-2-w0-e40816-kserve-79fb46654d-s242x\" (UID: \"4ad9b048-925c-4d01-a839-a5f25d4dd02d\") " pod="e2e-models-e2e-worker-w0-e40816/distinct-2-w0-e40816-kserve-79fb46654d-s242x"
Nov 03 20:19:41.422031 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:41.422020    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ad9b048-925c-4d01-a839-a5f25d4dd02d-dshm\") pod \"distinct-2-w0-e40816-kserve-79fb46654d-s242x\" (UID: \"4ad9b048-925c-4d01-a839-a5f25d4dd02d\") " pod="e2e-models-e2e-worker-w0-e40816/distinct-2-w0-e40816-kserve-79fb46654d-s242x"
Nov 03 20:19:41.422078 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:41.422066    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ad9b048-925c-4d01-a839-a5f25d4dd02d-home\") pod \"distinct-2-w0-e40816-kserve-79fb46654d-s242x\" (UID: \"4ad9b048-925c-4d01-a839-a5f25d4dd02d\") " pod="e2e-models-e2e-worker-w0-e40816/distinct-2-w0-e40816-kserve-79fb46654d-s242x"
Nov 03 20:19:41.526384 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:41.526341    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ad9b048-925c-4d01-a839-a5f25d4dd02d-tmp-dir\") pod \"distinct-2-w0-e40816-kserve-79fb46654d-s242x\" (UID: \"4ad9b048-925c-4d01-a839-a5f25d4dd02d\") " pod="e2e-models-e2e-worker-w0-e40816/distinct-2-w0-e40816-kserve-79fb46654d-s242x"
Nov 03 20:19:41.527200 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:41.527176    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-924nf\" (UniqueName: \"kubernetes.io/projected/4ad9b048-925c-4d01-a839-a5f25d4dd02d-kube-api-access-924nf\") pod \"distinct-2-w0-e40816-kserve-79fb46654d-s242x\" (UID: \"4ad9b048-925c-4d01-a839-a5f25d4dd02d\") " pod="e2e-models-e2e-worker-w0-e40816/distinct-2-w0-e40816-kserve-79fb46654d-s242x"
Nov 03 20:19:41.527739 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:41.527714    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ad9b048-925c-4d01-a839-a5f25d4dd02d-model-cache\") pod \"distinct-2-w0-e40816-kserve-79fb46654d-s242x\" (UID: \"4ad9b048-925c-4d01-a839-a5f25d4dd02d\") " pod="e2e-models-e2e-worker-w0-e40816/distinct-2-w0-e40816-kserve-79fb46654d-s242x"
Nov 03 20:19:41.529261 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:41.529191    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ad9b048-925c-4d01-a839-a5f25d4dd02d-tls-certs\") pod \"distinct-2-w0-e40816-kserve-79fb46654d-s242x\" (UID: \"4ad9b048-925c-4d01-a839-a5f25d4dd02d\") " pod="e2e-models-e2e-worker-w0-e40816/distinct-2-w0-e40816-kserve-79fb46654d-s242x"
Nov 03 20:19:41.540816 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:41.535164    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ad9b048-925c-4d01-a839-a5f25d4dd02d-dshm\") pod \"distinct-2-w0-e40816-kserve-79fb46654d-s242x\" (UID: \"4ad9b048-925c-4d01-a839-a5f25d4dd02d\") " pod="e2e-models-e2e-worker-w0-e40816/distinct-2-w0-e40816-kserve-79fb46654d-s242x"
Nov 03 20:19:41.540816 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:41.535258    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ad9b048-925c-4d01-a839-a5f25d4dd02d-home\") pod \"distinct-2-w0-e40816-kserve-79fb46654d-s242x\" (UID: \"4ad9b048-925c-4d01-a839-a5f25d4dd02d\") " pod="e2e-models-e2e-worker-w0-e40816/distinct-2-w0-e40816-kserve-79fb46654d-s242x"
Nov 03 20:19:41.540816 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:41.529111    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ad9b048-925c-4d01-a839-a5f25d4dd02d-model-cache\") pod \"distinct-2-w0-e40816-kserve-79fb46654d-s242x\" (UID: \"4ad9b048-925c-4d01-a839-a5f25d4dd02d\") " pod="e2e-models-e2e-worker-w0-e40816/distinct-2-w0-e40816-kserve-79fb46654d-s242x"
Nov 03 20:19:41.540816 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:41.527091    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ad9b048-925c-4d01-a839-a5f25d4dd02d-tmp-dir\") pod \"distinct-2-w0-e40816-kserve-79fb46654d-s242x\" (UID: \"4ad9b048-925c-4d01-a839-a5f25d4dd02d\") " pod="e2e-models-e2e-worker-w0-e40816/distinct-2-w0-e40816-kserve-79fb46654d-s242x"
Nov 03 20:19:41.544465 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:41.544434    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ad9b048-925c-4d01-a839-a5f25d4dd02d-tls-certs\") pod \"distinct-2-w0-e40816-kserve-79fb46654d-s242x\" (UID: \"4ad9b048-925c-4d01-a839-a5f25d4dd02d\") " pod="e2e-models-e2e-worker-w0-e40816/distinct-2-w0-e40816-kserve-79fb46654d-s242x"
Nov 03 20:19:41.574341 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:41.552098    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ad9b048-925c-4d01-a839-a5f25d4dd02d-home\") pod \"distinct-2-w0-e40816-kserve-79fb46654d-s242x\" (UID: \"4ad9b048-925c-4d01-a839-a5f25d4dd02d\") " pod="e2e-models-e2e-worker-w0-e40816/distinct-2-w0-e40816-kserve-79fb46654d-s242x"
Nov 03 20:19:41.574341 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:41.557236    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ad9b048-925c-4d01-a839-a5f25d4dd02d-dshm\") pod \"distinct-2-w0-e40816-kserve-79fb46654d-s242x\" (UID: \"4ad9b048-925c-4d01-a839-a5f25d4dd02d\") " pod="e2e-models-e2e-worker-w0-e40816/distinct-2-w0-e40816-kserve-79fb46654d-s242x"
Nov 03 20:19:41.574341 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:41.563557    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-924nf\" (UniqueName: \"kubernetes.io/projected/4ad9b048-925c-4d01-a839-a5f25d4dd02d-kube-api-access-924nf\") pod \"distinct-2-w0-e40816-kserve-79fb46654d-s242x\" (UID: \"4ad9b048-925c-4d01-a839-a5f25d4dd02d\") " pod="e2e-models-e2e-worker-w0-e40816/distinct-2-w0-e40816-kserve-79fb46654d-s242x"
Nov 03 20:19:41.625809 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:41.624941    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e40816/distinct-2-w0-e40816-kserve-79fb46654d-s242x"
Nov 03 20:19:41.879706 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:41.879605    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g69f8"]
Nov 03 20:19:42.019911 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:42.019884    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-e40816/distinct-2-w0-e40816-kserve-79fb46654d-s242x"]
Nov 03 20:19:42.022301 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:19:42.022264    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4ad9b048_925c_4d01_a839_a5f25d4dd02d.slice/crio-c0add56f78dc489b025b78d9f763d2c9a70e77b19c0546cd5e4551a48519dd49 WatchSource:0}: Error finding container c0add56f78dc489b025b78d9f763d2c9a70e77b19c0546cd5e4551a48519dd49: Status 404 returned error can't find the container with id c0add56f78dc489b025b78d9f763d2c9a70e77b19c0546cd5e4551a48519dd49
Nov 03 20:19:42.345292 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:42.345249    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e40816/distinct-2-w0-e40816-kserve-79fb46654d-s242x" event={"ID":"4ad9b048-925c-4d01-a839-a5f25d4dd02d","Type":"ContainerStarted","Data":"3b63087f2e199ac26d89e7d01f1f7e0ffe300e43c3dd7858cf7f07376f678516"}
Nov 03 20:19:42.345817 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:42.345310    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e40816/distinct-2-w0-e40816-kserve-79fb46654d-s242x" event={"ID":"4ad9b048-925c-4d01-a839-a5f25d4dd02d","Type":"ContainerStarted","Data":"c0add56f78dc489b025b78d9f763d2c9a70e77b19c0546cd5e4551a48519dd49"}
Nov 03 20:19:42.389584 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:42.389515    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-e40816/distinct-2-w0-e40816-kserve-79fb46654d-s242x" podStartSLOduration=1.389493551 podStartE2EDuration="1.389493551s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:19:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:19:42.385586726 +0000 UTC m=+916.538248612" watchObservedRunningTime="2026-11-03 20:19:42.389493551 +0000 UTC m=+916.542155438"
Nov 03 20:19:42.629400 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:42.629299    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e40816/distinct-2-w0-e40816-kserve-79fb46654d-s242x"
Nov 03 20:19:44.713801 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:44.712936    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/premium-w3-482722-kserve-689dc7bd5d-x8rjp" podUID="369045c3-4d6a-49a7-9bc6-f2939883bee8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:19:44.828799 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:44.828020    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz" podUID="3a715ef1-38e9-4386-9eec-e10e8df11a10" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": context deadline exceeded"
Nov 03 20:19:44.828799 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:44.828251    5570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-9973d9/premium-w2-9973d9-kserve-5c6dd4669c-5gv8x" podUID="cefc89ad-7ae1-433b-979d-6601678a6d98" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:19:44.842009 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:44.835606    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/premium-w4-e80203-kserve-5f9f445456-bms9d" podUID="6ad76efc-20c2-4951-b22a-c5a39cd39ed4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Nov 03 20:19:44.900799 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:44.900178    5570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-482722/distinct-w3-482722-kserve-d876967c-kd488" podUID="9a97f9cc-2261-471d-9053-51ee965850bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:19:44.961387 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:44.961340    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/distinct-w3-482722-kserve-d876967c-kd488" podUID="9a97f9cc-2261-471d-9053-51ee965850bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded"
Nov 03 20:19:45.084867 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:45.069001    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/premium-w1-6fc748-kserve-7474db9cd7-nf2xk" podUID="830e062e-83fc-4de7-91a7-5534dac1d46b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:19:45.095606 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:45.095559    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh" podUID="2d3647f2-89e9-4908-a8ee-d84e527abeb0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Nov 03 20:19:45.202915 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:45.202862    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-2-w6-9e1bcb-kserve-7d96d9f6f6-4f7fl" podUID="56353b0f-08ee-4ada-9dc0-ac5afd5a59c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:19:45.204327 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:45.204291    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/simulator-w0-e40816-kserve-688f79796f-2tvw5" podUID="c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Nov 03 20:19:45.318802 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:45.314903    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/distinct-2-w4-e80203-kserve-756f9b9bc6-69gxj" podUID="ccf5dc2f-f116-4917-bb1a-b136d8ed17f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded"
Nov 03 20:19:45.465795 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:45.459341    5570 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz" podUID="3a715ef1-38e9-4386-9eec-e10e8df11a10" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:19:45.638805 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:45.638413    5570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-482722/premium-w3-482722-kserve-689dc7bd5d-x8rjp" podUID="369045c3-4d6a-49a7-9bc6-f2939883bee8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:19:45.832042 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:45.831920    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz" podUID="3a715ef1-38e9-4386-9eec-e10e8df11a10" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:19:45.833635 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:45.833602    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz"
Nov 03 20:19:46.086110 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:46.085975    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/premium-w1-6fc748-kserve-7474db9cd7-nf2xk" podUID="830e062e-83fc-4de7-91a7-5534dac1d46b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:19:46.091515 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:46.087635    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6fc748/premium-w1-6fc748-kserve-7474db9cd7-nf2xk"
Nov 03 20:19:46.101940 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:46.101893    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh" podUID="2d3647f2-89e9-4908-a8ee-d84e527abeb0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:19:46.103401 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:46.103008    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-9e1bcb/premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh"
Nov 03 20:19:46.134799 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:46.127387    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz"
Nov 03 20:19:46.171800 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:46.164316    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6fc748/premium-w1-6fc748-kserve-7474db9cd7-nf2xk"
Nov 03 20:19:46.255252 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:46.254737    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-9e1bcb/premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh"
Nov 03 20:19:50.359459 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:50.359421    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-482722/distinct-2-w3-482722-kserve-74bf58cf7d-v9cjw"]
Nov 03 20:19:50.364411 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:50.364389    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-482722/distinct-2-w3-482722-kserve-74bf58cf7d-v9cjw"
Nov 03 20:19:50.395678 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:50.384580    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-482722/distinct-2-w3-482722-kserve-74bf58cf7d-v9cjw"]
Nov 03 20:19:50.395678 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:50.392907    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-482722\"/\"distinct-2-w3-482722-kserve-self-signed-certs\""
Nov 03 20:19:50.417328 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:50.417280    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a-home\") pod \"distinct-2-w3-482722-kserve-74bf58cf7d-v9cjw\" (UID: \"9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a\") " pod="e2e-models-e2e-worker-w3-482722/distinct-2-w3-482722-kserve-74bf58cf7d-v9cjw"
Nov 03 20:19:50.417530 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:50.417412    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a-tls-certs\") pod \"distinct-2-w3-482722-kserve-74bf58cf7d-v9cjw\" (UID: \"9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a\") " pod="e2e-models-e2e-worker-w3-482722/distinct-2-w3-482722-kserve-74bf58cf7d-v9cjw"
Nov 03 20:19:50.417530 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:50.417453    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vlf8\" (UniqueName: \"kubernetes.io/projected/9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a-kube-api-access-6vlf8\") pod \"distinct-2-w3-482722-kserve-74bf58cf7d-v9cjw\" (UID: \"9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a\") " pod="e2e-models-e2e-worker-w3-482722/distinct-2-w3-482722-kserve-74bf58cf7d-v9cjw"
Nov 03 20:19:50.417698 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:50.417671    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a-tmp-dir\") pod \"distinct-2-w3-482722-kserve-74bf58cf7d-v9cjw\" (UID: \"9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a\") " pod="e2e-models-e2e-worker-w3-482722/distinct-2-w3-482722-kserve-74bf58cf7d-v9cjw"
Nov 03 20:19:50.417890 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:50.417869    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a-dshm\") pod \"distinct-2-w3-482722-kserve-74bf58cf7d-v9cjw\" (UID: \"9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a\") " pod="e2e-models-e2e-worker-w3-482722/distinct-2-w3-482722-kserve-74bf58cf7d-v9cjw"
Nov 03 20:19:50.417994 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:50.417911    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a-model-cache\") pod \"distinct-2-w3-482722-kserve-74bf58cf7d-v9cjw\" (UID: \"9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a\") " pod="e2e-models-e2e-worker-w3-482722/distinct-2-w3-482722-kserve-74bf58cf7d-v9cjw"
Nov 03 20:19:50.519931 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:50.519877    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a-dshm\") pod \"distinct-2-w3-482722-kserve-74bf58cf7d-v9cjw\" (UID: \"9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a\") " pod="e2e-models-e2e-worker-w3-482722/distinct-2-w3-482722-kserve-74bf58cf7d-v9cjw"
Nov 03 20:19:50.519931 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:50.519933    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a-model-cache\") pod \"distinct-2-w3-482722-kserve-74bf58cf7d-v9cjw\" (UID: \"9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a\") " pod="e2e-models-e2e-worker-w3-482722/distinct-2-w3-482722-kserve-74bf58cf7d-v9cjw"
Nov 03 20:19:50.520181 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:50.520005    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a-home\") pod \"distinct-2-w3-482722-kserve-74bf58cf7d-v9cjw\" (UID: \"9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a\") " pod="e2e-models-e2e-worker-w3-482722/distinct-2-w3-482722-kserve-74bf58cf7d-v9cjw"
Nov 03 20:19:50.520181 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:50.520075    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a-tls-certs\") pod \"distinct-2-w3-482722-kserve-74bf58cf7d-v9cjw\" (UID: \"9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a\") " pod="e2e-models-e2e-worker-w3-482722/distinct-2-w3-482722-kserve-74bf58cf7d-v9cjw"
Nov 03 20:19:50.520181 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:50.520103    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vlf8\" (UniqueName: \"kubernetes.io/projected/9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a-kube-api-access-6vlf8\") pod \"distinct-2-w3-482722-kserve-74bf58cf7d-v9cjw\" (UID: \"9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a\") " pod="e2e-models-e2e-worker-w3-482722/distinct-2-w3-482722-kserve-74bf58cf7d-v9cjw"
Nov 03 20:19:50.520341 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:50.520299    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a-tmp-dir\") pod \"distinct-2-w3-482722-kserve-74bf58cf7d-v9cjw\" (UID: \"9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a\") " pod="e2e-models-e2e-worker-w3-482722/distinct-2-w3-482722-kserve-74bf58cf7d-v9cjw"
Nov 03 20:19:50.520421 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:50.520390    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a-model-cache\") pod \"distinct-2-w3-482722-kserve-74bf58cf7d-v9cjw\" (UID: \"9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a\") " pod="e2e-models-e2e-worker-w3-482722/distinct-2-w3-482722-kserve-74bf58cf7d-v9cjw"
Nov 03 20:19:50.520480 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:50.520462    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a-home\") pod \"distinct-2-w3-482722-kserve-74bf58cf7d-v9cjw\" (UID: \"9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a\") " pod="e2e-models-e2e-worker-w3-482722/distinct-2-w3-482722-kserve-74bf58cf7d-v9cjw"
Nov 03 20:19:50.520597 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:50.520579    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a-tmp-dir\") pod \"distinct-2-w3-482722-kserve-74bf58cf7d-v9cjw\" (UID: \"9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a\") " pod="e2e-models-e2e-worker-w3-482722/distinct-2-w3-482722-kserve-74bf58cf7d-v9cjw"
Nov 03 20:19:50.522233 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:50.522215    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a-dshm\") pod \"distinct-2-w3-482722-kserve-74bf58cf7d-v9cjw\" (UID: \"9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a\") " pod="e2e-models-e2e-worker-w3-482722/distinct-2-w3-482722-kserve-74bf58cf7d-v9cjw"
Nov 03 20:19:50.522504 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:50.522487    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a-tls-certs\") pod \"distinct-2-w3-482722-kserve-74bf58cf7d-v9cjw\" (UID: \"9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a\") " pod="e2e-models-e2e-worker-w3-482722/distinct-2-w3-482722-kserve-74bf58cf7d-v9cjw"
Nov 03 20:19:50.540296 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:50.540260    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vlf8\" (UniqueName: \"kubernetes.io/projected/9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a-kube-api-access-6vlf8\") pod \"distinct-2-w3-482722-kserve-74bf58cf7d-v9cjw\" (UID: \"9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a\") " pod="e2e-models-e2e-worker-w3-482722/distinct-2-w3-482722-kserve-74bf58cf7d-v9cjw"
Nov 03 20:19:50.698859 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:50.687089    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-482722/distinct-2-w3-482722-kserve-74bf58cf7d-v9cjw"
Nov 03 20:19:51.088692 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:51.088660    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-482722/distinct-2-w3-482722-kserve-74bf58cf7d-v9cjw"]
Nov 03 20:19:51.089042 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:19:51.089020    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9d3c8e6e_078d_4b56_be86_b3a5eb4e2d9a.slice/crio-46282515f5ca787be6ccfb24c4a7baef19ea56ef06b6c9004e7952176a40da75 WatchSource:0}: Error finding container 46282515f5ca787be6ccfb24c4a7baef19ea56ef06b6c9004e7952176a40da75: Status 404 returned error can't find the container with id 46282515f5ca787be6ccfb24c4a7baef19ea56ef06b6c9004e7952176a40da75
Nov 03 20:19:51.628977 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:51.628935    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-e40816/distinct-2-w0-e40816-kserve-79fb46654d-s242x"
Nov 03 20:19:51.642548 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:51.642517    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-e40816/distinct-2-w0-e40816-kserve-79fb46654d-s242x"
Nov 03 20:19:51.804274 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:51.804235    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-482722/distinct-2-w3-482722-kserve-74bf58cf7d-v9cjw" event={"ID":"9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a","Type":"ContainerStarted","Data":"75218134286e7df2bb090b7a834fc8bb8a15f1219e7402cd637a818db00e97e1"}
Nov 03 20:19:51.804477 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:51.804284    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-482722/distinct-2-w3-482722-kserve-74bf58cf7d-v9cjw" event={"ID":"9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a","Type":"ContainerStarted","Data":"46282515f5ca787be6ccfb24c4a7baef19ea56ef06b6c9004e7952176a40da75"}
Nov 03 20:19:51.816192 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:51.816156    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e40816/distinct-2-w0-e40816-kserve-79fb46654d-s242x"
Nov 03 20:19:51.847589 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:51.847524    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-482722/distinct-2-w3-482722-kserve-74bf58cf7d-v9cjw" podStartSLOduration=1.847505052 podStartE2EDuration="1.847505052s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:19:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:19:51.847503875 +0000 UTC m=+926.000165774" watchObservedRunningTime="2026-11-03 20:19:51.847505052 +0000 UTC m=+926.000166942"
Nov 03 20:19:52.687760 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:19:52.687722    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-482722/distinct-2-w3-482722-kserve-74bf58cf7d-v9cjw"
Nov 03 20:19:57.229485 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:19:57.229444    5570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8bwhz" containerName="extractor"
Nov 03 20:20:00.687638 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:00.687606    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-482722/distinct-2-w3-482722-kserve-74bf58cf7d-v9cjw"
Nov 03 20:20:00.700541 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:00.700513    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-482722/distinct-2-w3-482722-kserve-74bf58cf7d-v9cjw"
Nov 03 20:20:00.878197 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:00.878156    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-482722/distinct-2-w3-482722-kserve-74bf58cf7d-v9cjw"
Nov 03 20:20:01.559604 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:01.559566    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50725cca-29cpj"]
Nov 03 20:20:01.564070 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:01.564031    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50725cca-29cpj"
Nov 03 20:20:01.589844 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:01.589810    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50725cca-29cpj"]
Nov 03 20:20:01.598255 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:01.598223    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:20:01.598437 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:01.598356    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l7s5w\""
Nov 03 20:20:01.683530 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:01.683492    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c5d552d-f9a3-40f2-8742-95e484f55ef1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-50725cca-29cpj\" (UID: \"4c5d552d-f9a3-40f2-8742-95e484f55ef1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50725cca-29cpj"
Nov 03 20:20:01.683719 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:01.683661    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c5d552d-f9a3-40f2-8742-95e484f55ef1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-50725cca-29cpj\" (UID: \"4c5d552d-f9a3-40f2-8742-95e484f55ef1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50725cca-29cpj"
Nov 03 20:20:01.683855 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:01.683838    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xkpg\" (UniqueName: \"kubernetes.io/projected/4c5d552d-f9a3-40f2-8742-95e484f55ef1-kube-api-access-7xkpg\") pod \"maas-api-revoke-sub-50725cca-29cpj\" (UID: \"4c5d552d-f9a3-40f2-8742-95e484f55ef1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50725cca-29cpj"
Nov 03 20:20:01.785255 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:01.785217    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c5d552d-f9a3-40f2-8742-95e484f55ef1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-50725cca-29cpj\" (UID: \"4c5d552d-f9a3-40f2-8742-95e484f55ef1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50725cca-29cpj"
Nov 03 20:20:01.785699 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:01.785341    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xkpg\" (UniqueName: \"kubernetes.io/projected/4c5d552d-f9a3-40f2-8742-95e484f55ef1-kube-api-access-7xkpg\") pod \"maas-api-revoke-sub-50725cca-29cpj\" (UID: \"4c5d552d-f9a3-40f2-8742-95e484f55ef1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50725cca-29cpj"
Nov 03 20:20:01.785699 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:01.785491    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c5d552d-f9a3-40f2-8742-95e484f55ef1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-50725cca-29cpj\" (UID: \"4c5d552d-f9a3-40f2-8742-95e484f55ef1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50725cca-29cpj"
Nov 03 20:20:01.786042 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:01.786018    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c5d552d-f9a3-40f2-8742-95e484f55ef1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-50725cca-29cpj\" (UID: \"4c5d552d-f9a3-40f2-8742-95e484f55ef1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50725cca-29cpj"
Nov 03 20:20:01.787935 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:01.787912    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c5d552d-f9a3-40f2-8742-95e484f55ef1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-50725cca-29cpj\" (UID: \"4c5d552d-f9a3-40f2-8742-95e484f55ef1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50725cca-29cpj"
Nov 03 20:20:01.853045 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:01.852954    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xkpg\" (UniqueName: \"kubernetes.io/projected/4c5d552d-f9a3-40f2-8742-95e484f55ef1-kube-api-access-7xkpg\") pod \"maas-api-revoke-sub-50725cca-29cpj\" (UID: \"4c5d552d-f9a3-40f2-8742-95e484f55ef1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50725cca-29cpj"
Nov 03 20:20:01.888875 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:01.888840    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50725cca-29cpj"
Nov 03 20:20:02.231395 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:02.231368    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50725cca-29cpj"]
Nov 03 20:20:02.232483 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:20:02.232455    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c5d552d_f9a3_40f2_8742_95e484f55ef1.slice/crio-f1912e362b8a2f966d92182832c31891c1ab2d533620a08e4f1199d1fec45afb WatchSource:0}: Error finding container f1912e362b8a2f966d92182832c31891c1ab2d533620a08e4f1199d1fec45afb: Status 404 returned error can't find the container with id f1912e362b8a2f966d92182832c31891c1ab2d533620a08e4f1199d1fec45afb
Nov 03 20:20:02.915922 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:02.915808    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50725cca-29cpj" event={"ID":"4c5d552d-f9a3-40f2-8742-95e484f55ef1","Type":"ContainerStarted","Data":"3c87f84b45f8993dd0fa9408f5defa5c488f5624343541f2832e37a68a4a4fbe"}
Nov 03 20:20:02.915922 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:02.915879    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50725cca-29cpj" event={"ID":"4c5d552d-f9a3-40f2-8742-95e484f55ef1","Type":"ContainerStarted","Data":"f1912e362b8a2f966d92182832c31891c1ab2d533620a08e4f1199d1fec45afb"}
Nov 03 20:20:02.962432 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:02.962374    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50725cca-29cpj" podStartSLOduration=1.9623556199999999 podStartE2EDuration="1.96235562s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:20:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:20:02.959097073 +0000 UTC m=+937.111758961" watchObservedRunningTime="2026-11-03 20:20:02.96235562 +0000 UTC m=+937.115017507"
Nov 03 20:20:03.924471 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:03.924378    5570 generic.go:356] "Generic (PLEG): container finished" podID="4c5d552d-f9a3-40f2-8742-95e484f55ef1" containerID="3c87f84b45f8993dd0fa9408f5defa5c488f5624343541f2832e37a68a4a4fbe" exitCode=0
Nov 03 20:20:03.924945 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:03.924462    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50725cca-29cpj" event={"ID":"4c5d552d-f9a3-40f2-8742-95e484f55ef1","Type":"ContainerDied","Data":"3c87f84b45f8993dd0fa9408f5defa5c488f5624343541f2832e37a68a4a4fbe"}
Nov 03 20:20:05.445800 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:05.444436    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50725cca-29cpj"
Nov 03 20:20:05.531381 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:05.531288    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c5d552d-f9a3-40f2-8742-95e484f55ef1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c5d552d-f9a3-40f2-8742-95e484f55ef1-maas-api-openshift-ca\") pod \"4c5d552d-f9a3-40f2-8742-95e484f55ef1\" (UID: \"4c5d552d-f9a3-40f2-8742-95e484f55ef1\") "
Nov 03 20:20:05.537238 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:05.537206    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c5d552d-f9a3-40f2-8742-95e484f55ef1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c5d552d-f9a3-40f2-8742-95e484f55ef1-maas-api-xks-ca\") pod \"4c5d552d-f9a3-40f2-8742-95e484f55ef1\" (UID: \"4c5d552d-f9a3-40f2-8742-95e484f55ef1\") "
Nov 03 20:20:05.607168 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:05.607132    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c5d552d-f9a3-40f2-8742-95e484f55ef1-kube-api-access-7xkpg\" (UniqueName: \"kubernetes.io/projected/4c5d552d-f9a3-40f2-8742-95e484f55ef1-kube-api-access-7xkpg\") pod \"4c5d552d-f9a3-40f2-8742-95e484f55ef1\" (UID: \"4c5d552d-f9a3-40f2-8742-95e484f55ef1\") "
Nov 03 20:20:05.613807 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:05.537057    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c5d552d-f9a3-40f2-8742-95e484f55ef1-maas-api-openshift-ca" pod "4c5d552d-f9a3-40f2-8742-95e484f55ef1" (UID: "4c5d552d-f9a3-40f2-8742-95e484f55ef1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:20:05.613807 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:05.605895    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c5d552d-f9a3-40f2-8742-95e484f55ef1-maas-api-xks-ca" pod "4c5d552d-f9a3-40f2-8742-95e484f55ef1" (UID: "4c5d552d-f9a3-40f2-8742-95e484f55ef1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:20:05.629723 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:05.629680    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c5d552d-f9a3-40f2-8742-95e484f55ef1-kube-api-access-7xkpg" pod "4c5d552d-f9a3-40f2-8742-95e484f55ef1" (UID: "4c5d552d-f9a3-40f2-8742-95e484f55ef1"). InnerVolumeSpecName "kube-api-access-7xkpg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:20:05.720858 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:05.715196    5570 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c5d552d-f9a3-40f2-8742-95e484f55ef1-maas-api-xks-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:20:05.721134 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:05.721101    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xkpg\" (UniqueName: \"kubernetes.io/projected/4c5d552d-f9a3-40f2-8742-95e484f55ef1-kube-api-access-7xkpg\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:20:05.721269 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:05.721252    5570 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c5d552d-f9a3-40f2-8742-95e484f55ef1-maas-api-openshift-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:20:06.314000 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:06.296346    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50725cca-29cpj"
Nov 03 20:20:06.598890 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:06.591989    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50725cca-29cpj" event={"ID":"4c5d552d-f9a3-40f2-8742-95e484f55ef1","Type":"ContainerDied","Data":"f1912e362b8a2f966d92182832c31891c1ab2d533620a08e4f1199d1fec45afb"}
Nov 03 20:20:06.598890 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:06.592036    5570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f1912e362b8a2f966d92182832c31891c1ab2d533620a08e4f1199d1fec45afb"
Nov 03 20:20:06.706800 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:06.706014    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/premium-w2-9973d9-kserve-5c6dd4669c-5gv8x" podUID="cefc89ad-7ae1-433b-979d-6601678a6d98" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Nov 03 20:20:06.809848 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:06.808969    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/distinct-2-w0-e40816-kserve-79fb46654d-s242x" podUID="4ad9b048-925c-4d01-a839-a5f25d4dd02d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded"
Nov 03 20:20:06.871720 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:06.871678    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/distinct-2-w3-482722-kserve-74bf58cf7d-v9cjw" podUID="9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:06.944935 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:06.944887    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/distinct-w3-482722-kserve-d876967c-kd488" podUID="9a97f9cc-2261-471d-9053-51ee965850bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:07.242797 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:07.242563    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/simulator-w0-e40816-kserve-688f79796f-2tvw5" podUID="c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Nov 03 20:20:07.279912 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:07.278183    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/premium-w1-6fc748-kserve-7474db9cd7-nf2xk" podUID="830e062e-83fc-4de7-91a7-5534dac1d46b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:07.339807 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:07.312994    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/distinct-2-w4-e80203-kserve-756f9b9bc6-69gxj" podUID="ccf5dc2f-f116-4917-bb1a-b136d8ed17f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded"
Nov 03 20:20:07.437699 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:07.320104    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-2-w6-9e1bcb-kserve-7d96d9f6f6-4f7fl" podUID="56353b0f-08ee-4ada-9dc0-ac5afd5a59c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:07.761549 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:07.700023    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/premium-w3-482722-kserve-689dc7bd5d-x8rjp" podUID="369045c3-4d6a-49a7-9bc6-f2939883bee8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Nov 03 20:20:07.761549 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:07.706172    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/premium-w2-9973d9-kserve-5c6dd4669c-5gv8x" podUID="cefc89ad-7ae1-433b-979d-6601678a6d98" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:07.761549 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:07.706586    5570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-9e1bcb/premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh" podUID="2d3647f2-89e9-4908-a8ee-d84e527abeb0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:07.761549 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:07.758302    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9973d9/premium-w2-9973d9-kserve-5c6dd4669c-5gv8x"
Nov 03 20:20:07.811454 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:07.810389    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/distinct-2-w0-e40816-kserve-79fb46654d-s242x" podUID="4ad9b048-925c-4d01-a839-a5f25d4dd02d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded"
Nov 03 20:20:07.811454 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:07.810834    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e40816/distinct-2-w0-e40816-kserve-79fb46654d-s242x"
Nov 03 20:20:07.876329 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:07.876285    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/distinct-2-w3-482722-kserve-74bf58cf7d-v9cjw" podUID="9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded"
Nov 03 20:20:07.888618 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:07.888587    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-482722/distinct-2-w3-482722-kserve-74bf58cf7d-v9cjw"
Nov 03 20:20:07.963801 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:07.960305    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/distinct-w3-482722-kserve-d876967c-kd488" podUID="9a97f9cc-2261-471d-9053-51ee965850bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:07.979799 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:07.975955    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-482722/distinct-w3-482722-kserve-d876967c-kd488"
Nov 03 20:20:08.140029 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:08.139913    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz" podUID="3a715ef1-38e9-4386-9eec-e10e8df11a10" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:08.261800 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:08.257692    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/simulator-w0-e40816-kserve-688f79796f-2tvw5" podUID="c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:08.261800 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:08.258530    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e40816/simulator-w0-e40816-kserve-688f79796f-2tvw5"
Nov 03 20:20:08.363797 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:08.361023    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh" podUID="2d3647f2-89e9-4908-a8ee-d84e527abeb0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Nov 03 20:20:08.391006 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:08.390898    5570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-e40816/simulator-w0-e40816-kserve-688f79796f-2tvw5" podUID="c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:08.454805 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:08.454551    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/distinct-2-w4-e80203-kserve-756f9b9bc6-69gxj" podUID="ccf5dc2f-f116-4917-bb1a-b136d8ed17f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded"
Nov 03 20:20:08.467252 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:08.455823    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-e80203/distinct-2-w4-e80203-kserve-756f9b9bc6-69gxj"
Nov 03 20:20:08.815801 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:08.739240    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/premium-w4-e80203-kserve-5f9f445456-bms9d" podUID="6ad76efc-20c2-4951-b22a-c5a39cd39ed4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:08.864022 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:08.863965    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/distinct-2-w0-e40816-kserve-79fb46654d-s242x" podUID="4ad9b048-925c-4d01-a839-a5f25d4dd02d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded"
Nov 03 20:20:08.889801 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:08.816521    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-2-w6-9e1bcb-kserve-7d96d9f6f6-4f7fl" podUID="56353b0f-08ee-4ada-9dc0-ac5afd5a59c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:08.964205 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:08.816759    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/premium-w3-482722-kserve-689dc7bd5d-x8rjp" podUID="369045c3-4d6a-49a7-9bc6-f2939883bee8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:08.994800 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:08.977926    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-2-w6-9e1bcb-kserve-7d96d9f6f6-4f7fl"
Nov 03 20:20:08.994800 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:08.978046    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-482722/premium-w3-482722-kserve-689dc7bd5d-x8rjp"
Nov 03 20:20:08.994800 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:08.816205    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/premium-w2-9973d9-kserve-5c6dd4669c-5gv8x" podUID="cefc89ad-7ae1-433b-979d-6601678a6d98" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Nov 03 20:20:08.994800 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:08.890463    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/distinct-2-w3-482722-kserve-74bf58cf7d-v9cjw" podUID="9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded"
Nov 03 20:20:09.010936 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:09.010886    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/distinct-w3-482722-kserve-d876967c-kd488" podUID="9a97f9cc-2261-471d-9053-51ee965850bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:09.160512 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:09.160416    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz" podUID="3a715ef1-38e9-4386-9eec-e10e8df11a10" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:09.267219 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:09.267176    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/premium-w1-6fc748-kserve-7474db9cd7-nf2xk" podUID="830e062e-83fc-4de7-91a7-5534dac1d46b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Nov 03 20:20:09.355723 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:09.300604    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz"
Nov 03 20:20:09.356961 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:09.306031    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/simulator-w0-e40816-kserve-688f79796f-2tvw5" podUID="c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:09.383799 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:09.362093    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh" podUID="2d3647f2-89e9-4908-a8ee-d84e527abeb0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:09.384210 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:09.384186    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-9e1bcb/premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh"
Nov 03 20:20:09.386879 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:09.386404    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-e80203/distinct-2-w4-e80203-kserve-756f9b9bc6-69gxj"
Nov 03 20:20:09.625361 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:09.625328    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-482722/distinct-w3-482722-kserve-d876967c-kd488"
Nov 03 20:20:09.880855 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:09.880070    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/distinct-2-w0-e40816-kserve-79fb46654d-s242x" podUID="4ad9b048-925c-4d01-a839-a5f25d4dd02d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:09.880855 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:09.880709    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/premium-w4-e80203-kserve-5f9f445456-bms9d" podUID="6ad76efc-20c2-4951-b22a-c5a39cd39ed4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:09.915802 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:09.881574    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-e80203/premium-w4-e80203-kserve-5f9f445456-bms9d"
Nov 03 20:20:09.974935 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:09.974884    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/premium-w3-482722-kserve-689dc7bd5d-x8rjp" podUID="369045c3-4d6a-49a7-9bc6-f2939883bee8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:10.004798 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:09.975214    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-2-w6-9e1bcb-kserve-7d96d9f6f6-4f7fl" podUID="56353b0f-08ee-4ada-9dc0-ac5afd5a59c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:10.004798 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:09.982663    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/premium-w2-9973d9-kserve-5c6dd4669c-5gv8x" podUID="cefc89ad-7ae1-433b-979d-6601678a6d98" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:10.004798 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:09.994916    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/distinct-2-w3-482722-kserve-74bf58cf7d-v9cjw" podUID="9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:10.302071 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:10.299878    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz" podUID="3a715ef1-38e9-4386-9eec-e10e8df11a10" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:10.418401 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:10.393480    5570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-e80203/distinct-2-w4-e80203-kserve-756f9b9bc6-69gxj" podUID="ccf5dc2f-f116-4917-bb1a-b136d8ed17f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:10.418401 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:10.393564    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/simulator-w0-e40816-kserve-688f79796f-2tvw5" podUID="c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:10.418401 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:10.410921    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/distinct-2-w4-e80203-kserve-756f9b9bc6-69gxj" podUID="ccf5dc2f-f116-4917-bb1a-b136d8ed17f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded"
Nov 03 20:20:10.418401 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:10.414934    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh" podUID="2d3647f2-89e9-4908-a8ee-d84e527abeb0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:10.515281 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:10.515238    5570 patch_prober.go:29] interesting pod/e2e-worker-w2-9973d9-openshift-default-ffb6475-nz52s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:20:10.517572 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:10.517534    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-nz52s" podUID="974badc1-41ad-461c-8719-e1f3f836ccbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:11.014800 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:10.984881    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/premium-w4-e80203-kserve-5f9f445456-bms9d" podUID="6ad76efc-20c2-4951-b22a-c5a39cd39ed4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:11.033800 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:10.995897    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/distinct-2-w0-e40816-kserve-79fb46654d-s242x" podUID="4ad9b048-925c-4d01-a839-a5f25d4dd02d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded"
Nov 03 20:20:11.050857 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:11.015181    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/distinct-2-w3-482722-kserve-74bf58cf7d-v9cjw" podUID="9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:11.088801 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:11.015234    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/premium-w3-482722-kserve-689dc7bd5d-x8rjp" podUID="369045c3-4d6a-49a7-9bc6-f2939883bee8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Nov 03 20:20:11.100800 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:11.015255    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-2-w6-9e1bcb-kserve-7d96d9f6f6-4f7fl" podUID="56353b0f-08ee-4ada-9dc0-ac5afd5a59c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded"
Nov 03 20:20:11.124907 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:11.015277    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/premium-w2-9973d9-kserve-5c6dd4669c-5gv8x" podUID="cefc89ad-7ae1-433b-979d-6601678a6d98" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:11.125531 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:11.097894    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/distinct-w3-482722-kserve-d876967c-kd488" podUID="9a97f9cc-2261-471d-9053-51ee965850bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:11.174747 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:11.174383    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/premium-w1-6fc748-kserve-7474db9cd7-nf2xk" podUID="830e062e-83fc-4de7-91a7-5534dac1d46b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:11.301154 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:11.301028    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz" podUID="3a715ef1-38e9-4386-9eec-e10e8df11a10" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:11.414398 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:11.414350    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/simulator-w0-e40816-kserve-688f79796f-2tvw5" podUID="c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Nov 03 20:20:11.441422 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:11.441374    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh" podUID="2d3647f2-89e9-4908-a8ee-d84e527abeb0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Nov 03 20:20:11.464793 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:11.443617    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/distinct-2-w4-e80203-kserve-756f9b9bc6-69gxj" podUID="ccf5dc2f-f116-4917-bb1a-b136d8ed17f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:11.482177 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:11.482115    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-e80203/distinct-2-w4-e80203-kserve-756f9b9bc6-69gxj"
Nov 03 20:20:11.724458 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:11.724346    5570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-482722/distinct-2-w3-482722-kserve-74bf58cf7d-v9cjw" podUID="9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/health\": context deadline exceeded"
Nov 03 20:20:12.104851 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:12.079553    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e40816/distinct-2-w0-e40816-kserve-79fb46654d-s242x"
Nov 03 20:20:12.105506 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:12.105465    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/premium-w3-482722-kserve-689dc7bd5d-x8rjp" podUID="369045c3-4d6a-49a7-9bc6-f2939883bee8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:12.119965 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:12.119921    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/distinct-2-w3-482722-kserve-74bf58cf7d-v9cjw" podUID="9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded"
Nov 03 20:20:12.162797 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:12.063046    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/premium-w4-e80203-kserve-5f9f445456-bms9d" podUID="6ad76efc-20c2-4951-b22a-c5a39cd39ed4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:12.162797 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:12.126057    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/premium-w2-9973d9-kserve-5c6dd4669c-5gv8x" podUID="cefc89ad-7ae1-433b-979d-6601678a6d98" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:12.162797 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:12.126559    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/distinct-w3-482722-kserve-d876967c-kd488" podUID="9a97f9cc-2261-471d-9053-51ee965850bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:12.162797 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:12.126892    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-482722/distinct-w3-482722-kserve-d876967c-kd488"
Nov 03 20:20:12.162797 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:12.128198    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-2-w6-9e1bcb-kserve-7d96d9f6f6-4f7fl" podUID="56353b0f-08ee-4ada-9dc0-ac5afd5a59c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:12.187472 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:12.185959    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/premium-w1-6fc748-kserve-7474db9cd7-nf2xk" podUID="830e062e-83fc-4de7-91a7-5534dac1d46b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:12.187472 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:12.186410    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6fc748/premium-w1-6fc748-kserve-7474db9cd7-nf2xk"
Nov 03 20:20:12.312808 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:12.305316    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz" podUID="3a715ef1-38e9-4386-9eec-e10e8df11a10" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": context deadline exceeded"
Nov 03 20:20:12.514061 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:12.349886    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-7458z"]
Nov 03 20:20:12.514276 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:12.438997    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/simulator-w0-e40816-kserve-688f79796f-2tvw5" podUID="c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Nov 03 20:20:12.514355 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:12.455743    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh" podUID="2d3647f2-89e9-4908-a8ee-d84e527abeb0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Nov 03 20:20:12.634949 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:12.485095    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/distinct-2-w4-e80203-kserve-756f9b9bc6-69gxj" podUID="ccf5dc2f-f116-4917-bb1a-b136d8ed17f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:12.644511 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:12.644466    5570 cpu_manager.go:414] "containerMap: removing container" podUID="4c5d552d-f9a3-40f2-8742-95e484f55ef1" containerName="revoke-keys"
Nov 03 20:20:12.644732 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:12.644717    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c5d552d-f9a3-40f2-8742-95e484f55ef1" containerName="revoke-keys"
Nov 03 20:20:12.645962 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:12.645933    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c5d552d-f9a3-40f2-8742-95e484f55ef1" containerName="revoke-keys"
Nov 03 20:20:12.684660 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:12.662823    5570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-e40816/distinct-2-w0-e40816-kserve-79fb46654d-s242x" podUID="4ad9b048-925c-4d01-a839-a5f25d4dd02d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:12.685167 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:12.685139    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-482722/premium-w3-482722-kserve-689dc7bd5d-x8rjp"
Nov 03 20:20:12.685552 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:12.685532    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-7458z"]
Nov 03 20:20:12.685836 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:12.685820    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-482722/distinct-2-w3-482722-kserve-74bf58cf7d-v9cjw"
Nov 03 20:20:12.687471 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:12.687254    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-7458z"
Nov 03 20:20:12.758429 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:12.756365    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c3d9f2d-dcbb-4fb3-a54f-ed9cdef4118e-istio-envoy\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-7458z\" (UID: \"3c3d9f2d-dcbb-4fb3-a54f-ed9cdef4118e\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-7458z"
Nov 03 20:20:12.761518 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:12.761486    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wfct\" (UniqueName: \"kubernetes.io/projected/3c3d9f2d-dcbb-4fb3-a54f-ed9cdef4118e-kube-api-access-8wfct\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-7458z\" (UID: \"3c3d9f2d-dcbb-4fb3-a54f-ed9cdef4118e\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-7458z"
Nov 03 20:20:12.761816 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:12.761767    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c3d9f2d-dcbb-4fb3-a54f-ed9cdef4118e-istio-data\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-7458z\" (UID: \"3c3d9f2d-dcbb-4fb3-a54f-ed9cdef4118e\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-7458z"
Nov 03 20:20:12.761992 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:12.761975    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c3d9f2d-dcbb-4fb3-a54f-ed9cdef4118e-istiod-ca-cert\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-7458z\" (UID: \"3c3d9f2d-dcbb-4fb3-a54f-ed9cdef4118e\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-7458z"
Nov 03 20:20:12.762144 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:12.762126    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c3d9f2d-dcbb-4fb3-a54f-ed9cdef4118e-workload-socket\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-7458z\" (UID: \"3c3d9f2d-dcbb-4fb3-a54f-ed9cdef4118e\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-7458z"
Nov 03 20:20:12.762361 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:12.762339    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c3d9f2d-dcbb-4fb3-a54f-ed9cdef4118e-istio-podinfo\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-7458z\" (UID: \"3c3d9f2d-dcbb-4fb3-a54f-ed9cdef4118e\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-7458z"
Nov 03 20:20:12.762506 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:12.762488    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c3d9f2d-dcbb-4fb3-a54f-ed9cdef4118e-istio-token\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-7458z\" (UID: \"3c3d9f2d-dcbb-4fb3-a54f-ed9cdef4118e\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-7458z"
Nov 03 20:20:12.762710 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:12.762683    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c3d9f2d-dcbb-4fb3-a54f-ed9cdef4118e-workload-certs\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-7458z\" (UID: \"3c3d9f2d-dcbb-4fb3-a54f-ed9cdef4118e\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-7458z"
Nov 03 20:20:12.762895 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:12.762880    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c3d9f2d-dcbb-4fb3-a54f-ed9cdef4118e-credential-socket\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-7458z\" (UID: \"3c3d9f2d-dcbb-4fb3-a54f-ed9cdef4118e\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-7458z"
Nov 03 20:20:12.763195 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:12.757672    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-e80203/distinct-2-w4-e80203-kserve-756f9b9bc6-69gxj"
Nov 03 20:20:12.783117 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:12.783020    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-482722/distinct-w3-482722-kserve-d876967c-kd488"
Nov 03 20:20:12.788926 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:12.788903    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-9e1bcb/premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh"
Nov 03 20:20:12.820230 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:12.789403    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz"
Nov 03 20:20:12.820230 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:12.796928    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6fc748/premium-w1-6fc748-kserve-7474db9cd7-nf2xk"
Nov 03 20:20:12.820230 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:12.818802    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9973d9/premium-w2-9973d9-kserve-5c6dd4669c-5gv8x"
Nov 03 20:20:12.830390 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:12.829588    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e40816/simulator-w0-e40816-kserve-688f79796f-2tvw5"
Nov 03 20:20:12.832895 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:12.832003    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-e80203/premium-w4-e80203-kserve-5f9f445456-bms9d"
Nov 03 20:20:12.833996 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:12.833976    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-2-w6-9e1bcb-kserve-7d96d9f6f6-4f7fl"
Nov 03 20:20:12.868061 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:12.864793    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c3d9f2d-dcbb-4fb3-a54f-ed9cdef4118e-istio-data\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-7458z\" (UID: \"3c3d9f2d-dcbb-4fb3-a54f-ed9cdef4118e\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-7458z"
Nov 03 20:20:12.868061 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:12.864877    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c3d9f2d-dcbb-4fb3-a54f-ed9cdef4118e-istiod-ca-cert\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-7458z\" (UID: \"3c3d9f2d-dcbb-4fb3-a54f-ed9cdef4118e\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-7458z"
Nov 03 20:20:12.868061 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:12.864920    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c3d9f2d-dcbb-4fb3-a54f-ed9cdef4118e-workload-socket\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-7458z\" (UID: \"3c3d9f2d-dcbb-4fb3-a54f-ed9cdef4118e\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-7458z"
Nov 03 20:20:12.868061 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:12.865010    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c3d9f2d-dcbb-4fb3-a54f-ed9cdef4118e-istio-podinfo\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-7458z\" (UID: \"3c3d9f2d-dcbb-4fb3-a54f-ed9cdef4118e\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-7458z"
Nov 03 20:20:12.868061 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:12.865034    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c3d9f2d-dcbb-4fb3-a54f-ed9cdef4118e-istio-token\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-7458z\" (UID: \"3c3d9f2d-dcbb-4fb3-a54f-ed9cdef4118e\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-7458z"
Nov 03 20:20:12.868061 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:12.865134    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c3d9f2d-dcbb-4fb3-a54f-ed9cdef4118e-workload-certs\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-7458z\" (UID: \"3c3d9f2d-dcbb-4fb3-a54f-ed9cdef4118e\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-7458z"
Nov 03 20:20:12.868061 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:12.865193    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c3d9f2d-dcbb-4fb3-a54f-ed9cdef4118e-credential-socket\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-7458z\" (UID: \"3c3d9f2d-dcbb-4fb3-a54f-ed9cdef4118e\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-7458z"
Nov 03 20:20:12.868061 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:12.865398    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c3d9f2d-dcbb-4fb3-a54f-ed9cdef4118e-istio-envoy\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-7458z\" (UID: \"3c3d9f2d-dcbb-4fb3-a54f-ed9cdef4118e\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-7458z"
Nov 03 20:20:12.868061 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:12.865521    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wfct\" (UniqueName: \"kubernetes.io/projected/3c3d9f2d-dcbb-4fb3-a54f-ed9cdef4118e-kube-api-access-8wfct\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-7458z\" (UID: \"3c3d9f2d-dcbb-4fb3-a54f-ed9cdef4118e\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-7458z"
Nov 03 20:20:12.868061 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:12.866864    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c3d9f2d-dcbb-4fb3-a54f-ed9cdef4118e-workload-certs\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-7458z\" (UID: \"3c3d9f2d-dcbb-4fb3-a54f-ed9cdef4118e\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-7458z"
Nov 03 20:20:12.868061 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:12.866930    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c3d9f2d-dcbb-4fb3-a54f-ed9cdef4118e-istio-data\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-7458z\" (UID: \"3c3d9f2d-dcbb-4fb3-a54f-ed9cdef4118e\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-7458z"
Nov 03 20:20:12.868061 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:12.867130    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c3d9f2d-dcbb-4fb3-a54f-ed9cdef4118e-credential-socket\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-7458z\" (UID: \"3c3d9f2d-dcbb-4fb3-a54f-ed9cdef4118e\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-7458z"
Nov 03 20:20:12.868061 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:12.867641    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c3d9f2d-dcbb-4fb3-a54f-ed9cdef4118e-istiod-ca-cert\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-7458z\" (UID: \"3c3d9f2d-dcbb-4fb3-a54f-ed9cdef4118e\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-7458z"
Nov 03 20:20:12.868061 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:12.867968    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c3d9f2d-dcbb-4fb3-a54f-ed9cdef4118e-workload-socket\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-7458z\" (UID: \"3c3d9f2d-dcbb-4fb3-a54f-ed9cdef4118e\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-7458z"
Nov 03 20:20:12.878976 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:12.878942    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c3d9f2d-dcbb-4fb3-a54f-ed9cdef4118e-istio-podinfo\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-7458z\" (UID: \"3c3d9f2d-dcbb-4fb3-a54f-ed9cdef4118e\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-7458z"
Nov 03 20:20:12.891575 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:12.891523    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wfct\" (UniqueName: \"kubernetes.io/projected/3c3d9f2d-dcbb-4fb3-a54f-ed9cdef4118e-kube-api-access-8wfct\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-7458z\" (UID: \"3c3d9f2d-dcbb-4fb3-a54f-ed9cdef4118e\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-7458z"
Nov 03 20:20:12.893457 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:12.893425    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c3d9f2d-dcbb-4fb3-a54f-ed9cdef4118e-istio-envoy\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-7458z\" (UID: \"3c3d9f2d-dcbb-4fb3-a54f-ed9cdef4118e\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-7458z"
Nov 03 20:20:12.898691 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:12.898656    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c3d9f2d-dcbb-4fb3-a54f-ed9cdef4118e-istio-token\") pod \"e2e-worker-w4-e80203-openshift-default-5b6b7d855c-7458z\" (UID: \"3c3d9f2d-dcbb-4fb3-a54f-ed9cdef4118e\") " pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-7458z"
Nov 03 20:20:13.020882 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:13.020848    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-7458z"
Nov 03 20:20:13.539760 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:20:13.539726    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c3d9f2d_dcbb_4fb3_a54f_ed9cdef4118e.slice/crio-5b0f25bdb6d594fa90824eebc1bd7252a9ff48bf3b98c363cb7a49093003939f WatchSource:0}: Error finding container 5b0f25bdb6d594fa90824eebc1bd7252a9ff48bf3b98c363cb7a49093003939f: Status 404 returned error can't find the container with id 5b0f25bdb6d594fa90824eebc1bd7252a9ff48bf3b98c363cb7a49093003939f
Nov 03 20:20:13.540970 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:13.540939    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-7458z"]
Nov 03 20:20:13.542950 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:13.542910    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:20:13.543034 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:13.542999    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:20:13.543076 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:13.543050    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:20:13.911007 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:13.910915    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-7458z" event={"ID":"3c3d9f2d-dcbb-4fb3-a54f-ed9cdef4118e","Type":"ContainerStarted","Data":"5cfb10265f083a5dc0871a30cf040733f11efc2fb138b8ba9e9ecd8ad963fc77"}
Nov 03 20:20:13.911007 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:13.910986    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-7458z" event={"ID":"3c3d9f2d-dcbb-4fb3-a54f-ed9cdef4118e","Type":"ContainerStarted","Data":"5b0f25bdb6d594fa90824eebc1bd7252a9ff48bf3b98c363cb7a49093003939f"}
Nov 03 20:20:13.964154 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:13.963601    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-7458z" podStartSLOduration=1.9635814470000001 podStartE2EDuration="1.963581447s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:20:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:20:13.962702894 +0000 UTC m=+948.115364785" watchObservedRunningTime="2026-11-03 20:20:13.963581447 +0000 UTC m=+948.116243335"
Nov 03 20:20:14.021801 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:14.021756    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-7458z"
Nov 03 20:20:14.023044 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:14.023010    5570 patch_prober.go:29] interesting pod/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-7458z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Nov 03 20:20:14.023205 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:14.023060    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-7458z" podUID="3c3d9f2d-dcbb-4fb3-a54f-ed9cdef4118e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Nov 03 20:20:15.033252 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:15.033193    5570 patch_prober.go:29] interesting pod/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-7458z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Nov 03 20:20:15.033755 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:15.033282    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-7458z" podUID="3c3d9f2d-dcbb-4fb3-a54f-ed9cdef4118e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Nov 03 20:20:16.021889 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:16.021851    5570 patch_prober.go:29] interesting pod/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-7458z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Nov 03 20:20:16.022096 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:16.021927    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-7458z" podUID="3c3d9f2d-dcbb-4fb3-a54f-ed9cdef4118e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Nov 03 20:20:17.032832 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:17.032734    5570 patch_prober.go:29] interesting pod/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-7458z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Nov 03 20:20:17.033366 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:17.032832    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-7458z" podUID="3c3d9f2d-dcbb-4fb3-a54f-ed9cdef4118e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Nov 03 20:20:18.021993 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:18.021953    5570 patch_prober.go:29] interesting pod/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-7458z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Nov 03 20:20:18.022193 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:18.022024    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-7458z" podUID="3c3d9f2d-dcbb-4fb3-a54f-ed9cdef4118e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Nov 03 20:20:18.833248 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:18.833208    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0afe7ff-hlp9w"]
Nov 03 20:20:18.854093 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:18.854059    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0afe7ff-hlp9w"
Nov 03 20:20:18.869378 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:18.869332    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0afe7ff-hlp9w"]
Nov 03 20:20:18.870707 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:18.870654    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:20:18.870895 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:18.870866    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l7s5w\""
Nov 03 20:20:18.915736 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:18.915701    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a935f9f-1413-4063-9622-1147b8b80dbf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d0afe7ff-hlp9w\" (UID: \"0a935f9f-1413-4063-9622-1147b8b80dbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0afe7ff-hlp9w"
Nov 03 20:20:18.915962 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:18.915765    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a935f9f-1413-4063-9622-1147b8b80dbf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d0afe7ff-hlp9w\" (UID: \"0a935f9f-1413-4063-9622-1147b8b80dbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0afe7ff-hlp9w"
Nov 03 20:20:18.916171 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:18.916143    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mj552\" (UniqueName: \"kubernetes.io/projected/0a935f9f-1413-4063-9622-1147b8b80dbf-kube-api-access-mj552\") pod \"maas-api-revoke-sub-d0afe7ff-hlp9w\" (UID: \"0a935f9f-1413-4063-9622-1147b8b80dbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0afe7ff-hlp9w"
Nov 03 20:20:19.017882 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:19.017850    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a935f9f-1413-4063-9622-1147b8b80dbf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d0afe7ff-hlp9w\" (UID: \"0a935f9f-1413-4063-9622-1147b8b80dbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0afe7ff-hlp9w"
Nov 03 20:20:19.018075 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:19.017901    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a935f9f-1413-4063-9622-1147b8b80dbf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d0afe7ff-hlp9w\" (UID: \"0a935f9f-1413-4063-9622-1147b8b80dbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0afe7ff-hlp9w"
Nov 03 20:20:19.018142 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:19.018106    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mj552\" (UniqueName: \"kubernetes.io/projected/0a935f9f-1413-4063-9622-1147b8b80dbf-kube-api-access-mj552\") pod \"maas-api-revoke-sub-d0afe7ff-hlp9w\" (UID: \"0a935f9f-1413-4063-9622-1147b8b80dbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0afe7ff-hlp9w"
Nov 03 20:20:19.018672 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:19.018649    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a935f9f-1413-4063-9622-1147b8b80dbf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d0afe7ff-hlp9w\" (UID: \"0a935f9f-1413-4063-9622-1147b8b80dbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0afe7ff-hlp9w"
Nov 03 20:20:19.020334 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:19.020306    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a935f9f-1413-4063-9622-1147b8b80dbf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d0afe7ff-hlp9w\" (UID: \"0a935f9f-1413-4063-9622-1147b8b80dbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0afe7ff-hlp9w"
Nov 03 20:20:19.021402 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:19.021380    5570 patch_prober.go:29] interesting pod/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-7458z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Nov 03 20:20:19.021457 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:19.021436    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-7458z" podUID="3c3d9f2d-dcbb-4fb3-a54f-ed9cdef4118e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Nov 03 20:20:19.063761 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:19.063718    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mj552\" (UniqueName: \"kubernetes.io/projected/0a935f9f-1413-4063-9622-1147b8b80dbf-kube-api-access-mj552\") pod \"maas-api-revoke-sub-d0afe7ff-hlp9w\" (UID: \"0a935f9f-1413-4063-9622-1147b8b80dbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0afe7ff-hlp9w"
Nov 03 20:20:19.185050 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:19.184962    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0afe7ff-hlp9w"
Nov 03 20:20:19.516304 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:19.516275    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0afe7ff-hlp9w"]
Nov 03 20:20:19.517194 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:20:19.517171    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a935f9f_1413_4063_9622_1147b8b80dbf.slice/crio-0b9607f7ad2a5df339115743ca94640d8d160d4b3f284c74c111ce90555fb39c WatchSource:0}: Error finding container 0b9607f7ad2a5df339115743ca94640d8d160d4b3f284c74c111ce90555fb39c: Status 404 returned error can't find the container with id 0b9607f7ad2a5df339115743ca94640d8d160d4b3f284c74c111ce90555fb39c
Nov 03 20:20:20.021029 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:20.020985    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0afe7ff-hlp9w" event={"ID":"0a935f9f-1413-4063-9622-1147b8b80dbf","Type":"ContainerStarted","Data":"3417a92520cda35689055bd3a5c9c034576455ff70c4a138c5f6fd0f3141d5f7"}
Nov 03 20:20:20.021494 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:20.021063    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0afe7ff-hlp9w" event={"ID":"0a935f9f-1413-4063-9622-1147b8b80dbf","Type":"ContainerStarted","Data":"0b9607f7ad2a5df339115743ca94640d8d160d4b3f284c74c111ce90555fb39c"}
Nov 03 20:20:20.021545 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:20.021514    5570 patch_prober.go:29] interesting pod/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-7458z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Nov 03 20:20:20.021614 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:20.021582    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-7458z" podUID="3c3d9f2d-dcbb-4fb3-a54f-ed9cdef4118e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Nov 03 20:20:20.065744 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:20.065641    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0afe7ff-hlp9w" podStartSLOduration=2.065625553 podStartE2EDuration="2.065625553s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:20:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:20:20.06306247 +0000 UTC m=+954.215724357" watchObservedRunningTime="2026-11-03 20:20:20.065625553 +0000 UTC m=+954.218287493"
Nov 03 20:20:21.032745 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:21.032710    5570 generic.go:356] "Generic (PLEG): container finished" podID="0a935f9f-1413-4063-9622-1147b8b80dbf" containerID="3417a92520cda35689055bd3a5c9c034576455ff70c4a138c5f6fd0f3141d5f7" exitCode=0
Nov 03 20:20:21.033155 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:21.032946    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0afe7ff-hlp9w" event={"ID":"0a935f9f-1413-4063-9622-1147b8b80dbf","Type":"ContainerDied","Data":"3417a92520cda35689055bd3a5c9c034576455ff70c4a138c5f6fd0f3141d5f7"}
Nov 03 20:20:21.383715 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:21.383621    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-7458z"
Nov 03 20:20:21.384154 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:21.384130    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-7458z"
Nov 03 20:20:21.385264 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:21.385235    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-7458z"
Nov 03 20:20:22.264225 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:22.264197    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0afe7ff-hlp9w"
Nov 03 20:20:22.399256 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:22.399155    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a935f9f-1413-4063-9622-1147b8b80dbf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a935f9f-1413-4063-9622-1147b8b80dbf-maas-api-xks-ca\") pod \"0a935f9f-1413-4063-9622-1147b8b80dbf\" (UID: \"0a935f9f-1413-4063-9622-1147b8b80dbf\") "
Nov 03 20:20:22.399446 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:22.399410    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a935f9f-1413-4063-9622-1147b8b80dbf-kube-api-access-mj552\" (UniqueName: \"kubernetes.io/projected/0a935f9f-1413-4063-9622-1147b8b80dbf-kube-api-access-mj552\") pod \"0a935f9f-1413-4063-9622-1147b8b80dbf\" (UID: \"0a935f9f-1413-4063-9622-1147b8b80dbf\") "
Nov 03 20:20:22.399523 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:22.399491    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a935f9f-1413-4063-9622-1147b8b80dbf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a935f9f-1413-4063-9622-1147b8b80dbf-maas-api-openshift-ca\") pod \"0a935f9f-1413-4063-9622-1147b8b80dbf\" (UID: \"0a935f9f-1413-4063-9622-1147b8b80dbf\") "
Nov 03 20:20:22.399954 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:22.399924    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a935f9f-1413-4063-9622-1147b8b80dbf-maas-api-openshift-ca" pod "0a935f9f-1413-4063-9622-1147b8b80dbf" (UID: "0a935f9f-1413-4063-9622-1147b8b80dbf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:20:22.400363 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:22.400335    5570 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a935f9f-1413-4063-9622-1147b8b80dbf-maas-api-openshift-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:20:22.401898 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:22.401855    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a935f9f-1413-4063-9622-1147b8b80dbf-maas-api-xks-ca" pod "0a935f9f-1413-4063-9622-1147b8b80dbf" (UID: "0a935f9f-1413-4063-9622-1147b8b80dbf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:20:22.402234 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:22.402209    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a935f9f-1413-4063-9622-1147b8b80dbf-kube-api-access-mj552" pod "0a935f9f-1413-4063-9622-1147b8b80dbf" (UID: "0a935f9f-1413-4063-9622-1147b8b80dbf"). InnerVolumeSpecName "kube-api-access-mj552". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:20:22.502163 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:22.502119    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mj552\" (UniqueName: \"kubernetes.io/projected/0a935f9f-1413-4063-9622-1147b8b80dbf-kube-api-access-mj552\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:20:22.502163 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:22.502158    5570 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a935f9f-1413-4063-9622-1147b8b80dbf-maas-api-xks-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:20:23.061628 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:23.061593    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0afe7ff-hlp9w"
Nov 03 20:20:23.061628 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:23.061621    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0afe7ff-hlp9w" event={"ID":"0a935f9f-1413-4063-9622-1147b8b80dbf","Type":"ContainerDied","Data":"0b9607f7ad2a5df339115743ca94640d8d160d4b3f284c74c111ce90555fb39c"}
Nov 03 20:20:23.061962 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:23.061671    5570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0b9607f7ad2a5df339115743ca94640d8d160d4b3f284c74c111ce90555fb39c"
Nov 03 20:20:26.085426 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:26.085388    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-mt7dd"]
Nov 03 20:20:26.086177 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:26.086146    5570 cpu_manager.go:414] "containerMap: removing container" podUID="0a935f9f-1413-4063-9622-1147b8b80dbf" containerName="revoke-keys"
Nov 03 20:20:26.086177 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:26.086167    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a935f9f-1413-4063-9622-1147b8b80dbf" containerName="revoke-keys"
Nov 03 20:20:26.086343 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:26.086297    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a935f9f-1413-4063-9622-1147b8b80dbf" containerName="revoke-keys"
Nov 03 20:20:26.100057 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:26.100026    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-mt7dd"
Nov 03 20:20:26.116749 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:26.116722    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-mt7dd"]
Nov 03 20:20:26.119066 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:26.119037    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-xcxnz"]
Nov 03 20:20:26.147032 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:26.146949    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-xcxnz"
Nov 03 20:20:26.150757 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:26.148609    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-xcxnz"]
Nov 03 20:20:26.182353 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:26.182315    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55945440-d40e-4235-8b84-d1e2a013372d-credential-socket\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-mt7dd\" (UID: \"55945440-d40e-4235-8b84-d1e2a013372d\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-mt7dd"
Nov 03 20:20:26.182487 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:26.182386    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55945440-d40e-4235-8b84-d1e2a013372d-istio-podinfo\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-mt7dd\" (UID: \"55945440-d40e-4235-8b84-d1e2a013372d\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-mt7dd"
Nov 03 20:20:26.182548 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:26.182501    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blxqs\" (UniqueName: \"kubernetes.io/projected/55945440-d40e-4235-8b84-d1e2a013372d-kube-api-access-blxqs\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-mt7dd\" (UID: \"55945440-d40e-4235-8b84-d1e2a013372d\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-mt7dd"
Nov 03 20:20:26.182602 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:26.182546    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55945440-d40e-4235-8b84-d1e2a013372d-workload-socket\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-mt7dd\" (UID: \"55945440-d40e-4235-8b84-d1e2a013372d\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-mt7dd"
Nov 03 20:20:26.182689 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:26.182673    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55945440-d40e-4235-8b84-d1e2a013372d-workload-certs\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-mt7dd\" (UID: \"55945440-d40e-4235-8b84-d1e2a013372d\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-mt7dd"
Nov 03 20:20:26.182748 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:26.182734    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55945440-d40e-4235-8b84-d1e2a013372d-istio-data\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-mt7dd\" (UID: \"55945440-d40e-4235-8b84-d1e2a013372d\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-mt7dd"
Nov 03 20:20:26.182894 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:26.182876    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55945440-d40e-4235-8b84-d1e2a013372d-istio-envoy\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-mt7dd\" (UID: \"55945440-d40e-4235-8b84-d1e2a013372d\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-mt7dd"
Nov 03 20:20:26.182938 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:26.182918    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55945440-d40e-4235-8b84-d1e2a013372d-istiod-ca-cert\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-mt7dd\" (UID: \"55945440-d40e-4235-8b84-d1e2a013372d\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-mt7dd"
Nov 03 20:20:26.182990 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:26.182981    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55945440-d40e-4235-8b84-d1e2a013372d-istio-token\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-mt7dd\" (UID: \"55945440-d40e-4235-8b84-d1e2a013372d\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-mt7dd"
Nov 03 20:20:26.284309 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:26.284270    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63727774-9118-4b83-bede-04208e8d59b9-istio-token\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-xcxnz\" (UID: \"63727774-9118-4b83-bede-04208e8d59b9\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-xcxnz"
Nov 03 20:20:26.284482 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:26.284320    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55945440-d40e-4235-8b84-d1e2a013372d-workload-socket\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-mt7dd\" (UID: \"55945440-d40e-4235-8b84-d1e2a013372d\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-mt7dd"
Nov 03 20:20:26.284482 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:26.284412    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63727774-9118-4b83-bede-04208e8d59b9-istio-podinfo\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-xcxnz\" (UID: \"63727774-9118-4b83-bede-04208e8d59b9\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-xcxnz"
Nov 03 20:20:26.284556 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:26.284478    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63727774-9118-4b83-bede-04208e8d59b9-istio-data\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-xcxnz\" (UID: \"63727774-9118-4b83-bede-04208e8d59b9\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-xcxnz"
Nov 03 20:20:26.284587 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:26.284572    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55945440-d40e-4235-8b84-d1e2a013372d-workload-certs\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-mt7dd\" (UID: \"55945440-d40e-4235-8b84-d1e2a013372d\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-mt7dd"
Nov 03 20:20:26.284621 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:26.284610    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63727774-9118-4b83-bede-04208e8d59b9-istio-envoy\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-xcxnz\" (UID: \"63727774-9118-4b83-bede-04208e8d59b9\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-xcxnz"
Nov 03 20:20:26.284698 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:26.284651    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55945440-d40e-4235-8b84-d1e2a013372d-istio-data\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-mt7dd\" (UID: \"55945440-d40e-4235-8b84-d1e2a013372d\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-mt7dd"
Nov 03 20:20:26.284737 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:26.284708    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63727774-9118-4b83-bede-04208e8d59b9-workload-certs\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-xcxnz\" (UID: \"63727774-9118-4b83-bede-04208e8d59b9\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-xcxnz"
Nov 03 20:20:26.284815 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:26.284794    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55945440-d40e-4235-8b84-d1e2a013372d-workload-socket\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-mt7dd\" (UID: \"55945440-d40e-4235-8b84-d1e2a013372d\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-mt7dd"
Nov 03 20:20:26.284890 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:26.284871    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55945440-d40e-4235-8b84-d1e2a013372d-workload-certs\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-mt7dd\" (UID: \"55945440-d40e-4235-8b84-d1e2a013372d\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-mt7dd"
Nov 03 20:20:26.284929 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:26.284879    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55945440-d40e-4235-8b84-d1e2a013372d-istio-envoy\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-mt7dd\" (UID: \"55945440-d40e-4235-8b84-d1e2a013372d\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-mt7dd"
Nov 03 20:20:26.284971 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:26.284945    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55945440-d40e-4235-8b84-d1e2a013372d-istiod-ca-cert\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-mt7dd\" (UID: \"55945440-d40e-4235-8b84-d1e2a013372d\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-mt7dd"
Nov 03 20:20:26.285014 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:26.284989    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63727774-9118-4b83-bede-04208e8d59b9-istiod-ca-cert\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-xcxnz\" (UID: \"63727774-9118-4b83-bede-04208e8d59b9\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-xcxnz"
Nov 03 20:20:26.285066 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:26.285014    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55945440-d40e-4235-8b84-d1e2a013372d-istio-token\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-mt7dd\" (UID: \"55945440-d40e-4235-8b84-d1e2a013372d\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-mt7dd"
Nov 03 20:20:26.285066 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:26.285022    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55945440-d40e-4235-8b84-d1e2a013372d-istio-data\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-mt7dd\" (UID: \"55945440-d40e-4235-8b84-d1e2a013372d\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-mt7dd"
Nov 03 20:20:26.285170 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:26.285089    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63727774-9118-4b83-bede-04208e8d59b9-credential-socket\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-xcxnz\" (UID: \"63727774-9118-4b83-bede-04208e8d59b9\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-xcxnz"
Nov 03 20:20:26.285293 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:26.285262    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55945440-d40e-4235-8b84-d1e2a013372d-credential-socket\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-mt7dd\" (UID: \"55945440-d40e-4235-8b84-d1e2a013372d\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-mt7dd"
Nov 03 20:20:26.285426 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:26.285319    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-858bb\" (UniqueName: \"kubernetes.io/projected/63727774-9118-4b83-bede-04208e8d59b9-kube-api-access-858bb\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-xcxnz\" (UID: \"63727774-9118-4b83-bede-04208e8d59b9\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-xcxnz"
Nov 03 20:20:26.285518 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:26.285417    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55945440-d40e-4235-8b84-d1e2a013372d-istio-podinfo\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-mt7dd\" (UID: \"55945440-d40e-4235-8b84-d1e2a013372d\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-mt7dd"
Nov 03 20:20:26.285607 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:26.285591    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blxqs\" (UniqueName: \"kubernetes.io/projected/55945440-d40e-4235-8b84-d1e2a013372d-kube-api-access-blxqs\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-mt7dd\" (UID: \"55945440-d40e-4235-8b84-d1e2a013372d\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-mt7dd"
Nov 03 20:20:26.285675 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:26.285619    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63727774-9118-4b83-bede-04208e8d59b9-workload-socket\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-xcxnz\" (UID: \"63727774-9118-4b83-bede-04208e8d59b9\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-xcxnz"
Nov 03 20:20:26.285986 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:26.285956    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55945440-d40e-4235-8b84-d1e2a013372d-istiod-ca-cert\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-mt7dd\" (UID: \"55945440-d40e-4235-8b84-d1e2a013372d\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-mt7dd"
Nov 03 20:20:26.286264 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:26.286248    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55945440-d40e-4235-8b84-d1e2a013372d-credential-socket\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-mt7dd\" (UID: \"55945440-d40e-4235-8b84-d1e2a013372d\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-mt7dd"
Nov 03 20:20:26.287349 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:26.287330    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55945440-d40e-4235-8b84-d1e2a013372d-istio-envoy\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-mt7dd\" (UID: \"55945440-d40e-4235-8b84-d1e2a013372d\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-mt7dd"
Nov 03 20:20:26.288005 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:26.287980    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55945440-d40e-4235-8b84-d1e2a013372d-istio-podinfo\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-mt7dd\" (UID: \"55945440-d40e-4235-8b84-d1e2a013372d\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-mt7dd"
Nov 03 20:20:26.305303 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:26.305276    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55945440-d40e-4235-8b84-d1e2a013372d-istio-token\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-mt7dd\" (UID: \"55945440-d40e-4235-8b84-d1e2a013372d\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-mt7dd"
Nov 03 20:20:26.305458 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:26.305359    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-blxqs\" (UniqueName: \"kubernetes.io/projected/55945440-d40e-4235-8b84-d1e2a013372d-kube-api-access-blxqs\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-mt7dd\" (UID: \"55945440-d40e-4235-8b84-d1e2a013372d\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-mt7dd"
Nov 03 20:20:26.386823 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:26.386724    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63727774-9118-4b83-bede-04208e8d59b9-workload-certs\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-xcxnz\" (UID: \"63727774-9118-4b83-bede-04208e8d59b9\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-xcxnz"
Nov 03 20:20:26.386993 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:26.386885    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63727774-9118-4b83-bede-04208e8d59b9-istiod-ca-cert\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-xcxnz\" (UID: \"63727774-9118-4b83-bede-04208e8d59b9\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-xcxnz"
Nov 03 20:20:26.386993 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:26.386922    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63727774-9118-4b83-bede-04208e8d59b9-credential-socket\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-xcxnz\" (UID: \"63727774-9118-4b83-bede-04208e8d59b9\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-xcxnz"
Nov 03 20:20:26.387119 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:26.387011    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-858bb\" (UniqueName: \"kubernetes.io/projected/63727774-9118-4b83-bede-04208e8d59b9-kube-api-access-858bb\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-xcxnz\" (UID: \"63727774-9118-4b83-bede-04208e8d59b9\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-xcxnz"
Nov 03 20:20:26.387119 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:26.387097    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63727774-9118-4b83-bede-04208e8d59b9-workload-socket\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-xcxnz\" (UID: \"63727774-9118-4b83-bede-04208e8d59b9\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-xcxnz"
Nov 03 20:20:26.387224 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:26.387139    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63727774-9118-4b83-bede-04208e8d59b9-istio-token\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-xcxnz\" (UID: \"63727774-9118-4b83-bede-04208e8d59b9\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-xcxnz"
Nov 03 20:20:26.387224 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:26.387161    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63727774-9118-4b83-bede-04208e8d59b9-workload-certs\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-xcxnz\" (UID: \"63727774-9118-4b83-bede-04208e8d59b9\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-xcxnz"
Nov 03 20:20:26.387224 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:26.387188    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63727774-9118-4b83-bede-04208e8d59b9-istio-podinfo\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-xcxnz\" (UID: \"63727774-9118-4b83-bede-04208e8d59b9\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-xcxnz"
Nov 03 20:20:26.387379 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:26.387231    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63727774-9118-4b83-bede-04208e8d59b9-istio-data\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-xcxnz\" (UID: \"63727774-9118-4b83-bede-04208e8d59b9\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-xcxnz"
Nov 03 20:20:26.387379 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:26.387263    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63727774-9118-4b83-bede-04208e8d59b9-istio-envoy\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-xcxnz\" (UID: \"63727774-9118-4b83-bede-04208e8d59b9\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-xcxnz"
Nov 03 20:20:26.387482 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:26.387437    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63727774-9118-4b83-bede-04208e8d59b9-workload-socket\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-xcxnz\" (UID: \"63727774-9118-4b83-bede-04208e8d59b9\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-xcxnz"
Nov 03 20:20:26.387660 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:26.387630    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63727774-9118-4b83-bede-04208e8d59b9-credential-socket\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-xcxnz\" (UID: \"63727774-9118-4b83-bede-04208e8d59b9\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-xcxnz"
Nov 03 20:20:26.387870 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:26.387834    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63727774-9118-4b83-bede-04208e8d59b9-istiod-ca-cert\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-xcxnz\" (UID: \"63727774-9118-4b83-bede-04208e8d59b9\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-xcxnz"
Nov 03 20:20:26.388017 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:26.387995    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63727774-9118-4b83-bede-04208e8d59b9-istio-data\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-xcxnz\" (UID: \"63727774-9118-4b83-bede-04208e8d59b9\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-xcxnz"
Nov 03 20:20:26.389529 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:26.389507    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63727774-9118-4b83-bede-04208e8d59b9-istio-envoy\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-xcxnz\" (UID: \"63727774-9118-4b83-bede-04208e8d59b9\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-xcxnz"
Nov 03 20:20:26.389734 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:26.389716    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63727774-9118-4b83-bede-04208e8d59b9-istio-podinfo\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-xcxnz\" (UID: \"63727774-9118-4b83-bede-04208e8d59b9\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-xcxnz"
Nov 03 20:20:26.408236 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:26.408210    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63727774-9118-4b83-bede-04208e8d59b9-istio-token\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-xcxnz\" (UID: \"63727774-9118-4b83-bede-04208e8d59b9\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-xcxnz"
Nov 03 20:20:26.408342 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:26.408321    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-858bb\" (UniqueName: \"kubernetes.io/projected/63727774-9118-4b83-bede-04208e8d59b9-kube-api-access-858bb\") pod \"e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-xcxnz\" (UID: \"63727774-9118-4b83-bede-04208e8d59b9\") " pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-xcxnz"
Nov 03 20:20:26.416735 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:26.416717    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-mt7dd"
Nov 03 20:20:26.465406 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:26.465367    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-xcxnz"
Nov 03 20:20:26.541135 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:26.541092    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-c98nc"]
Nov 03 20:20:26.547847 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:26.547803    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-c98nc"
Nov 03 20:20:26.574144 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:26.574115    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-c98nc"]
Nov 03 20:20:26.715336 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:26.715304    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsbh2\" (UniqueName: \"kubernetes.io/projected/755f6eab-5d03-4368-9670-b1402084474c-kube-api-access-bsbh2\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-c98nc\" (UID: \"755f6eab-5d03-4368-9670-b1402084474c\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-c98nc"
Nov 03 20:20:26.715538 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:26.715422    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/755f6eab-5d03-4368-9670-b1402084474c-istio-podinfo\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-c98nc\" (UID: \"755f6eab-5d03-4368-9670-b1402084474c\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-c98nc"
Nov 03 20:20:26.715538 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:26.715449    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/755f6eab-5d03-4368-9670-b1402084474c-istio-envoy\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-c98nc\" (UID: \"755f6eab-5d03-4368-9670-b1402084474c\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-c98nc"
Nov 03 20:20:26.715622 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:26.715548    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/755f6eab-5d03-4368-9670-b1402084474c-istiod-ca-cert\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-c98nc\" (UID: \"755f6eab-5d03-4368-9670-b1402084474c\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-c98nc"
Nov 03 20:20:26.715622 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:26.715579    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/755f6eab-5d03-4368-9670-b1402084474c-workload-socket\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-c98nc\" (UID: \"755f6eab-5d03-4368-9670-b1402084474c\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-c98nc"
Nov 03 20:20:26.715622 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:26.715611    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/755f6eab-5d03-4368-9670-b1402084474c-istio-token\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-c98nc\" (UID: \"755f6eab-5d03-4368-9670-b1402084474c\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-c98nc"
Nov 03 20:20:26.715717 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:26.715704    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/755f6eab-5d03-4368-9670-b1402084474c-workload-certs\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-c98nc\" (UID: \"755f6eab-5d03-4368-9670-b1402084474c\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-c98nc"
Nov 03 20:20:26.715758 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:26.715726    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/755f6eab-5d03-4368-9670-b1402084474c-credential-socket\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-c98nc\" (UID: \"755f6eab-5d03-4368-9670-b1402084474c\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-c98nc"
Nov 03 20:20:26.715843 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:26.715831    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/755f6eab-5d03-4368-9670-b1402084474c-istio-data\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-c98nc\" (UID: \"755f6eab-5d03-4368-9670-b1402084474c\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-c98nc"
Nov 03 20:20:26.735842 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:20:26.735769    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55945440_d40e_4235_8b84_d1e2a013372d.slice/crio-8b1e9f15e462251ff6d98491599b0a0ed6b83cbdf0f9ba718552b9a2c23c3f79 WatchSource:0}: Error finding container 8b1e9f15e462251ff6d98491599b0a0ed6b83cbdf0f9ba718552b9a2c23c3f79: Status 404 returned error can't find the container with id 8b1e9f15e462251ff6d98491599b0a0ed6b83cbdf0f9ba718552b9a2c23c3f79
Nov 03 20:20:26.735958 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:26.735857    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-mt7dd"]
Nov 03 20:20:26.738525 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:26.738491    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:20:26.738606 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:26.738553    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:20:26.738606 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:26.738581    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:20:26.808718 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:26.808691    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-xcxnz"]
Nov 03 20:20:26.813005 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:20:26.812975    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63727774_9118_4b83_bede_04208e8d59b9.slice/crio-0e6626c5f047215d8796f5a752597e077e75813ee563b18ff17cc55aac9cd69c WatchSource:0}: Error finding container 0e6626c5f047215d8796f5a752597e077e75813ee563b18ff17cc55aac9cd69c: Status 404 returned error can't find the container with id 0e6626c5f047215d8796f5a752597e077e75813ee563b18ff17cc55aac9cd69c
Nov 03 20:20:26.815677 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:26.815645    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:20:26.815761 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:26.815721    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:20:26.815761 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:26.815752    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:20:26.816819 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:26.816796    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/755f6eab-5d03-4368-9670-b1402084474c-istio-podinfo\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-c98nc\" (UID: \"755f6eab-5d03-4368-9670-b1402084474c\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-c98nc"
Nov 03 20:20:26.816898 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:26.816832    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/755f6eab-5d03-4368-9670-b1402084474c-istio-envoy\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-c98nc\" (UID: \"755f6eab-5d03-4368-9670-b1402084474c\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-c98nc"
Nov 03 20:20:26.816898 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:26.816884    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/755f6eab-5d03-4368-9670-b1402084474c-istiod-ca-cert\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-c98nc\" (UID: \"755f6eab-5d03-4368-9670-b1402084474c\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-c98nc"
Nov 03 20:20:26.817003 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:26.816910    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/755f6eab-5d03-4368-9670-b1402084474c-workload-socket\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-c98nc\" (UID: \"755f6eab-5d03-4368-9670-b1402084474c\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-c98nc"
Nov 03 20:20:26.817003 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:26.816943    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/755f6eab-5d03-4368-9670-b1402084474c-istio-token\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-c98nc\" (UID: \"755f6eab-5d03-4368-9670-b1402084474c\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-c98nc"
Nov 03 20:20:26.817003 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:26.816988    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/755f6eab-5d03-4368-9670-b1402084474c-workload-certs\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-c98nc\" (UID: \"755f6eab-5d03-4368-9670-b1402084474c\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-c98nc"
Nov 03 20:20:26.817157 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:26.817007    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/755f6eab-5d03-4368-9670-b1402084474c-credential-socket\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-c98nc\" (UID: \"755f6eab-5d03-4368-9670-b1402084474c\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-c98nc"
Nov 03 20:20:26.817157 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:26.817072    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/755f6eab-5d03-4368-9670-b1402084474c-istio-data\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-c98nc\" (UID: \"755f6eab-5d03-4368-9670-b1402084474c\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-c98nc"
Nov 03 20:20:26.817157 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:26.817148    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsbh2\" (UniqueName: \"kubernetes.io/projected/755f6eab-5d03-4368-9670-b1402084474c-kube-api-access-bsbh2\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-c98nc\" (UID: \"755f6eab-5d03-4368-9670-b1402084474c\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-c98nc"
Nov 03 20:20:26.817917 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:26.817892    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/755f6eab-5d03-4368-9670-b1402084474c-workload-certs\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-c98nc\" (UID: \"755f6eab-5d03-4368-9670-b1402084474c\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-c98nc"
Nov 03 20:20:26.818199 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:26.818176    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/755f6eab-5d03-4368-9670-b1402084474c-istio-data\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-c98nc\" (UID: \"755f6eab-5d03-4368-9670-b1402084474c\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-c98nc"
Nov 03 20:20:26.818293 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:26.818219    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/755f6eab-5d03-4368-9670-b1402084474c-istiod-ca-cert\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-c98nc\" (UID: \"755f6eab-5d03-4368-9670-b1402084474c\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-c98nc"
Nov 03 20:20:26.818293 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:26.818225    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/755f6eab-5d03-4368-9670-b1402084474c-credential-socket\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-c98nc\" (UID: \"755f6eab-5d03-4368-9670-b1402084474c\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-c98nc"
Nov 03 20:20:26.818472 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:26.818450    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/755f6eab-5d03-4368-9670-b1402084474c-workload-socket\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-c98nc\" (UID: \"755f6eab-5d03-4368-9670-b1402084474c\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-c98nc"
Nov 03 20:20:26.819297 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:26.819274    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/755f6eab-5d03-4368-9670-b1402084474c-istio-envoy\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-c98nc\" (UID: \"755f6eab-5d03-4368-9670-b1402084474c\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-c98nc"
Nov 03 20:20:26.819417 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:26.819399    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/755f6eab-5d03-4368-9670-b1402084474c-istio-podinfo\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-c98nc\" (UID: \"755f6eab-5d03-4368-9670-b1402084474c\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-c98nc"
Nov 03 20:20:26.837963 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:26.837934    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/755f6eab-5d03-4368-9670-b1402084474c-istio-token\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-c98nc\" (UID: \"755f6eab-5d03-4368-9670-b1402084474c\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-c98nc"
Nov 03 20:20:26.838213 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:26.838196    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsbh2\" (UniqueName: \"kubernetes.io/projected/755f6eab-5d03-4368-9670-b1402084474c-kube-api-access-bsbh2\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-c98nc\" (UID: \"755f6eab-5d03-4368-9670-b1402084474c\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-c98nc"
Nov 03 20:20:26.880233 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:26.880204    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-c98nc"
Nov 03 20:20:26.951745 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:26.951703    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-xszg4"]
Nov 03 20:20:26.965240 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:26.965143    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-5bn7t"]
Nov 03 20:20:26.965610 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:26.965294    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-xszg4"
Nov 03 20:20:26.980378 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:26.978662    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-xszg4"]
Nov 03 20:20:26.980378 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:26.978805    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-5bn7t"
Nov 03 20:20:27.035699 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:27.035667    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26831c71-2091-4090-843d-f7268d257114-istio-data\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-5bn7t\" (UID: \"26831c71-2091-4090-843d-f7268d257114\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-5bn7t"
Nov 03 20:20:27.035998 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:27.035740    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26831c71-2091-4090-843d-f7268d257114-istio-token\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-5bn7t\" (UID: \"26831c71-2091-4090-843d-f7268d257114\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-5bn7t"
Nov 03 20:20:27.035998 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:27.035812    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ee1819e-4649-4d3e-9cfc-0d2d72c1b481-istio-data\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-xszg4\" (UID: \"4ee1819e-4649-4d3e-9cfc-0d2d72c1b481\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-xszg4"
Nov 03 20:20:27.035998 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:27.035836    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ee1819e-4649-4d3e-9cfc-0d2d72c1b481-istiod-ca-cert\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-xszg4\" (UID: \"4ee1819e-4649-4d3e-9cfc-0d2d72c1b481\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-xszg4"
Nov 03 20:20:27.035998 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:27.035909    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26831c71-2091-4090-843d-f7268d257114-istio-podinfo\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-5bn7t\" (UID: \"26831c71-2091-4090-843d-f7268d257114\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-5bn7t"
Nov 03 20:20:27.036411 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:27.036391    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26831c71-2091-4090-843d-f7268d257114-credential-socket\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-5bn7t\" (UID: \"26831c71-2091-4090-843d-f7268d257114\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-5bn7t"
Nov 03 20:20:27.036451 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:27.036423    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26831c71-2091-4090-843d-f7268d257114-workload-socket\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-5bn7t\" (UID: \"26831c71-2091-4090-843d-f7268d257114\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-5bn7t"
Nov 03 20:20:27.036451 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:27.036441    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26831c71-2091-4090-843d-f7268d257114-istio-envoy\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-5bn7t\" (UID: \"26831c71-2091-4090-843d-f7268d257114\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-5bn7t"
Nov 03 20:20:27.036507 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:27.036459    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26831c71-2091-4090-843d-f7268d257114-istiod-ca-cert\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-5bn7t\" (UID: \"26831c71-2091-4090-843d-f7268d257114\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-5bn7t"
Nov 03 20:20:27.036507 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:27.036488    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cljz\" (UniqueName: \"kubernetes.io/projected/26831c71-2091-4090-843d-f7268d257114-kube-api-access-9cljz\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-5bn7t\" (UID: \"26831c71-2091-4090-843d-f7268d257114\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-5bn7t"
Nov 03 20:20:27.036570 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:27.036517    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ee1819e-4649-4d3e-9cfc-0d2d72c1b481-istio-envoy\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-xszg4\" (UID: \"4ee1819e-4649-4d3e-9cfc-0d2d72c1b481\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-xszg4"
Nov 03 20:20:27.036570 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:27.036543    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ee1819e-4649-4d3e-9cfc-0d2d72c1b481-credential-socket\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-xszg4\" (UID: \"4ee1819e-4649-4d3e-9cfc-0d2d72c1b481\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-xszg4"
Nov 03 20:20:27.036570 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:27.036558    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbpvb\" (UniqueName: \"kubernetes.io/projected/4ee1819e-4649-4d3e-9cfc-0d2d72c1b481-kube-api-access-bbpvb\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-xszg4\" (UID: \"4ee1819e-4649-4d3e-9cfc-0d2d72c1b481\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-xszg4"
Nov 03 20:20:27.036662 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:27.036584    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26831c71-2091-4090-843d-f7268d257114-workload-certs\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-5bn7t\" (UID: \"26831c71-2091-4090-843d-f7268d257114\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-5bn7t"
Nov 03 20:20:27.036662 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:27.036610    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ee1819e-4649-4d3e-9cfc-0d2d72c1b481-istio-token\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-xszg4\" (UID: \"4ee1819e-4649-4d3e-9cfc-0d2d72c1b481\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-xszg4"
Nov 03 20:20:27.036662 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:27.036651    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ee1819e-4649-4d3e-9cfc-0d2d72c1b481-workload-certs\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-xszg4\" (UID: \"4ee1819e-4649-4d3e-9cfc-0d2d72c1b481\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-xszg4"
Nov 03 20:20:27.036749 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:27.036684    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ee1819e-4649-4d3e-9cfc-0d2d72c1b481-istio-podinfo\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-xszg4\" (UID: \"4ee1819e-4649-4d3e-9cfc-0d2d72c1b481\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-xszg4"
Nov 03 20:20:27.036749 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:27.036710    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ee1819e-4649-4d3e-9cfc-0d2d72c1b481-workload-socket\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-xszg4\" (UID: \"4ee1819e-4649-4d3e-9cfc-0d2d72c1b481\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-xszg4"
Nov 03 20:20:27.046161 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:27.046075    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-5bn7t"]
Nov 03 20:20:27.175026 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:27.174983    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cljz\" (UniqueName: \"kubernetes.io/projected/26831c71-2091-4090-843d-f7268d257114-kube-api-access-9cljz\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-5bn7t\" (UID: \"26831c71-2091-4090-843d-f7268d257114\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-5bn7t"
Nov 03 20:20:27.193594 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:27.193561    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ee1819e-4649-4d3e-9cfc-0d2d72c1b481-istio-envoy\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-xszg4\" (UID: \"4ee1819e-4649-4d3e-9cfc-0d2d72c1b481\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-xszg4"
Nov 03 20:20:27.193869 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:27.193846    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ee1819e-4649-4d3e-9cfc-0d2d72c1b481-credential-socket\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-xszg4\" (UID: \"4ee1819e-4649-4d3e-9cfc-0d2d72c1b481\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-xszg4"
Nov 03 20:20:27.194045 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:27.194028    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbpvb\" (UniqueName: \"kubernetes.io/projected/4ee1819e-4649-4d3e-9cfc-0d2d72c1b481-kube-api-access-bbpvb\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-xszg4\" (UID: \"4ee1819e-4649-4d3e-9cfc-0d2d72c1b481\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-xszg4"
Nov 03 20:20:27.194278 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:27.194256    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26831c71-2091-4090-843d-f7268d257114-workload-certs\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-5bn7t\" (UID: \"26831c71-2091-4090-843d-f7268d257114\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-5bn7t"
Nov 03 20:20:27.194429 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:27.194411    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ee1819e-4649-4d3e-9cfc-0d2d72c1b481-istio-token\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-xszg4\" (UID: \"4ee1819e-4649-4d3e-9cfc-0d2d72c1b481\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-xszg4"
Nov 03 20:20:27.194579 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:27.194566    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ee1819e-4649-4d3e-9cfc-0d2d72c1b481-workload-certs\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-xszg4\" (UID: \"4ee1819e-4649-4d3e-9cfc-0d2d72c1b481\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-xszg4"
Nov 03 20:20:27.194714 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:27.194699    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ee1819e-4649-4d3e-9cfc-0d2d72c1b481-istio-podinfo\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-xszg4\" (UID: \"4ee1819e-4649-4d3e-9cfc-0d2d72c1b481\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-xszg4"
Nov 03 20:20:27.194864 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:27.194848    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ee1819e-4649-4d3e-9cfc-0d2d72c1b481-workload-socket\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-xszg4\" (UID: \"4ee1819e-4649-4d3e-9cfc-0d2d72c1b481\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-xszg4"
Nov 03 20:20:27.195135 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:27.195115    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26831c71-2091-4090-843d-f7268d257114-istio-data\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-5bn7t\" (UID: \"26831c71-2091-4090-843d-f7268d257114\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-5bn7t"
Nov 03 20:20:27.197313 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:27.197287    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26831c71-2091-4090-843d-f7268d257114-istio-token\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-5bn7t\" (UID: \"26831c71-2091-4090-843d-f7268d257114\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-5bn7t"
Nov 03 20:20:27.197517 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:27.197493    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ee1819e-4649-4d3e-9cfc-0d2d72c1b481-istio-data\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-xszg4\" (UID: \"4ee1819e-4649-4d3e-9cfc-0d2d72c1b481\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-xszg4"
Nov 03 20:20:27.197659 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:27.197641    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ee1819e-4649-4d3e-9cfc-0d2d72c1b481-istiod-ca-cert\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-xszg4\" (UID: \"4ee1819e-4649-4d3e-9cfc-0d2d72c1b481\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-xszg4"
Nov 03 20:20:27.197840 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:27.197820    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26831c71-2091-4090-843d-f7268d257114-istio-podinfo\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-5bn7t\" (UID: \"26831c71-2091-4090-843d-f7268d257114\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-5bn7t"
Nov 03 20:20:27.198073 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:27.198054    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26831c71-2091-4090-843d-f7268d257114-credential-socket\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-5bn7t\" (UID: \"26831c71-2091-4090-843d-f7268d257114\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-5bn7t"
Nov 03 20:20:27.198216 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:27.198175    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26831c71-2091-4090-843d-f7268d257114-workload-socket\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-5bn7t\" (UID: \"26831c71-2091-4090-843d-f7268d257114\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-5bn7t"
Nov 03 20:20:27.198321 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:27.198306    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26831c71-2091-4090-843d-f7268d257114-istio-envoy\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-5bn7t\" (UID: \"26831c71-2091-4090-843d-f7268d257114\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-5bn7t"
Nov 03 20:20:27.198426 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:27.198412    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26831c71-2091-4090-843d-f7268d257114-istiod-ca-cert\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-5bn7t\" (UID: \"26831c71-2091-4090-843d-f7268d257114\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-5bn7t"
Nov 03 20:20:27.209120 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:27.196813    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ee1819e-4649-4d3e-9cfc-0d2d72c1b481-credential-socket\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-xszg4\" (UID: \"4ee1819e-4649-4d3e-9cfc-0d2d72c1b481\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-xszg4"
Nov 03 20:20:27.209120 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:27.195702    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26831c71-2091-4090-843d-f7268d257114-workload-certs\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-5bn7t\" (UID: \"26831c71-2091-4090-843d-f7268d257114\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-5bn7t"
Nov 03 20:20:27.213226 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:27.196270    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ee1819e-4649-4d3e-9cfc-0d2d72c1b481-workload-certs\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-xszg4\" (UID: \"4ee1819e-4649-4d3e-9cfc-0d2d72c1b481\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-xszg4"
Nov 03 20:20:27.260349 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:27.260264    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26831c71-2091-4090-843d-f7268d257114-istiod-ca-cert\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-5bn7t\" (UID: \"26831c71-2091-4090-843d-f7268d257114\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-5bn7t"
Nov 03 20:20:27.263531 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:27.262145    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26831c71-2091-4090-843d-f7268d257114-istio-data\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-5bn7t\" (UID: \"26831c71-2091-4090-843d-f7268d257114\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-5bn7t"
Nov 03 20:20:27.263531 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:27.262842    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ee1819e-4649-4d3e-9cfc-0d2d72c1b481-istiod-ca-cert\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-xszg4\" (UID: \"4ee1819e-4649-4d3e-9cfc-0d2d72c1b481\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-xszg4"
Nov 03 20:20:27.263531 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:27.263328    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ee1819e-4649-4d3e-9cfc-0d2d72c1b481-workload-socket\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-xszg4\" (UID: \"4ee1819e-4649-4d3e-9cfc-0d2d72c1b481\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-xszg4"
Nov 03 20:20:27.266131 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:27.266111    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26831c71-2091-4090-843d-f7268d257114-workload-socket\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-5bn7t\" (UID: \"26831c71-2091-4090-843d-f7268d257114\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-5bn7t"
Nov 03 20:20:27.269190 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:27.269164    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26831c71-2091-4090-843d-f7268d257114-credential-socket\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-5bn7t\" (UID: \"26831c71-2091-4090-843d-f7268d257114\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-5bn7t"
Nov 03 20:20:27.276638 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:27.276610    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ee1819e-4649-4d3e-9cfc-0d2d72c1b481-istio-data\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-xszg4\" (UID: \"4ee1819e-4649-4d3e-9cfc-0d2d72c1b481\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-xszg4"
Nov 03 20:20:27.316030 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:27.311270    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbpvb\" (UniqueName: \"kubernetes.io/projected/4ee1819e-4649-4d3e-9cfc-0d2d72c1b481-kube-api-access-bbpvb\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-xszg4\" (UID: \"4ee1819e-4649-4d3e-9cfc-0d2d72c1b481\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-xszg4"
Nov 03 20:20:27.316030 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:27.315426    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26831c71-2091-4090-843d-f7268d257114-istio-podinfo\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-5bn7t\" (UID: \"26831c71-2091-4090-843d-f7268d257114\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-5bn7t"
Nov 03 20:20:27.319789 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:27.319323    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26831c71-2091-4090-843d-f7268d257114-istio-envoy\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-5bn7t\" (UID: \"26831c71-2091-4090-843d-f7268d257114\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-5bn7t"
Nov 03 20:20:27.319789 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:27.319522    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ee1819e-4649-4d3e-9cfc-0d2d72c1b481-istio-token\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-xszg4\" (UID: \"4ee1819e-4649-4d3e-9cfc-0d2d72c1b481\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-xszg4"
Nov 03 20:20:27.335613 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:27.333418    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ee1819e-4649-4d3e-9cfc-0d2d72c1b481-istio-podinfo\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-xszg4\" (UID: \"4ee1819e-4649-4d3e-9cfc-0d2d72c1b481\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-xszg4"
Nov 03 20:20:27.335613 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:27.334979    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26831c71-2091-4090-843d-f7268d257114-istio-token\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-5bn7t\" (UID: \"26831c71-2091-4090-843d-f7268d257114\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-5bn7t"
Nov 03 20:20:27.340827 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:27.340185    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ee1819e-4649-4d3e-9cfc-0d2d72c1b481-istio-envoy\") pod \"e2e-worker-w3-482722-openshift-default-7645487df-xszg4\" (UID: \"4ee1819e-4649-4d3e-9cfc-0d2d72c1b481\") " pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-xszg4"
Nov 03 20:20:27.343315 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:27.342288    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-xszg4"
Nov 03 20:20:27.364297 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:27.364251    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-c98nc"]
Nov 03 20:20:27.378515 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:27.377877    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:20:27.378515 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:27.378020    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:20:27.378515 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:27.378115    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:20:27.388402 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:27.382766    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cljz\" (UniqueName: \"kubernetes.io/projected/26831c71-2091-4090-843d-f7268d257114-kube-api-access-9cljz\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-5bn7t\" (UID: \"26831c71-2091-4090-843d-f7268d257114\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-5bn7t"
Nov 03 20:20:27.390091 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:27.390056    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-mt7dd" event={"ID":"55945440-d40e-4235-8b84-d1e2a013372d","Type":"ContainerStarted","Data":"3846399a93c8fddbeb80930fda59b34d3625c7cf8f08fb90f62591110fad7bec"}
Nov 03 20:20:27.390193 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:27.390121    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-mt7dd" event={"ID":"55945440-d40e-4235-8b84-d1e2a013372d","Type":"ContainerStarted","Data":"8b1e9f15e462251ff6d98491599b0a0ed6b83cbdf0f9ba718552b9a2c23c3f79"}
Nov 03 20:20:27.392831 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:27.392803    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-xcxnz" event={"ID":"63727774-9118-4b83-bede-04208e8d59b9","Type":"ContainerStarted","Data":"c6574ca71b7f10f0ce3bcd5398a085b8f18c8a09ee7a532e8935c39af660977c"}
Nov 03 20:20:27.392926 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:27.392860    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-xcxnz" event={"ID":"63727774-9118-4b83-bede-04208e8d59b9","Type":"ContainerStarted","Data":"0e6626c5f047215d8796f5a752597e077e75813ee563b18ff17cc55aac9cd69c"}
Nov 03 20:20:27.414021 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:20:27.413974    5570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8bwhz" containerName="extractor"
Nov 03 20:20:27.417052 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:27.416988    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-mt7dd"
Nov 03 20:20:27.418606 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:27.418575    5570 patch_prober.go:29] interesting pod/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-mt7dd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Nov 03 20:20:27.418751 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:27.418628    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-mt7dd" podUID="55945440-d40e-4235-8b84-d1e2a013372d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Nov 03 20:20:27.444636 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:27.444580    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-mt7dd" podStartSLOduration=1.444559558 podStartE2EDuration="1.444559558s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:20:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:20:27.440631446 +0000 UTC m=+961.593293332" watchObservedRunningTime="2026-11-03 20:20:27.444559558 +0000 UTC m=+961.597221447"
Nov 03 20:20:27.471374 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:27.469349    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-xcxnz"
Nov 03 20:20:27.471619 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:27.471484    5570 patch_prober.go:29] interesting pod/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-xcxnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Nov 03 20:20:27.471619 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:27.471541    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-xcxnz" podUID="63727774-9118-4b83-bede-04208e8d59b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Nov 03 20:20:27.493392 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:27.492411    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-xcxnz" podStartSLOduration=1.492396973 podStartE2EDuration="1.492396973s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:20:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:20:27.491923451 +0000 UTC m=+961.644585339" watchObservedRunningTime="2026-11-03 20:20:27.492396973 +0000 UTC m=+961.645058861"
Nov 03 20:20:27.662464 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:27.662108    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-5bn7t"
Nov 03 20:20:27.731201 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:27.731173    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-xszg4"]
Nov 03 20:20:27.732753 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:20:27.732726    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ee1819e_4649_4d3e_9cfc_0d2d72c1b481.slice/crio-3295b90f37b38c58c3dac7d240a875b80fc8cb0d376f52bad2d2dfc01a7093f1 WatchSource:0}: Error finding container 3295b90f37b38c58c3dac7d240a875b80fc8cb0d376f52bad2d2dfc01a7093f1: Status 404 returned error can't find the container with id 3295b90f37b38c58c3dac7d240a875b80fc8cb0d376f52bad2d2dfc01a7093f1
Nov 03 20:20:27.735084 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:27.735049    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:20:27.735160 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:27.735146    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:20:27.735194 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:27.735176    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:20:28.126962 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:28.124673    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-5bn7t"]
Nov 03 20:20:28.221025 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:28.220966    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:20:28.221615 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:28.221588    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:20:28.221746 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:28.221730    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:20:28.451813 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:28.441042    5570 patch_prober.go:29] interesting pod/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-mt7dd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Nov 03 20:20:28.451813 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:28.441096    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-mt7dd" podUID="55945440-d40e-4235-8b84-d1e2a013372d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Nov 03 20:20:28.469914 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:28.469105    5570 patch_prober.go:29] interesting pod/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-xcxnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Nov 03 20:20:28.469914 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:28.469156    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-xcxnz" podUID="63727774-9118-4b83-bede-04208e8d59b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Nov 03 20:20:28.608297 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:28.608212    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-5bn7t" event={"ID":"26831c71-2091-4090-843d-f7268d257114","Type":"ContainerStarted","Data":"6f18c125bb7b96723a2a1230ebd605a97e671aa34b786080797ff7650238c5f4"}
Nov 03 20:20:28.641814 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:28.614480    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-xszg4" event={"ID":"4ee1819e-4649-4d3e-9cfc-0d2d72c1b481","Type":"ContainerStarted","Data":"c70dfe54e707de3f4706134c4a875053c59da1192e0ce8ce4a82e0c984572532"}
Nov 03 20:20:28.641814 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:28.614548    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-xszg4" event={"ID":"4ee1819e-4649-4d3e-9cfc-0d2d72c1b481","Type":"ContainerStarted","Data":"3295b90f37b38c58c3dac7d240a875b80fc8cb0d376f52bad2d2dfc01a7093f1"}
Nov 03 20:20:28.724114 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:28.724061    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-c98nc" event={"ID":"755f6eab-5d03-4368-9670-b1402084474c","Type":"ContainerStarted","Data":"a414a503eff894ce1fa7aaee149e172cf442c1cd2ba6f8cd05579632d356f512"}
Nov 03 20:20:28.724498 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:28.724454    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-c98nc" event={"ID":"755f6eab-5d03-4368-9670-b1402084474c","Type":"ContainerStarted","Data":"e3767356ae7f055f3f8ffb0b6a7d0d90d09e33d2e9e54712809d90f637d27725"}
Nov 03 20:20:28.883854 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:28.883291    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-c98nc"
Nov 03 20:20:28.883854 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:28.883830    5570 patch_prober.go:29] interesting pod/e2e-worker-w2-9973d9-openshift-default-ffb6475-c98nc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Nov 03 20:20:28.884132 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:28.883894    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-c98nc" podUID="755f6eab-5d03-4368-9670-b1402084474c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Nov 03 20:20:28.919795 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:28.888609    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-c98nc" podStartSLOduration=2.88859127 podStartE2EDuration="2.88859127s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:20:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:20:28.813179578 +0000 UTC m=+962.965841468" watchObservedRunningTime="2026-11-03 20:20:28.88859127 +0000 UTC m=+963.041253158"
Nov 03 20:20:28.920285 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:28.920213    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-xszg4" podStartSLOduration=2.92019581 podStartE2EDuration="2.92019581s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:20:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:20:28.688663366 +0000 UTC m=+962.841325253" watchObservedRunningTime="2026-11-03 20:20:28.92019581 +0000 UTC m=+963.072857723"
Nov 03 20:20:29.350430 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:29.350390    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-xszg4"
Nov 03 20:20:29.351171 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:29.351145    5570 patch_prober.go:29] interesting pod/e2e-worker-w3-482722-openshift-default-7645487df-xszg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Nov 03 20:20:29.351349 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:29.351323    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-xszg4" podUID="4ee1819e-4649-4d3e-9cfc-0d2d72c1b481" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Nov 03 20:20:29.420913 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:29.420879    5570 patch_prober.go:29] interesting pod/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-mt7dd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Nov 03 20:20:29.421181 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:29.421154    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-mt7dd" podUID="55945440-d40e-4235-8b84-d1e2a013372d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Nov 03 20:20:29.467699 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:29.467661    5570 patch_prober.go:29] interesting pod/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-xcxnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Nov 03 20:20:29.468006 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:29.467974    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-xcxnz" podUID="63727774-9118-4b83-bede-04208e8d59b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Nov 03 20:20:29.510118 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:29.510071    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh" podUID="2d3647f2-89e9-4908-a8ee-d84e527abeb0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:29.803749 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:29.803697    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/distinct-2-w4-e80203-kserve-756f9b9bc6-69gxj" podUID="ccf5dc2f-f116-4917-bb1a-b136d8ed17f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:29.868878 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:29.868834    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-5bn7t" event={"ID":"26831c71-2091-4090-843d-f7268d257114","Type":"ContainerStarted","Data":"50869a3fc874fdc6908769230eea560a48902e803a7a881735d87e7285089421"}
Nov 03 20:20:29.896886 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:29.884331    5570 patch_prober.go:29] interesting pod/e2e-worker-w2-9973d9-openshift-default-ffb6475-c98nc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Nov 03 20:20:29.897214 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:29.884395    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/distinct-2-w0-e40816-kserve-79fb46654d-s242x" podUID="4ad9b048-925c-4d01-a839-a5f25d4dd02d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:29.897528 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:29.897176    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-c98nc" podUID="755f6eab-5d03-4368-9670-b1402084474c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Nov 03 20:20:29.995087 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:29.994322    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/distinct-2-w3-482722-kserve-74bf58cf7d-v9cjw" podUID="9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:30.007280 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:30.006663    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-5bn7t" podStartSLOduration=4.006642783 podStartE2EDuration="4.006642783s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:20:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:20:29.924869167 +0000 UTC m=+964.077531059" watchObservedRunningTime="2026-11-03 20:20:30.006642783 +0000 UTC m=+964.159304676"
Nov 03 20:20:30.018033 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:30.016922    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/premium-w3-482722-kserve-689dc7bd5d-x8rjp" podUID="369045c3-4d6a-49a7-9bc6-f2939883bee8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:30.146864 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:30.132875    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-2-w6-9e1bcb-kserve-7d96d9f6f6-4f7fl" podUID="56353b0f-08ee-4ada-9dc0-ac5afd5a59c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:30.158923 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:30.158864    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/premium-w4-e80203-kserve-5f9f445456-bms9d" podUID="6ad76efc-20c2-4951-b22a-c5a39cd39ed4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:30.184218 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:30.184172    5570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-e80203/distinct-2-w4-e80203-kserve-756f9b9bc6-69gxj" podUID="ccf5dc2f-f116-4917-bb1a-b136d8ed17f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:30.304036 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:30.303975    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz" podUID="3a715ef1-38e9-4386-9eec-e10e8df11a10" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": context deadline exceeded"
Nov 03 20:20:30.349886 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:30.349846    5570 patch_prober.go:29] interesting pod/e2e-worker-w3-482722-openshift-default-7645487df-xszg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Nov 03 20:20:30.350152 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:30.350119    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-xszg4" podUID="4ee1819e-4649-4d3e-9cfc-0d2d72c1b481" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Nov 03 20:20:30.422574 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:30.421188    5570 patch_prober.go:29] interesting pod/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-mt7dd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Nov 03 20:20:30.422574 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:30.421250    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-mt7dd" podUID="55945440-d40e-4235-8b84-d1e2a013372d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Nov 03 20:20:30.467798 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:30.467617    5570 patch_prober.go:29] interesting pod/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-xcxnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Nov 03 20:20:30.467798 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:30.467678    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-xcxnz" podUID="63727774-9118-4b83-bede-04208e8d59b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Nov 03 20:20:30.514018 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:30.513955    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh" podUID="2d3647f2-89e9-4908-a8ee-d84e527abeb0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:30.514870 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:30.514838    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-9e1bcb/premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh"
Nov 03 20:20:30.766813 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:30.742091    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-5bn7t"
Nov 03 20:20:30.796186 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:30.795209    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/distinct-w3-482722-kserve-d876967c-kd488" podUID="9a97f9cc-2261-471d-9053-51ee965850bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:30.828796 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:30.808425    5570 patch_prober.go:29] interesting pod/e2e-worker-w0-e40816-openshift-default-744c57967c-5bn7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body=
Nov 03 20:20:30.828796 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:30.808493    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-5bn7t" podUID="26831c71-2091-4090-843d-f7268d257114" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused"
Nov 03 20:20:30.831155 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:30.829869    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/distinct-2-w4-e80203-kserve-756f9b9bc6-69gxj" podUID="ccf5dc2f-f116-4917-bb1a-b136d8ed17f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded"
Nov 03 20:20:30.831155 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:30.830290    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/premium-w1-6fc748-kserve-7474db9cd7-nf2xk" podUID="830e062e-83fc-4de7-91a7-5534dac1d46b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Nov 03 20:20:30.831155 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:30.830968    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-e80203/distinct-2-w4-e80203-kserve-756f9b9bc6-69gxj"
Nov 03 20:20:30.893368 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:30.891419    5570 patch_prober.go:29] interesting pod/e2e-worker-w2-9973d9-openshift-default-ffb6475-c98nc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Nov 03 20:20:30.893368 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:30.891477    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-c98nc" podUID="755f6eab-5d03-4368-9670-b1402084474c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Nov 03 20:20:30.970816 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:30.913012    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/distinct-2-w0-e40816-kserve-79fb46654d-s242x" podUID="4ad9b048-925c-4d01-a839-a5f25d4dd02d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded"
Nov 03 20:20:31.179966 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:31.074081    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/distinct-2-w3-482722-kserve-74bf58cf7d-v9cjw" podUID="9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded"
Nov 03 20:20:31.257516 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:31.209970    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-2-w6-9e1bcb-kserve-7d96d9f6f6-4f7fl" podUID="56353b0f-08ee-4ada-9dc0-ac5afd5a59c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:31.615823 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:31.614140    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh" podUID="2d3647f2-89e9-4908-a8ee-d84e527abeb0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:31.626010 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:31.625977    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e40816/distinct-2-w0-e40816-kserve-79fb46654d-s242x"
Nov 03 20:20:32.196801 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:31.767367    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/premium-w4-e80203-kserve-5f9f445456-bms9d" podUID="6ad76efc-20c2-4951-b22a-c5a39cd39ed4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Nov 03 20:20:32.196801 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:31.838938    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/simulator-w0-e40816-kserve-688f79796f-2tvw5" podUID="c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Nov 03 20:20:32.196801 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:31.897901    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/distinct-w3-482722-kserve-d876967c-kd488" podUID="9a97f9cc-2261-471d-9053-51ee965850bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:32.196801 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:31.902839    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz" podUID="3a715ef1-38e9-4386-9eec-e10e8df11a10" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:32.196801 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:31.942292    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/premium-w3-482722-kserve-689dc7bd5d-x8rjp" podUID="369045c3-4d6a-49a7-9bc6-f2939883bee8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:32.196801 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:31.942339    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/premium-w2-9973d9-kserve-5c6dd4669c-5gv8x" podUID="cefc89ad-7ae1-433b-979d-6601678a6d98" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:32.196801 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:31.973456    5570 patch_prober.go:29] interesting pod/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-xcxnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Nov 03 20:20:32.196801 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:32.114597    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-xcxnz" podUID="63727774-9118-4b83-bede-04208e8d59b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Nov 03 20:20:32.196801 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:32.107618    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/distinct-2-w4-e80203-kserve-756f9b9bc6-69gxj" podUID="ccf5dc2f-f116-4917-bb1a-b136d8ed17f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:32.196801 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:32.075983    5570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-482722/distinct-2-w3-482722-kserve-74bf58cf7d-v9cjw" podUID="9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/health\": context deadline exceeded"
Nov 03 20:20:32.196801 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:31.973556    5570 patch_prober.go:29] interesting pod/e2e-worker-w3-482722-openshift-default-7645487df-xszg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Nov 03 20:20:32.196801 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:32.114686    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-xszg4" podUID="4ee1819e-4649-4d3e-9cfc-0d2d72c1b481" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Nov 03 20:20:32.196801 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:32.120028    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/premium-w1-6fc748-kserve-7474db9cd7-nf2xk" podUID="830e062e-83fc-4de7-91a7-5534dac1d46b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:32.196801 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:32.184767    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/distinct-2-w3-482722-kserve-74bf58cf7d-v9cjw" podUID="9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:32.196801 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:32.184930    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/distinct-2-w0-e40816-kserve-79fb46654d-s242x" podUID="4ad9b048-925c-4d01-a839-a5f25d4dd02d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded"
Nov 03 20:20:32.242688 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:32.230117    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-2-w6-9e1bcb-kserve-7d96d9f6f6-4f7fl" podUID="56353b0f-08ee-4ada-9dc0-ac5afd5a59c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:32.268573 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:32.268526    5570 patch_prober.go:29] interesting pod/e2e-worker-w2-9973d9-openshift-default-ffb6475-c98nc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Nov 03 20:20:32.269079 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:32.269049    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-c98nc" podUID="755f6eab-5d03-4368-9670-b1402084474c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Nov 03 20:20:32.384984 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:32.384946    5570 patch_prober.go:29] interesting pod/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-mt7dd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Nov 03 20:20:32.452204 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:32.452061    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-mt7dd" podUID="55945440-d40e-4235-8b84-d1e2a013372d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Nov 03 20:20:32.556923 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:32.556890    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-482722/distinct-2-w3-482722-kserve-74bf58cf7d-v9cjw"
Nov 03 20:20:32.999800 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:32.731727    5570 patch_prober.go:29] interesting pod/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-mt7dd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Nov 03 20:20:32.999800 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:32.891565    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh" podUID="2d3647f2-89e9-4908-a8ee-d84e527abeb0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:32.999800 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:32.955370    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-mt7dd" podUID="55945440-d40e-4235-8b84-d1e2a013372d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Nov 03 20:20:32.999800 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:32.733344    5570 patch_prober.go:29] interesting pod/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-xcxnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Nov 03 20:20:32.999800 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:32.955446    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-xcxnz" podUID="63727774-9118-4b83-bede-04208e8d59b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Nov 03 20:20:32.999800 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:32.770855    5570 patch_prober.go:29] interesting pod/e2e-worker-w3-482722-openshift-default-7645487df-xszg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Nov 03 20:20:32.999800 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:32.955492    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-xszg4" podUID="4ee1819e-4649-4d3e-9cfc-0d2d72c1b481" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Nov 03 20:20:32.999800 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:32.793903    5570 patch_prober.go:29] interesting pod/e2e-worker-w0-e40816-openshift-default-744c57967c-5bn7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body=
Nov 03 20:20:32.999800 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:32.955538    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-5bn7t" podUID="26831c71-2091-4090-843d-f7268d257114" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused"
Nov 03 20:20:32.999800 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:32.893232    5570 patch_prober.go:29] interesting pod/e2e-worker-w2-9973d9-openshift-default-ffb6475-c98nc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Nov 03 20:20:32.999800 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:32.955718    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-c98nc" podUID="755f6eab-5d03-4368-9670-b1402084474c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Nov 03 20:20:32.999800 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:32.955880    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-2-w6-9e1bcb-kserve-7d96d9f6f6-4f7fl"
Nov 03 20:20:33.087901 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:33.087855    5570 patch_prober.go:29] interesting pod/e2e-worker-w0-e40816-openshift-default-744c57967c-5bn7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body=
Nov 03 20:20:33.088196 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:33.088161    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-5bn7t" podUID="26831c71-2091-4090-843d-f7268d257114" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused"
Nov 03 20:20:33.180382 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:33.180337    5570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-6fc748/premium-w1-6fc748-kserve-7474db9cd7-nf2xk" podUID="830e062e-83fc-4de7-91a7-5534dac1d46b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:33.243801 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:33.185337    5570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-e40816/distinct-2-w0-e40816-kserve-79fb46654d-s242x" podUID="4ad9b048-925c-4d01-a839-a5f25d4dd02d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:33.374339 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:33.374242    5570 patch_prober.go:29] interesting pod/e2e-worker-w3-482722-openshift-default-7645487df-xszg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Nov 03 20:20:33.374573 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:33.374545    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-xszg4" podUID="4ee1819e-4649-4d3e-9cfc-0d2d72c1b481" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Nov 03 20:20:33.406114 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:33.406078    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-e80203/premium-w4-e80203-kserve-5f9f445456-bms9d"
Nov 03 20:20:33.479445 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:33.479413    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-482722/distinct-w3-482722-kserve-d876967c-kd488"
Nov 03 20:20:33.480571 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:33.480549    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz"
Nov 03 20:20:33.498277 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:33.498246    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-482722/premium-w3-482722-kserve-689dc7bd5d-x8rjp"
Nov 03 20:20:33.498583 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:33.498563    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6fc748/premium-w1-6fc748-kserve-7474db9cd7-nf2xk"
Nov 03 20:20:33.503577 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:33.502914    5570 patch_prober.go:29] interesting pod/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-mt7dd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Nov 03 20:20:33.503577 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:33.502972    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-mt7dd" podUID="55945440-d40e-4235-8b84-d1e2a013372d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Nov 03 20:20:33.503577 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:33.503048    5570 patch_prober.go:29] interesting pod/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-xcxnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Nov 03 20:20:33.503577 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:33.503068    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-xcxnz" podUID="63727774-9118-4b83-bede-04208e8d59b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Nov 03 20:20:33.606142 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:33.606091    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/premium-w4-e80203-kserve-5f9f445456-bms9d" podUID="6ad76efc-20c2-4951-b22a-c5a39cd39ed4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:33.627574 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:33.627471    5570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-e80203/premium-w4-e80203-kserve-5f9f445456-bms9d" podUID="6ad76efc-20c2-4951-b22a-c5a39cd39ed4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:33.681854 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:33.672623    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/simulator-w0-e40816-kserve-688f79796f-2tvw5" podUID="c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:33.681854 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:33.673100    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e40816/simulator-w0-e40816-kserve-688f79796f-2tvw5"
Nov 03 20:20:33.720290 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:33.719682    5570 patch_prober.go:29] interesting pod/e2e-worker-w0-e40816-openshift-default-744c57967c-5bn7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body=
Nov 03 20:20:33.720290 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:33.719748    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-5bn7t" podUID="26831c71-2091-4090-843d-f7268d257114" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused"
Nov 03 20:20:33.731948 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:33.731902    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/premium-w1-6fc748-kserve-7474db9cd7-nf2xk" podUID="830e062e-83fc-4de7-91a7-5534dac1d46b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:33.827800 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:33.734316    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz" podUID="3a715ef1-38e9-4386-9eec-e10e8df11a10" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:33.827800 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:33.734358    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/distinct-w3-482722-kserve-d876967c-kd488" podUID="9a97f9cc-2261-471d-9053-51ee965850bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded"
Nov 03 20:20:33.827800 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:33.734392    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/premium-w2-9973d9-kserve-5c6dd4669c-5gv8x" podUID="cefc89ad-7ae1-433b-979d-6601678a6d98" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Nov 03 20:20:33.827800 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:33.734408    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/premium-w3-482722-kserve-689dc7bd5d-x8rjp" podUID="369045c3-4d6a-49a7-9bc6-f2939883bee8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Nov 03 20:20:33.827800 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:33.790132    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9973d9/premium-w2-9973d9-kserve-5c6dd4669c-5gv8x"
Nov 03 20:20:33.827800 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:33.734960    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-2-w6-9e1bcb-kserve-7d96d9f6f6-4f7fl" podUID="56353b0f-08ee-4ada-9dc0-ac5afd5a59c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded"
Nov 03 20:20:33.827800 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:33.734992    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/distinct-2-w0-e40816-kserve-79fb46654d-s242x" podUID="4ad9b048-925c-4d01-a839-a5f25d4dd02d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded"
Nov 03 20:20:33.827800 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:33.735012    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/distinct-2-w4-e80203-kserve-756f9b9bc6-69gxj" podUID="ccf5dc2f-f116-4917-bb1a-b136d8ed17f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded"
Nov 03 20:20:33.827800 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:33.777338    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/distinct-2-w3-482722-kserve-74bf58cf7d-v9cjw" podUID="9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:33.964869 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:33.915392    5570 patch_prober.go:29] interesting pod/e2e-worker-w2-9973d9-openshift-default-ffb6475-c98nc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Nov 03 20:20:33.964869 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:33.915451    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-c98nc" podUID="755f6eab-5d03-4368-9670-b1402084474c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Nov 03 20:20:33.973805 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:33.966696    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh" podUID="2d3647f2-89e9-4908-a8ee-d84e527abeb0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Nov 03 20:20:34.462355 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:34.462320    5570 patch_prober.go:29] interesting pod/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-mt7dd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Nov 03 20:20:34.536807 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:34.536064    5570 patch_prober.go:29] interesting pod/e2e-worker-w3-482722-openshift-default-7645487df-xszg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Nov 03 20:20:34.536807 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:34.536124    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-xszg4" podUID="4ee1819e-4649-4d3e-9cfc-0d2d72c1b481" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Nov 03 20:20:34.538184 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:34.537119    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-mt7dd" podUID="55945440-d40e-4235-8b84-d1e2a013372d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Nov 03 20:20:34.538184 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:34.537420    5570 patch_prober.go:29] interesting pod/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-xcxnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Nov 03 20:20:34.538184 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:34.537460    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-xcxnz" podUID="63727774-9118-4b83-bede-04208e8d59b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Nov 03 20:20:34.709849 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:34.668907    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/premium-w4-e80203-kserve-5f9f445456-bms9d" podUID="6ad76efc-20c2-4951-b22a-c5a39cd39ed4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:34.792178 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:34.792084    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-2-w6-9e1bcb-kserve-7d96d9f6f6-4f7fl" podUID="56353b0f-08ee-4ada-9dc0-ac5afd5a59c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:34.792905 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:34.792868    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/premium-w1-6fc748-kserve-7474db9cd7-nf2xk" podUID="830e062e-83fc-4de7-91a7-5534dac1d46b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Nov 03 20:20:34.793329 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:34.793299    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz" podUID="3a715ef1-38e9-4386-9eec-e10e8df11a10" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:34.793729 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:34.793705    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/distinct-w3-482722-kserve-d876967c-kd488" podUID="9a97f9cc-2261-471d-9053-51ee965850bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded"
Nov 03 20:20:34.794164 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:34.794140    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/premium-w2-9973d9-kserve-5c6dd4669c-5gv8x" podUID="cefc89ad-7ae1-433b-979d-6601678a6d98" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Nov 03 20:20:34.794529 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:34.794504    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/premium-w3-482722-kserve-689dc7bd5d-x8rjp" podUID="369045c3-4d6a-49a7-9bc6-f2939883bee8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Nov 03 20:20:34.837797 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:34.795732    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/simulator-w0-e40816-kserve-688f79796f-2tvw5" podUID="c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Nov 03 20:20:34.837797 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:34.816596    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/distinct-2-w3-482722-kserve-74bf58cf7d-v9cjw" podUID="9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:34.837797 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:34.817124    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/distinct-2-w0-e40816-kserve-79fb46654d-s242x" podUID="4ad9b048-925c-4d01-a839-a5f25d4dd02d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:34.837797 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:34.817430    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/distinct-2-w4-e80203-kserve-756f9b9bc6-69gxj" podUID="ccf5dc2f-f116-4917-bb1a-b136d8ed17f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded"
Nov 03 20:20:34.840426 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:34.840389    5570 patch_prober.go:29] interesting pod/e2e-worker-w0-e40816-openshift-default-744c57967c-5bn7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body=
Nov 03 20:20:34.865689 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:34.865649    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-5bn7t" podUID="26831c71-2091-4090-843d-f7268d257114" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused"
Nov 03 20:20:34.865960 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:34.858906    5570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-9973d9/premium-w2-9973d9-kserve-5c6dd4669c-5gv8x" podUID="cefc89ad-7ae1-433b-979d-6601678a6d98" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:34.907798 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:34.901851    5570 patch_prober.go:29] interesting pod/e2e-worker-w2-9973d9-openshift-default-ffb6475-c98nc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Nov 03 20:20:34.907798 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:34.901944    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-c98nc" podUID="755f6eab-5d03-4368-9670-b1402084474c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Nov 03 20:20:34.907798 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:34.902980    5570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-482722/distinct-w3-482722-kserve-d876967c-kd488" podUID="9a97f9cc-2261-471d-9053-51ee965850bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:34.982801 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:34.971942    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh" podUID="2d3647f2-89e9-4908-a8ee-d84e527abeb0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:35.499204 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:35.499142    5570 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz" podUID="3a715ef1-38e9-4386-9eec-e10e8df11a10" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:35.704800 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:35.672084    5570 patch_prober.go:29] interesting pod/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-xcxnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Nov 03 20:20:36.006039 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:36.005989    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-xcxnz" podUID="63727774-9118-4b83-bede-04208e8d59b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Nov 03 20:20:36.006402 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:35.706099    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/premium-w4-e80203-kserve-5f9f445456-bms9d" podUID="6ad76efc-20c2-4951-b22a-c5a39cd39ed4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:36.034878 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:35.687596    5570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-482722/premium-w3-482722-kserve-689dc7bd5d-x8rjp" podUID="369045c3-4d6a-49a7-9bc6-f2939883bee8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:36.035228 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:35.850296    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/premium-w1-6fc748-kserve-7474db9cd7-nf2xk" podUID="830e062e-83fc-4de7-91a7-5534dac1d46b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Nov 03 20:20:36.061721 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:35.687143    5570 patch_prober.go:29] interesting pod/e2e-worker-w3-482722-openshift-default-7645487df-xszg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Nov 03 20:20:36.062095 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:36.062058    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-xszg4" podUID="4ee1819e-4649-4d3e-9cfc-0d2d72c1b481" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Nov 03 20:20:36.062275 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:35.850745    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/premium-w2-9973d9-kserve-5c6dd4669c-5gv8x" podUID="cefc89ad-7ae1-433b-979d-6601678a6d98" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Nov 03 20:20:36.088177 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:35.849993    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-2-w6-9e1bcb-kserve-7d96d9f6f6-4f7fl" podUID="56353b0f-08ee-4ada-9dc0-ac5afd5a59c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded"
Nov 03 20:20:36.118896 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:35.850481    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz" podUID="3a715ef1-38e9-4386-9eec-e10e8df11a10" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": context deadline exceeded"
Nov 03 20:20:36.120674 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:35.850609    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/distinct-w3-482722-kserve-d876967c-kd488" podUID="9a97f9cc-2261-471d-9053-51ee965850bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded"
Nov 03 20:20:36.125844 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:35.705139    5570 patch_prober.go:29] interesting pod/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-mt7dd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Nov 03 20:20:36.126133 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:36.126085    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-mt7dd" podUID="55945440-d40e-4235-8b84-d1e2a013372d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Nov 03 20:20:36.126298 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:35.737815    5570 patch_prober.go:29] interesting pod/e2e-worker-w0-e40816-openshift-default-744c57967c-5bn7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body=
Nov 03 20:20:36.131929 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:36.130296    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-5bn7t" podUID="26831c71-2091-4090-843d-f7268d257114" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused"
Nov 03 20:20:36.199842 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:35.853711    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/premium-w3-482722-kserve-689dc7bd5d-x8rjp" podUID="369045c3-4d6a-49a7-9bc6-f2939883bee8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:36.227801 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:35.880469    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/distinct-2-w4-e80203-kserve-756f9b9bc6-69gxj" podUID="ccf5dc2f-f116-4917-bb1a-b136d8ed17f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:36.227801 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:35.880872    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/distinct-2-w3-482722-kserve-74bf58cf7d-v9cjw" podUID="9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:36.227801 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:35.881182    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/simulator-w0-e40816-kserve-688f79796f-2tvw5" podUID="c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Nov 03 20:20:36.227801 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:35.886083    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/distinct-2-w0-e40816-kserve-79fb46654d-s242x" podUID="4ad9b048-925c-4d01-a839-a5f25d4dd02d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:36.227801 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:35.916111    5570 patch_prober.go:29] interesting pod/e2e-worker-w2-9973d9-openshift-default-ffb6475-c98nc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Nov 03 20:20:36.227801 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:36.201855    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-c98nc" podUID="755f6eab-5d03-4368-9670-b1402084474c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Nov 03 20:20:36.471823 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:36.466349    5570 patch_prober.go:29] interesting pod/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-mt7dd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Nov 03 20:20:36.471823 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:36.466430    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-mt7dd" podUID="55945440-d40e-4235-8b84-d1e2a013372d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Nov 03 20:20:36.504019 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:36.502679    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh" podUID="2d3647f2-89e9-4908-a8ee-d84e527abeb0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Nov 03 20:20:36.578923 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:36.578878    5570 patch_prober.go:29] interesting pod/e2e-worker-w3-482722-openshift-default-7645487df-xszg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Nov 03 20:20:36.582800 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:36.581866    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-xszg4" podUID="4ee1819e-4649-4d3e-9cfc-0d2d72c1b481" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Nov 03 20:20:36.666815 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:36.655973    5570 patch_prober.go:29] interesting pod/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-xcxnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Nov 03 20:20:36.666815 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:36.656059    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-xcxnz" podUID="63727774-9118-4b83-bede-04208e8d59b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Nov 03 20:20:36.999159 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:36.999119    5570 patch_prober.go:29] interesting pod/e2e-worker-w2-9973d9-openshift-default-ffb6475-c98nc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Nov 03 20:20:37.007946 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:37.007899    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-c98nc" podUID="755f6eab-5d03-4368-9670-b1402084474c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Nov 03 20:20:37.008269 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:37.000274    5570 patch_prober.go:29] interesting pod/e2e-worker-w0-e40816-openshift-default-744c57967c-5bn7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body=
Nov 03 20:20:37.011806 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:37.008406    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-5bn7t" podUID="26831c71-2091-4090-843d-f7268d257114" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused"
Nov 03 20:20:37.077734 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:37.077689    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/premium-w2-9973d9-kserve-5c6dd4669c-5gv8x" podUID="cefc89ad-7ae1-433b-979d-6601678a6d98" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:37.089440 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:37.089392    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-2-w6-9e1bcb-kserve-7d96d9f6f6-4f7fl" podUID="56353b0f-08ee-4ada-9dc0-ac5afd5a59c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:37.260866 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:37.145899    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/premium-w4-e80203-kserve-5f9f445456-bms9d" podUID="6ad76efc-20c2-4951-b22a-c5a39cd39ed4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Nov 03 20:20:37.522860 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:37.383966    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/distinct-2-w0-e40816-kserve-79fb46654d-s242x" podUID="4ad9b048-925c-4d01-a839-a5f25d4dd02d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:37.522860 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:37.392297    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/premium-w3-482722-kserve-689dc7bd5d-x8rjp" podUID="369045c3-4d6a-49a7-9bc6-f2939883bee8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Nov 03 20:20:37.522860 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:37.397727    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/distinct-2-w4-e80203-kserve-756f9b9bc6-69gxj" podUID="ccf5dc2f-f116-4917-bb1a-b136d8ed17f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded"
Nov 03 20:20:37.522860 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:37.407462    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/distinct-2-w3-482722-kserve-74bf58cf7d-v9cjw" podUID="9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded"
Nov 03 20:20:37.522860 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:37.408870    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/simulator-w0-e40816-kserve-688f79796f-2tvw5" podUID="c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Nov 03 20:20:37.522860 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:37.168758    5570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-2-w6-9e1bcb-kserve-7d96d9f6f6-4f7fl" podUID="56353b0f-08ee-4ada-9dc0-ac5afd5a59c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": context deadline exceeded"
Nov 03 20:20:37.522860 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:37.414682    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz" podUID="3a715ef1-38e9-4386-9eec-e10e8df11a10" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:37.522860 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:37.457164    5570 patch_prober.go:29] interesting pod/e2e-worker-w3-482722-openshift-default-7645487df-xszg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Nov 03 20:20:37.522860 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:37.457296    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-xszg4" podUID="4ee1819e-4649-4d3e-9cfc-0d2d72c1b481" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Nov 03 20:20:37.522860 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:37.488373    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/distinct-w3-482722-kserve-d876967c-kd488" podUID="9a97f9cc-2261-471d-9053-51ee965850bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded"
Nov 03 20:20:37.551838 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:37.528400    5570 patch_prober.go:29] interesting pod/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-xcxnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Nov 03 20:20:37.551838 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:37.145962    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/premium-w1-6fc748-kserve-7474db9cd7-nf2xk" podUID="830e062e-83fc-4de7-91a7-5534dac1d46b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Nov 03 20:20:37.551838 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:37.529083    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh" podUID="2d3647f2-89e9-4908-a8ee-d84e527abeb0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Nov 03 20:20:37.768795 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:37.767924    5570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-9e1bcb/premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh" podUID="2d3647f2-89e9-4908-a8ee-d84e527abeb0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:37.843738 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:37.528482    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-xcxnz" podUID="63727774-9118-4b83-bede-04208e8d59b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Nov 03 20:20:37.925483 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:20:37.919574    5570 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.611s"
Nov 03 20:20:37.976830 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:37.969218    5570 patch_prober.go:29] interesting pod/e2e-worker-w2-9973d9-openshift-default-ffb6475-c98nc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Nov 03 20:20:37.976830 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:37.969282    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-c98nc" podUID="755f6eab-5d03-4368-9670-b1402084474c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Nov 03 20:20:38.001931 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:37.999995    5570 patch_prober.go:29] interesting pod/e2e-worker-w0-e40816-openshift-default-744c57967c-5bn7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body=
Nov 03 20:20:38.100142 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:38.091729    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/premium-w2-9973d9-kserve-5c6dd4669c-5gv8x" podUID="cefc89ad-7ae1-433b-979d-6601678a6d98" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Nov 03 20:20:38.199561 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:38.114323    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-5bn7t" podUID="26831c71-2091-4090-843d-f7268d257114" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused"
Nov 03 20:20:38.292557 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:38.292513    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/premium-w4-e80203-kserve-5f9f445456-bms9d" podUID="6ad76efc-20c2-4951-b22a-c5a39cd39ed4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:38.314205 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:38.296915    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-2-w6-9e1bcb-kserve-7d96d9f6f6-4f7fl" podUID="56353b0f-08ee-4ada-9dc0-ac5afd5a59c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:38.338797 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:38.329368    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-mt7dd"
Nov 03 20:20:38.372849 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:38.343823    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-mt7dd"
Nov 03 20:20:38.372849 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:38.361145    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-mt7dd"
Nov 03 20:20:38.392702 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:38.392632    5570 patch_prober.go:29] interesting pod/e2e-worker-w3-482722-openshift-default-7645487df-xszg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Nov 03 20:20:38.393516 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:38.393168    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-xszg4" podUID="4ee1819e-4649-4d3e-9cfc-0d2d72c1b481" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Nov 03 20:20:38.393750 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:38.393371    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/premium-w3-482722-kserve-689dc7bd5d-x8rjp" podUID="369045c3-4d6a-49a7-9bc6-f2939883bee8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:38.394663 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:38.393394    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/distinct-2-w0-e40816-kserve-79fb46654d-s242x" podUID="4ad9b048-925c-4d01-a839-a5f25d4dd02d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded"
Nov 03 20:20:38.456956 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:38.456912    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/simulator-w0-e40816-kserve-688f79796f-2tvw5" podUID="c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Nov 03 20:20:38.458005 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:38.457969    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/distinct-2-w4-e80203-kserve-756f9b9bc6-69gxj" podUID="ccf5dc2f-f116-4917-bb1a-b136d8ed17f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded"
Nov 03 20:20:38.458839 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:38.458813    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/distinct-2-w3-482722-kserve-74bf58cf7d-v9cjw" podUID="9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded"
Nov 03 20:20:38.471807 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:38.460913    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz" podUID="3a715ef1-38e9-4386-9eec-e10e8df11a10" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": context deadline exceeded"
Nov 03 20:20:38.482399 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:38.482342    5570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-e40816/simulator-w0-e40816-kserve-688f79796f-2tvw5" podUID="c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:38.506190 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:38.498321    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/distinct-w3-482722-kserve-d876967c-kd488" podUID="9a97f9cc-2261-471d-9053-51ee965850bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:38.556808 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:38.550559    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh" podUID="2d3647f2-89e9-4908-a8ee-d84e527abeb0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:38.574792 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:38.574721    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/premium-w1-6fc748-kserve-7474db9cd7-nf2xk" podUID="830e062e-83fc-4de7-91a7-5534dac1d46b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Nov 03 20:20:38.815801 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:38.799617    5570 patch_prober.go:29] interesting pod/e2e-worker-w0-e40816-openshift-default-744c57967c-5bn7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body=
Nov 03 20:20:38.816111 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:38.816075    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-5bn7t" podUID="26831c71-2091-4090-843d-f7268d257114" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused"
Nov 03 20:20:38.975296 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:38.975258    5570 patch_prober.go:29] interesting pod/e2e-worker-w2-9973d9-openshift-default-ffb6475-c98nc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Nov 03 20:20:39.039557 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:39.039525    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-e80203/distinct-2-w4-e80203-kserve-756f9b9bc6-69gxj"
Nov 03 20:20:39.045825 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:39.045795    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-9e1bcb/premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh"
Nov 03 20:20:39.085195 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:39.081874    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e40816/distinct-2-w0-e40816-kserve-79fb46654d-s242x"
Nov 03 20:20:39.085195 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:39.085123    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6fc748/premium-w1-6fc748-kserve-7474db9cd7-nf2xk"
Nov 03 20:20:39.098798 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:38.980471    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-c98nc" podUID="755f6eab-5d03-4368-9670-b1402084474c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Nov 03 20:20:39.098798 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:39.095886    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e40816/simulator-w0-e40816-kserve-688f79796f-2tvw5"
Nov 03 20:20:39.102678 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:39.100865    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz"
Nov 03 20:20:39.147830 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:39.144679    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-482722/premium-w3-482722-kserve-689dc7bd5d-x8rjp"
Nov 03 20:20:39.217853 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:39.211907    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-2-w6-9e1bcb-kserve-7d96d9f6f6-4f7fl"
Nov 03 20:20:39.217853 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:39.212993    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/premium-w2-9973d9-kserve-5c6dd4669c-5gv8x" podUID="cefc89ad-7ae1-433b-979d-6601678a6d98" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Nov 03 20:20:39.274339 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:39.274303    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g69f8"]
Nov 03 20:20:39.323825 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:39.315894    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/premium-w4-e80203-kserve-5f9f445456-bms9d" podUID="6ad76efc-20c2-4951-b22a-c5a39cd39ed4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Nov 03 20:20:39.323825 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:39.323129    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9973d9/premium-w2-9973d9-kserve-5c6dd4669c-5gv8x"
Nov 03 20:20:39.360809 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:39.360178    5570 patch_prober.go:29] interesting pod/e2e-worker-w3-482722-openshift-default-7645487df-xszg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Nov 03 20:20:39.360809 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:39.360235    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-xszg4" podUID="4ee1819e-4649-4d3e-9cfc-0d2d72c1b481" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Nov 03 20:20:39.458129 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:39.458092    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-xcxnz"
Nov 03 20:20:39.502055 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:39.502024    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-xcxnz"
Nov 03 20:20:39.533998 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:39.533966    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-xcxnz"
Nov 03 20:20:39.551063 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:39.550129    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/distinct-w3-482722-kserve-d876967c-kd488" podUID="9a97f9cc-2261-471d-9053-51ee965850bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:39.560007 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:39.559967    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-e80203/premium-w4-e80203-kserve-5f9f445456-bms9d"
Nov 03 20:20:39.563928 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:39.563891    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/distinct-2-w3-482722-kserve-74bf58cf7d-v9cjw" podUID="9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:39.666853 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:39.649537    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-482722/distinct-2-w3-482722-kserve-74bf58cf7d-v9cjw"
Nov 03 20:20:39.685172 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:39.685069    5570 patch_prober.go:29] interesting pod/e2e-worker-w0-e40816-openshift-default-744c57967c-5bn7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body=
Nov 03 20:20:39.685476 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:39.685447    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-5bn7t" podUID="26831c71-2091-4090-843d-f7268d257114" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused"
Nov 03 20:20:39.692321 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:39.687927    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-482722/distinct-w3-482722-kserve-d876967c-kd488"
Nov 03 20:20:40.001415 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:40.001381    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-c98nc"
Nov 03 20:20:40.003176 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:40.003154    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-c98nc"
Nov 03 20:20:40.005116 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:40.005100    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-c98nc"
Nov 03 20:20:41.226925 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:41.226878    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/distinct-2-w0-e40816-kserve-79fb46654d-s242x" podUID="4ad9b048-925c-4d01-a839-a5f25d4dd02d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:41.266920 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:41.266868    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/premium-w4-e80203-kserve-5f9f445456-bms9d" podUID="6ad76efc-20c2-4951-b22a-c5a39cd39ed4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Nov 03 20:20:41.409663 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:41.409621    5570 patch_prober.go:29] interesting pod/e2e-worker-w3-482722-openshift-default-7645487df-xszg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:20:41.409960 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:41.409928    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-xszg4" podUID="4ee1819e-4649-4d3e-9cfc-0d2d72c1b481" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:41.669268 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:41.668427    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/distinct-2-w3-482722-kserve-74bf58cf7d-v9cjw" podUID="9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:41.706832 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:41.706500    5570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-482722/distinct-2-w3-482722-kserve-74bf58cf7d-v9cjw" podUID="9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:41.750584 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:41.749880    5570 patch_prober.go:29] interesting pod/e2e-worker-w0-e40816-openshift-default-744c57967c-5bn7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:20:41.750584 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:41.749953    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-5bn7t" podUID="26831c71-2091-4090-843d-f7268d257114" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:41.827483 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:41.792532    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz" podUID="3a715ef1-38e9-4386-9eec-e10e8df11a10" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:41.827483 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:41.794003    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/premium-w3-482722-kserve-689dc7bd5d-x8rjp" podUID="369045c3-4d6a-49a7-9bc6-f2939883bee8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Nov 03 20:20:41.827952 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:41.827907    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/distinct-w3-482722-kserve-d876967c-kd488" podUID="9a97f9cc-2261-471d-9053-51ee965850bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:41.879911 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:41.828270    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/simulator-w0-e40816-kserve-688f79796f-2tvw5" podUID="c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:41.879911 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:41.829000    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-2-w6-9e1bcb-kserve-7d96d9f6f6-4f7fl" podUID="56353b0f-08ee-4ada-9dc0-ac5afd5a59c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded"
Nov 03 20:20:41.879911 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:41.829025    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/premium-w1-6fc748-kserve-7474db9cd7-nf2xk" podUID="830e062e-83fc-4de7-91a7-5534dac1d46b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Nov 03 20:20:42.076800 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:42.072905    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/premium-w2-9973d9-kserve-5c6dd4669c-5gv8x" podUID="cefc89ad-7ae1-433b-979d-6601678a6d98" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:42.252946 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:42.252898    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/distinct-2-w0-e40816-kserve-79fb46654d-s242x" podUID="4ad9b048-925c-4d01-a839-a5f25d4dd02d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:42.318388 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:42.270067    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/premium-w4-e80203-kserve-5f9f445456-bms9d" podUID="6ad76efc-20c2-4951-b22a-c5a39cd39ed4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:42.623501 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:42.416200    5570 patch_prober.go:29] interesting pod/e2e-worker-w3-482722-openshift-default-7645487df-xszg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:20:42.623977 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:42.623840    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-xszg4" podUID="4ee1819e-4649-4d3e-9cfc-0d2d72c1b481" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:42.640810 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:42.536218    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-wj4gp"]
Nov 03 20:20:42.643503 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:42.643464    5570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-e40816/distinct-2-w0-e40816-kserve-79fb46654d-s242x" podUID="4ad9b048-925c-4d01-a839-a5f25d4dd02d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/health\": context deadline exceeded"
Nov 03 20:20:42.677799 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:42.644834    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh" podUID="2d3647f2-89e9-4908-a8ee-d84e527abeb0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Nov 03 20:20:42.679567 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:42.679531    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-wj4gp"]
Nov 03 20:20:42.679907 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:42.679887    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e40816/distinct-2-w0-e40816-kserve-79fb46654d-s242x"
Nov 03 20:20:42.680074 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:42.680061    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-xszg4"
Nov 03 20:20:42.680256 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:42.680242    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-e80203/premium-w4-e80203-kserve-5f9f445456-bms9d"
Nov 03 20:20:42.681986 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:42.681963    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-wj4gp"
Nov 03 20:20:42.740705 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:42.684507    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-5bn7t"
Nov 03 20:20:43.039942 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:42.723648    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/distinct-2-w3-482722-kserve-74bf58cf7d-v9cjw" podUID="9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded"
Nov 03 20:20:43.057799 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:42.795263    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/premium-w3-482722-kserve-689dc7bd5d-x8rjp" podUID="369045c3-4d6a-49a7-9bc6-f2939883bee8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:43.060244 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:43.060214    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-482722/distinct-2-w3-482722-kserve-74bf58cf7d-v9cjw"
Nov 03 20:20:43.060932 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:43.060910    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-482722/premium-w3-482722-kserve-689dc7bd5d-x8rjp"
Nov 03 20:20:43.179797 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:42.907446    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/simulator-w0-e40816-kserve-688f79796f-2tvw5" podUID="c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Nov 03 20:20:43.179797 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:42.907671    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-2-w6-9e1bcb-kserve-7d96d9f6f6-4f7fl" podUID="56353b0f-08ee-4ada-9dc0-ac5afd5a59c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded"
Nov 03 20:20:43.181067 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:43.181028    5570 patch_prober.go:29] interesting pod/e2e-worker-w0-e40816-openshift-default-744c57967c-5bn7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:20:43.181637 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:43.181599    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-5bn7t" podUID="26831c71-2091-4090-843d-f7268d257114" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:43.182843 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:42.907223    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/premium-w1-6fc748-kserve-7474db9cd7-nf2xk" podUID="830e062e-83fc-4de7-91a7-5534dac1d46b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:43.193014 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:43.093321    5570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-6fc748/premium-w1-6fc748-kserve-7474db9cd7-nf2xk" podUID="830e062e-83fc-4de7-91a7-5534dac1d46b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:43.199911 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:43.110003    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/distinct-w3-482722-kserve-d876967c-kd488" podUID="9a97f9cc-2261-471d-9053-51ee965850bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:43.209848 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:43.103666    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/premium-w2-9973d9-kserve-5c6dd4669c-5gv8x" podUID="cefc89ad-7ae1-433b-979d-6601678a6d98" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:43.231816 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:43.224982    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/distinct-2-w4-e80203-kserve-756f9b9bc6-69gxj" podUID="ccf5dc2f-f116-4917-bb1a-b136d8ed17f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:43.231816 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:42.795332    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz" podUID="3a715ef1-38e9-4386-9eec-e10e8df11a10" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:43.231816 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:43.193363    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e40816/simulator-w0-e40816-kserve-688f79796f-2tvw5"
Nov 03 20:20:43.231816 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:43.226598    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-2-w6-9e1bcb-kserve-7d96d9f6f6-4f7fl"
Nov 03 20:20:43.275317 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:43.259870    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6fc748/premium-w1-6fc748-kserve-7474db9cd7-nf2xk"
Nov 03 20:20:43.275317 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:43.260192    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-xszg4"
Nov 03 20:20:43.275317 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:43.260327    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-482722/distinct-w3-482722-kserve-d876967c-kd488"
Nov 03 20:20:43.275317 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:43.260409    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9973d9/premium-w2-9973d9-kserve-5c6dd4669c-5gv8x"
Nov 03 20:20:43.275317 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:43.260496    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz"
Nov 03 20:20:43.317531 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:43.317433    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/distinct-2-w0-e40816-kserve-79fb46654d-s242x" podUID="4ad9b048-925c-4d01-a839-a5f25d4dd02d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:43.669542 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:43.610182    5570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-e80203/premium-w4-e80203-kserve-5f9f445456-bms9d" podUID="6ad76efc-20c2-4951-b22a-c5a39cd39ed4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:43.669851 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:43.635926    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce7b1d98-c21f-4412-97ce-9dca59ce2649-istiod-ca-cert\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-wj4gp\" (UID: \"ce7b1d98-c21f-4412-97ce-9dca59ce2649\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-wj4gp"
Nov 03 20:20:43.670109 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:43.670084    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce7b1d98-c21f-4412-97ce-9dca59ce2649-istio-token\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-wj4gp\" (UID: \"ce7b1d98-c21f-4412-97ce-9dca59ce2649\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-wj4gp"
Nov 03 20:20:43.670312 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:43.670277    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce7b1d98-c21f-4412-97ce-9dca59ce2649-credential-socket\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-wj4gp\" (UID: \"ce7b1d98-c21f-4412-97ce-9dca59ce2649\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-wj4gp"
Nov 03 20:20:43.670920 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:43.670895    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce7b1d98-c21f-4412-97ce-9dca59ce2649-workload-certs\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-wj4gp\" (UID: \"ce7b1d98-c21f-4412-97ce-9dca59ce2649\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-wj4gp"
Nov 03 20:20:43.671131 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:43.671096    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce7b1d98-c21f-4412-97ce-9dca59ce2649-workload-socket\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-wj4gp\" (UID: \"ce7b1d98-c21f-4412-97ce-9dca59ce2649\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-wj4gp"
Nov 03 20:20:43.696837 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:43.673188    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce7b1d98-c21f-4412-97ce-9dca59ce2649-istio-data\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-wj4gp\" (UID: \"ce7b1d98-c21f-4412-97ce-9dca59ce2649\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-wj4gp"
Nov 03 20:20:43.696837 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:43.673390    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce7b1d98-c21f-4412-97ce-9dca59ce2649-istio-podinfo\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-wj4gp\" (UID: \"ce7b1d98-c21f-4412-97ce-9dca59ce2649\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-wj4gp"
Nov 03 20:20:43.696837 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:43.673439    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56hdq\" (UniqueName: \"kubernetes.io/projected/ce7b1d98-c21f-4412-97ce-9dca59ce2649-kube-api-access-56hdq\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-wj4gp\" (UID: \"ce7b1d98-c21f-4412-97ce-9dca59ce2649\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-wj4gp"
Nov 03 20:20:43.696837 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:43.673544    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce7b1d98-c21f-4412-97ce-9dca59ce2649-istio-envoy\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-wj4gp\" (UID: \"ce7b1d98-c21f-4412-97ce-9dca59ce2649\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-wj4gp"
Nov 03 20:20:43.739791 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:43.739727    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/premium-w4-e80203-kserve-5f9f445456-bms9d" podUID="6ad76efc-20c2-4951-b22a-c5a39cd39ed4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Nov 03 20:20:43.777847 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:43.772661    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh" podUID="2d3647f2-89e9-4908-a8ee-d84e527abeb0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Nov 03 20:20:43.865462 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:43.862476    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce7b1d98-c21f-4412-97ce-9dca59ce2649-workload-certs\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-wj4gp\" (UID: \"ce7b1d98-c21f-4412-97ce-9dca59ce2649\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-wj4gp"
Nov 03 20:20:43.875801 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:43.865887    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-9e1bcb/premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh"
Nov 03 20:20:43.889005 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:43.823819    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce7b1d98-c21f-4412-97ce-9dca59ce2649-workload-certs\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-wj4gp\" (UID: \"ce7b1d98-c21f-4412-97ce-9dca59ce2649\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-wj4gp"
Nov 03 20:20:43.889367 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:43.889334    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce7b1d98-c21f-4412-97ce-9dca59ce2649-workload-socket\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-wj4gp\" (UID: \"ce7b1d98-c21f-4412-97ce-9dca59ce2649\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-wj4gp"
Nov 03 20:20:43.889646 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:43.889626    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce7b1d98-c21f-4412-97ce-9dca59ce2649-istio-data\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-wj4gp\" (UID: \"ce7b1d98-c21f-4412-97ce-9dca59ce2649\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-wj4gp"
Nov 03 20:20:43.889879 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:43.889860    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce7b1d98-c21f-4412-97ce-9dca59ce2649-istio-podinfo\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-wj4gp\" (UID: \"ce7b1d98-c21f-4412-97ce-9dca59ce2649\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-wj4gp"
Nov 03 20:20:43.895800 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:43.891909    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56hdq\" (UniqueName: \"kubernetes.io/projected/ce7b1d98-c21f-4412-97ce-9dca59ce2649-kube-api-access-56hdq\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-wj4gp\" (UID: \"ce7b1d98-c21f-4412-97ce-9dca59ce2649\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-wj4gp"
Nov 03 20:20:43.896241 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:43.896214    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce7b1d98-c21f-4412-97ce-9dca59ce2649-istio-envoy\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-wj4gp\" (UID: \"ce7b1d98-c21f-4412-97ce-9dca59ce2649\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-wj4gp"
Nov 03 20:20:43.901944 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:43.901907    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce7b1d98-c21f-4412-97ce-9dca59ce2649-istiod-ca-cert\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-wj4gp\" (UID: \"ce7b1d98-c21f-4412-97ce-9dca59ce2649\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-wj4gp"
Nov 03 20:20:43.902171 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:43.902153    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce7b1d98-c21f-4412-97ce-9dca59ce2649-istio-token\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-wj4gp\" (UID: \"ce7b1d98-c21f-4412-97ce-9dca59ce2649\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-wj4gp"
Nov 03 20:20:43.967801 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:43.964537    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce7b1d98-c21f-4412-97ce-9dca59ce2649-credential-socket\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-wj4gp\" (UID: \"ce7b1d98-c21f-4412-97ce-9dca59ce2649\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-wj4gp"
Nov 03 20:20:44.021955 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:43.921852    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce7b1d98-c21f-4412-97ce-9dca59ce2649-istio-data\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-wj4gp\" (UID: \"ce7b1d98-c21f-4412-97ce-9dca59ce2649\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-wj4gp"
Nov 03 20:20:44.114439 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:43.921291    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce7b1d98-c21f-4412-97ce-9dca59ce2649-workload-socket\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-wj4gp\" (UID: \"ce7b1d98-c21f-4412-97ce-9dca59ce2649\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-wj4gp"
Nov 03 20:20:44.114728 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:44.022702    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce7b1d98-c21f-4412-97ce-9dca59ce2649-credential-socket\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-wj4gp\" (UID: \"ce7b1d98-c21f-4412-97ce-9dca59ce2649\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-wj4gp"
Nov 03 20:20:44.114922 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:44.087911    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/distinct-2-w3-482722-kserve-74bf58cf7d-v9cjw" podUID="9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:44.116145 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:44.114278    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/premium-w3-482722-kserve-689dc7bd5d-x8rjp" podUID="369045c3-4d6a-49a7-9bc6-f2939883bee8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:44.190504 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:44.190468    5570 patch_prober.go:29] interesting pod/e2e-worker-w0-e40816-openshift-default-744c57967c-5bn7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:20:44.214243 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:44.214212    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-5bn7t" podUID="26831c71-2091-4090-843d-f7268d257114" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:44.450037 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:44.201878    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-2-w6-9e1bcb-kserve-7d96d9f6f6-4f7fl" podUID="56353b0f-08ee-4ada-9dc0-ac5afd5a59c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded"
Nov 03 20:20:44.466800 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:44.204178    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/distinct-w3-482722-kserve-d876967c-kd488" podUID="9a97f9cc-2261-471d-9053-51ee965850bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded"
Nov 03 20:20:44.519805 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:44.214092    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/premium-w2-9973d9-kserve-5c6dd4669c-5gv8x" podUID="cefc89ad-7ae1-433b-979d-6601678a6d98" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Nov 03 20:20:44.520497 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:44.214142    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/premium-w1-6fc748-kserve-7474db9cd7-nf2xk" podUID="830e062e-83fc-4de7-91a7-5534dac1d46b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:44.520985 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:44.192541    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/simulator-w0-e40816-kserve-688f79796f-2tvw5" podUID="c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:44.521353 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:44.266421    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce7b1d98-c21f-4412-97ce-9dca59ce2649-istio-podinfo\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-wj4gp\" (UID: \"ce7b1d98-c21f-4412-97ce-9dca59ce2649\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-wj4gp"
Nov 03 20:20:44.521524 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:44.266917    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce7b1d98-c21f-4412-97ce-9dca59ce2649-istiod-ca-cert\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-wj4gp\" (UID: \"ce7b1d98-c21f-4412-97ce-9dca59ce2649\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-wj4gp"
Nov 03 20:20:44.521650 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:44.337259    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz" podUID="3a715ef1-38e9-4386-9eec-e10e8df11a10" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:44.522052 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:44.338534    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/distinct-2-w4-e80203-kserve-756f9b9bc6-69gxj" podUID="ccf5dc2f-f116-4917-bb1a-b136d8ed17f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded"
Nov 03 20:20:44.646128 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:44.646089    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce7b1d98-c21f-4412-97ce-9dca59ce2649-istio-envoy\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-wj4gp\" (UID: \"ce7b1d98-c21f-4412-97ce-9dca59ce2649\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-wj4gp"
Nov 03 20:20:44.647973 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:44.647931    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56hdq\" (UniqueName: \"kubernetes.io/projected/ce7b1d98-c21f-4412-97ce-9dca59ce2649-kube-api-access-56hdq\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-wj4gp\" (UID: \"ce7b1d98-c21f-4412-97ce-9dca59ce2649\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-wj4gp"
Nov 03 20:20:44.694499 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:44.694437    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/distinct-2-w0-e40816-kserve-79fb46654d-s242x" podUID="4ad9b048-925c-4d01-a839-a5f25d4dd02d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:44.695843 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:44.695617    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce7b1d98-c21f-4412-97ce-9dca59ce2649-istio-token\") pod \"e2e-worker-w0-e40816-openshift-default-744c57967c-wj4gp\" (UID: \"ce7b1d98-c21f-4412-97ce-9dca59ce2649\") " pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-wj4gp"
Nov 03 20:20:44.777004 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:44.776885    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/premium-w4-e80203-kserve-5f9f445456-bms9d" podUID="6ad76efc-20c2-4951-b22a-c5a39cd39ed4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:44.843979 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:44.843922    5570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-9973d9/premium-w2-9973d9-kserve-5c6dd4669c-5gv8x" podUID="cefc89ad-7ae1-433b-979d-6601678a6d98" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:44.986916 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:44.955012    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-xszg4"
Nov 03 20:20:45.381801 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:44.988665    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-wj4gp"
Nov 03 20:20:45.381801 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:44.988734    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh" podUID="2d3647f2-89e9-4908-a8ee-d84e527abeb0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:45.381801 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:45.135464    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/premium-w3-482722-kserve-689dc7bd5d-x8rjp" podUID="369045c3-4d6a-49a7-9bc6-f2939883bee8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:45.381801 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:45.135908    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/distinct-2-w3-482722-kserve-74bf58cf7d-v9cjw" podUID="9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded"
Nov 03 20:20:45.510236 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:45.510182    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-2-w6-9e1bcb-kserve-7d96d9f6f6-4f7fl" podUID="56353b0f-08ee-4ada-9dc0-ac5afd5a59c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:45.526804 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:45.511424    5570 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz" podUID="3a715ef1-38e9-4386-9eec-e10e8df11a10" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:45.566338 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:45.529117    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/distinct-2-w4-e80203-kserve-756f9b9bc6-69gxj" podUID="ccf5dc2f-f116-4917-bb1a-b136d8ed17f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:45.567899 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:45.567845    5570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-482722/premium-w3-482722-kserve-689dc7bd5d-x8rjp" podUID="369045c3-4d6a-49a7-9bc6-f2939883bee8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:45.568094 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:45.540028    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/premium-w1-6fc748-kserve-7474db9cd7-nf2xk" podUID="830e062e-83fc-4de7-91a7-5534dac1d46b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:45.569115 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:45.540061    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/distinct-w3-482722-kserve-d876967c-kd488" podUID="9a97f9cc-2261-471d-9053-51ee965850bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded"
Nov 03 20:20:45.571845 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:45.540756    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/premium-w2-9973d9-kserve-5c6dd4669c-5gv8x" podUID="cefc89ad-7ae1-433b-979d-6601678a6d98" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Nov 03 20:20:45.583830 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:45.538814    5570 patch_prober.go:29] interesting pod/e2e-worker-w0-e40816-openshift-default-744c57967c-5bn7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:20:45.583830 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:45.580441    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-5bn7t" podUID="26831c71-2091-4090-843d-f7268d257114" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:45.584134 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:45.540867    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz" podUID="3a715ef1-38e9-4386-9eec-e10e8df11a10" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": context deadline exceeded"
Nov 03 20:20:45.629938 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:45.540887    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/simulator-w0-e40816-kserve-688f79796f-2tvw5" podUID="c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Nov 03 20:20:45.632109 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:45.541649    5570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-482722/distinct-w3-482722-kserve-d876967c-kd488" podUID="9a97f9cc-2261-471d-9053-51ee965850bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:45.636056 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:45.636030    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-e80203/distinct-2-w4-e80203-kserve-756f9b9bc6-69gxj"
Nov 03 20:20:45.714902 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:45.714566    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/distinct-2-w0-e40816-kserve-79fb46654d-s242x" podUID="4ad9b048-925c-4d01-a839-a5f25d4dd02d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:45.816768 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:45.815343    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/premium-w4-e80203-kserve-5f9f445456-bms9d" podUID="6ad76efc-20c2-4951-b22a-c5a39cd39ed4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:46.084950 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:46.079297    5570 patch_prober.go:29] interesting pod/e2e-worker-w0-e40816-openshift-default-744c57967c-jjjr7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:20:46.084950 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:46.079365    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-jjjr7" podUID="7ca8412e-c6de-4cb5-944e-ba8afd2fa791" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:46.471798 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:46.471017    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh" podUID="2d3647f2-89e9-4908-a8ee-d84e527abeb0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Nov 03 20:20:46.495606 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:46.485878    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/premium-w3-482722-kserve-689dc7bd5d-x8rjp" podUID="369045c3-4d6a-49a7-9bc6-f2939883bee8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Nov 03 20:20:46.495606 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:46.490331    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/distinct-2-w3-482722-kserve-74bf58cf7d-v9cjw" podUID="9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:46.537023 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:46.536956    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-2-w6-9e1bcb-kserve-7d96d9f6f6-4f7fl" podUID="56353b0f-08ee-4ada-9dc0-ac5afd5a59c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:46.589379 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:46.571896    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/distinct-w3-482722-kserve-d876967c-kd488" podUID="9a97f9cc-2261-471d-9053-51ee965850bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:46.617837 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:46.592899    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz" podUID="3a715ef1-38e9-4386-9eec-e10e8df11a10" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:46.617837 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:46.593317    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/premium-w1-6fc748-kserve-7474db9cd7-nf2xk" podUID="830e062e-83fc-4de7-91a7-5534dac1d46b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Nov 03 20:20:46.617837 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:46.593614    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/premium-w2-9973d9-kserve-5c6dd4669c-5gv8x" podUID="cefc89ad-7ae1-433b-979d-6601678a6d98" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Nov 03 20:20:46.701308 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:46.701256    5570 patch_prober.go:29] interesting pod/e2e-worker-w0-e40816-openshift-default-744c57967c-5bn7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:20:46.893484 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:46.771256    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-5bn7t" podUID="26831c71-2091-4090-843d-f7268d257114" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:47.063288 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:46.884224    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/premium-w4-e80203-kserve-5f9f445456-bms9d" podUID="6ad76efc-20c2-4951-b22a-c5a39cd39ed4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Nov 03 20:20:47.071869 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:46.919800    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/simulator-w0-e40816-kserve-688f79796f-2tvw5" podUID="c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Nov 03 20:20:47.076805 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:46.926076    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/distinct-2-w0-e40816-kserve-79fb46654d-s242x" podUID="4ad9b048-925c-4d01-a839-a5f25d4dd02d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded"
Nov 03 20:20:47.110852 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:46.927546    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/distinct-2-w4-e80203-kserve-756f9b9bc6-69gxj" podUID="ccf5dc2f-f116-4917-bb1a-b136d8ed17f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded"
Nov 03 20:20:47.120390 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:47.120268    5570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-2-w6-9e1bcb-kserve-7d96d9f6f6-4f7fl" podUID="56353b0f-08ee-4ada-9dc0-ac5afd5a59c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:47.495878 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:47.495826    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/distinct-2-w3-482722-kserve-74bf58cf7d-v9cjw" podUID="9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:47.497423 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:47.497384    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh" podUID="2d3647f2-89e9-4908-a8ee-d84e527abeb0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:47.518824 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:47.502932    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/premium-w3-482722-kserve-689dc7bd5d-x8rjp" podUID="369045c3-4d6a-49a7-9bc6-f2939883bee8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:47.545822 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:47.538335    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-2-w6-9e1bcb-kserve-7d96d9f6f6-4f7fl" podUID="56353b0f-08ee-4ada-9dc0-ac5afd5a59c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:47.579821 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:47.576271    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/distinct-w3-482722-kserve-d876967c-kd488" podUID="9a97f9cc-2261-471d-9053-51ee965850bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:47.602148 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:47.602102    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/premium-w2-9973d9-kserve-5c6dd4669c-5gv8x" podUID="cefc89ad-7ae1-433b-979d-6601678a6d98" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:47.603701 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:47.603665    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz" podUID="3a715ef1-38e9-4386-9eec-e10e8df11a10" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": context deadline exceeded"
Nov 03 20:20:47.604895 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:47.604871    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/premium-w1-6fc748-kserve-7474db9cd7-nf2xk" podUID="830e062e-83fc-4de7-91a7-5534dac1d46b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Nov 03 20:20:47.948798 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:47.946877    5570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-9e1bcb/premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh" podUID="2d3647f2-89e9-4908-a8ee-d84e527abeb0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:48.068472 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:48.068425    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/premium-w4-e80203-kserve-5f9f445456-bms9d" podUID="6ad76efc-20c2-4951-b22a-c5a39cd39ed4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:48.093815 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:48.087675    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/distinct-2-w0-e40816-kserve-79fb46654d-s242x" podUID="4ad9b048-925c-4d01-a839-a5f25d4dd02d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:48.112975 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:48.112931    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/distinct-2-w4-e80203-kserve-756f9b9bc6-69gxj" podUID="ccf5dc2f-f116-4917-bb1a-b136d8ed17f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:48.131807 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:48.113982    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/simulator-w0-e40816-kserve-688f79796f-2tvw5" podUID="c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:48.384804 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:48.379707    5570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-e40816/simulator-w0-e40816-kserve-688f79796f-2tvw5" podUID="c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:48.384804 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:48.384591    5570 patch_prober.go:29] interesting pod/e2e-worker-w0-e40816-openshift-default-744c57967c-5bn7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:20:48.384804 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:48.384650    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-5bn7t" podUID="26831c71-2091-4090-843d-f7268d257114" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:48.469752 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:48.459628    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-482722/premium-w3-482722-kserve-689dc7bd5d-x8rjp"
Nov 03 20:20:48.529095 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:48.511822    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh" podUID="2d3647f2-89e9-4908-a8ee-d84e527abeb0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Nov 03 20:20:48.529095 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:48.512233    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/distinct-2-w3-482722-kserve-74bf58cf7d-v9cjw" podUID="9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded"
Nov 03 20:20:48.560253 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:48.542140    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-2-w6-9e1bcb-kserve-7d96d9f6f6-4f7fl" podUID="56353b0f-08ee-4ada-9dc0-ac5afd5a59c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:48.597224 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:48.593022    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/distinct-w3-482722-kserve-d876967c-kd488" podUID="9a97f9cc-2261-471d-9053-51ee965850bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded"
Nov 03 20:20:48.623886 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:48.622427    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/premium-w1-6fc748-kserve-7474db9cd7-nf2xk" podUID="830e062e-83fc-4de7-91a7-5534dac1d46b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:48.623886 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:48.622545    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/premium-w2-9973d9-kserve-5c6dd4669c-5gv8x" podUID="cefc89ad-7ae1-433b-979d-6601678a6d98" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:48.623886 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:48.622611    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz" podUID="3a715ef1-38e9-4386-9eec-e10e8df11a10" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": context deadline exceeded"
Nov 03 20:20:48.946039 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:48.942917    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz"
Nov 03 20:20:48.979848 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:48.978653    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-e80203/distinct-2-w4-e80203-kserve-756f9b9bc6-69gxj"
Nov 03 20:20:48.979848 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:48.978716    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-e80203/premium-w4-e80203-kserve-5f9f445456-bms9d"
Nov 03 20:20:48.979848 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:48.978737    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-wj4gp"]
Nov 03 20:20:48.979848 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:48.978851    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-9e1bcb/premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh"
Nov 03 20:20:48.979848 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:48.978883    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e40816/simulator-w0-e40816-kserve-688f79796f-2tvw5"
Nov 03 20:20:48.979848 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:48.978916    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-482722/distinct-2-w3-482722-kserve-74bf58cf7d-v9cjw"
Nov 03 20:20:48.979848 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:48.979001    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-482722/distinct-w3-482722-kserve-d876967c-kd488"
Nov 03 20:20:48.979848 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:48.979032    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6fc748/premium-w1-6fc748-kserve-7474db9cd7-nf2xk"
Nov 03 20:20:48.979848 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:48.979350    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-2-w6-9e1bcb-kserve-7d96d9f6f6-4f7fl"
Nov 03 20:20:48.989809 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:48.981933    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9973d9/premium-w2-9973d9-kserve-5c6dd4669c-5gv8x"
Nov 03 20:20:48.991393 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:48.991372    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e40816/distinct-2-w0-e40816-kserve-79fb46654d-s242x"
Nov 03 20:20:49.014120 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:49.014072    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:20:49.014399 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:49.014377    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:20:49.014579 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:49.014560    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:20:49.247653 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:49.244255    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-5bn7t"
Nov 03 20:20:49.247653 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:49.246691    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-5bn7t"
Nov 03 20:20:49.544311 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:49.544266    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3da0091e-b6g97"]
Nov 03 20:20:49.558009 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:49.557978    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3da0091e-b6g97"]
Nov 03 20:20:49.558252 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:49.558093    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3da0091e-b6g97"
Nov 03 20:20:49.578639 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:49.578605    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:20:49.579343 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:49.579321    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l7s5w\""
Nov 03 20:20:49.796984 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:49.788503    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwzlj\" (UniqueName: \"kubernetes.io/projected/72de3f5b-77f3-4631-ad8d-6065ad327287-kube-api-access-fwzlj\") pod \"maas-api-revoke-sub-3da0091e-b6g97\" (UID: \"72de3f5b-77f3-4631-ad8d-6065ad327287\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3da0091e-b6g97"
Nov 03 20:20:49.796984 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:49.789523    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72de3f5b-77f3-4631-ad8d-6065ad327287-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3da0091e-b6g97\" (UID: \"72de3f5b-77f3-4631-ad8d-6065ad327287\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3da0091e-b6g97"
Nov 03 20:20:49.796984 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:49.790616    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72de3f5b-77f3-4631-ad8d-6065ad327287-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3da0091e-b6g97\" (UID: \"72de3f5b-77f3-4631-ad8d-6065ad327287\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3da0091e-b6g97"
Nov 03 20:20:49.836717 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:49.836682    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-wj4gp" event={"ID":"ce7b1d98-c21f-4412-97ce-9dca59ce2649","Type":"ContainerStarted","Data":"2065ee88578f7048629b049802360f906227df9654abe61d45dbb67e25165dbb"}
Nov 03 20:20:49.837111 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:49.837088    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-wj4gp" event={"ID":"ce7b1d98-c21f-4412-97ce-9dca59ce2649","Type":"ContainerStarted","Data":"8e857f2889f0ace42cbd1f2f2fb3080b52dd87372ddeb2e190b2666a84690a0e"}
Nov 03 20:20:49.890604 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:49.890548    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-wj4gp" podStartSLOduration=7.890533615 podStartE2EDuration="7.890533615s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:20:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:20:49.887509742 +0000 UTC m=+984.040171629" watchObservedRunningTime="2026-11-03 20:20:49.890533615 +0000 UTC m=+984.043195539"
Nov 03 20:20:49.892545 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:49.892513    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72de3f5b-77f3-4631-ad8d-6065ad327287-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3da0091e-b6g97\" (UID: \"72de3f5b-77f3-4631-ad8d-6065ad327287\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3da0091e-b6g97"
Nov 03 20:20:49.892873 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:49.892844    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwzlj\" (UniqueName: \"kubernetes.io/projected/72de3f5b-77f3-4631-ad8d-6065ad327287-kube-api-access-fwzlj\") pod \"maas-api-revoke-sub-3da0091e-b6g97\" (UID: \"72de3f5b-77f3-4631-ad8d-6065ad327287\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3da0091e-b6g97"
Nov 03 20:20:49.894074 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:49.893255    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72de3f5b-77f3-4631-ad8d-6065ad327287-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3da0091e-b6g97\" (UID: \"72de3f5b-77f3-4631-ad8d-6065ad327287\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3da0091e-b6g97"
Nov 03 20:20:49.894175 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:49.894089    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72de3f5b-77f3-4631-ad8d-6065ad327287-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3da0091e-b6g97\" (UID: \"72de3f5b-77f3-4631-ad8d-6065ad327287\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3da0091e-b6g97"
Nov 03 20:20:49.895929 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:49.895893    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72de3f5b-77f3-4631-ad8d-6065ad327287-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3da0091e-b6g97\" (UID: \"72de3f5b-77f3-4631-ad8d-6065ad327287\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3da0091e-b6g97"
Nov 03 20:20:49.913685 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:49.913651    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwzlj\" (UniqueName: \"kubernetes.io/projected/72de3f5b-77f3-4631-ad8d-6065ad327287-kube-api-access-fwzlj\") pod \"maas-api-revoke-sub-3da0091e-b6g97\" (UID: \"72de3f5b-77f3-4631-ad8d-6065ad327287\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3da0091e-b6g97"
Nov 03 20:20:49.924358 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:49.924331    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3da0091e-b6g97"
Nov 03 20:20:49.989736 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:49.989705    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-wj4gp"
Nov 03 20:20:50.254799 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:50.248099    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3da0091e-b6g97"]
Nov 03 20:20:50.871504 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:50.871464    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3da0091e-b6g97" event={"ID":"72de3f5b-77f3-4631-ad8d-6065ad327287","Type":"ContainerStarted","Data":"886b41422b9f1b0fd3151bcd88be90244c135760b792a86285fd44871d58231a"}
Nov 03 20:20:50.872071 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:50.871515    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3da0091e-b6g97" event={"ID":"72de3f5b-77f3-4631-ad8d-6065ad327287","Type":"ContainerStarted","Data":"a827ebd19189a5bfd6278c2c4835e6cab992e53e0a3c813254fd5a8c368532d6"}
Nov 03 20:20:50.917346 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:50.917292    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3da0091e-b6g97" podStartSLOduration=1.917278546 podStartE2EDuration="1.917278546s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:20:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:20:50.912958138 +0000 UTC m=+985.065620015" watchObservedRunningTime="2026-11-03 20:20:50.917278546 +0000 UTC m=+985.069940433"
Nov 03 20:20:50.991850 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:50.991811    5570 patch_prober.go:29] interesting pod/e2e-worker-w0-e40816-openshift-default-744c57967c-wj4gp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Nov 03 20:20:50.992044 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:50.991876    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-wj4gp" podUID="ce7b1d98-c21f-4412-97ce-9dca59ce2649" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Nov 03 20:20:51.880983 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:51.880945    5570 generic.go:356] "Generic (PLEG): container finished" podID="72de3f5b-77f3-4631-ad8d-6065ad327287" containerID="886b41422b9f1b0fd3151bcd88be90244c135760b792a86285fd44871d58231a" exitCode=0
Nov 03 20:20:51.881462 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:51.881017    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3da0091e-b6g97" event={"ID":"72de3f5b-77f3-4631-ad8d-6065ad327287","Type":"ContainerDied","Data":"886b41422b9f1b0fd3151bcd88be90244c135760b792a86285fd44871d58231a"}
Nov 03 20:20:51.990117 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:51.990085    5570 patch_prober.go:29] interesting pod/e2e-worker-w0-e40816-openshift-default-744c57967c-wj4gp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Nov 03 20:20:51.990296 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:51.990138    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-wj4gp" podUID="ce7b1d98-c21f-4412-97ce-9dca59ce2649" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Nov 03 20:20:52.990330 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:52.990293    5570 patch_prober.go:29] interesting pod/e2e-worker-w0-e40816-openshift-default-744c57967c-wj4gp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Nov 03 20:20:52.990712 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:52.990361    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-wj4gp" podUID="ce7b1d98-c21f-4412-97ce-9dca59ce2649" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Nov 03 20:20:53.109108 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:53.109082    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3da0091e-b6g97"
Nov 03 20:20:53.223439 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:53.223406    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72de3f5b-77f3-4631-ad8d-6065ad327287-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72de3f5b-77f3-4631-ad8d-6065ad327287-maas-api-openshift-ca\") pod \"72de3f5b-77f3-4631-ad8d-6065ad327287\" (UID: \"72de3f5b-77f3-4631-ad8d-6065ad327287\") "
Nov 03 20:20:53.223629 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:53.223502    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72de3f5b-77f3-4631-ad8d-6065ad327287-kube-api-access-fwzlj\" (UniqueName: \"kubernetes.io/projected/72de3f5b-77f3-4631-ad8d-6065ad327287-kube-api-access-fwzlj\") pod \"72de3f5b-77f3-4631-ad8d-6065ad327287\" (UID: \"72de3f5b-77f3-4631-ad8d-6065ad327287\") "
Nov 03 20:20:53.223704 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:53.223690    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72de3f5b-77f3-4631-ad8d-6065ad327287-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72de3f5b-77f3-4631-ad8d-6065ad327287-maas-api-xks-ca\") pod \"72de3f5b-77f3-4631-ad8d-6065ad327287\" (UID: \"72de3f5b-77f3-4631-ad8d-6065ad327287\") "
Nov 03 20:20:53.223829 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:53.223800    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72de3f5b-77f3-4631-ad8d-6065ad327287-maas-api-openshift-ca" pod "72de3f5b-77f3-4631-ad8d-6065ad327287" (UID: "72de3f5b-77f3-4631-ad8d-6065ad327287"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:20:53.224215 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:53.224201    5570 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72de3f5b-77f3-4631-ad8d-6065ad327287-maas-api-openshift-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:20:53.226143 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:53.226113    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72de3f5b-77f3-4631-ad8d-6065ad327287-kube-api-access-fwzlj" pod "72de3f5b-77f3-4631-ad8d-6065ad327287" (UID: "72de3f5b-77f3-4631-ad8d-6065ad327287"). InnerVolumeSpecName "kube-api-access-fwzlj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:20:53.226260 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:53.226172    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72de3f5b-77f3-4631-ad8d-6065ad327287-maas-api-xks-ca" pod "72de3f5b-77f3-4631-ad8d-6065ad327287" (UID: "72de3f5b-77f3-4631-ad8d-6065ad327287"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:20:53.342004 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:53.341963    5570 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72de3f5b-77f3-4631-ad8d-6065ad327287-maas-api-xks-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:20:53.360879 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:53.360845    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwzlj\" (UniqueName: \"kubernetes.io/projected/72de3f5b-77f3-4631-ad8d-6065ad327287-kube-api-access-fwzlj\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:20:53.795555 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:53.795519    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2067241-ntph9"]
Nov 03 20:20:53.796289 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:53.796268    5570 cpu_manager.go:414] "containerMap: removing container" podUID="72de3f5b-77f3-4631-ad8d-6065ad327287" containerName="revoke-keys"
Nov 03 20:20:53.796289 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:53.796289    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72de3f5b-77f3-4631-ad8d-6065ad327287" containerName="revoke-keys"
Nov 03 20:20:53.796466 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:53.796417    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="72de3f5b-77f3-4631-ad8d-6065ad327287" containerName="revoke-keys"
Nov 03 20:20:53.800450 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:53.800423    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2067241-ntph9"
Nov 03 20:20:53.817352 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:53.817327    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2067241-ntph9"]
Nov 03 20:20:53.875366 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:53.875332    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrvr4\" (UniqueName: \"kubernetes.io/projected/aae4bf32-bc99-4177-aecb-4c7169a348c8-kube-api-access-lrvr4\") pod \"maas-api-revoke-sub-a2067241-ntph9\" (UID: \"aae4bf32-bc99-4177-aecb-4c7169a348c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2067241-ntph9"
Nov 03 20:20:53.875680 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:53.875651    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aae4bf32-bc99-4177-aecb-4c7169a348c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a2067241-ntph9\" (UID: \"aae4bf32-bc99-4177-aecb-4c7169a348c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2067241-ntph9"
Nov 03 20:20:53.875958 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:53.875934    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aae4bf32-bc99-4177-aecb-4c7169a348c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a2067241-ntph9\" (UID: \"aae4bf32-bc99-4177-aecb-4c7169a348c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2067241-ntph9"
Nov 03 20:20:53.900001 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:53.899974    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3da0091e-b6g97"
Nov 03 20:20:53.900165 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:53.900010    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3da0091e-b6g97" event={"ID":"72de3f5b-77f3-4631-ad8d-6065ad327287","Type":"ContainerDied","Data":"a827ebd19189a5bfd6278c2c4835e6cab992e53e0a3c813254fd5a8c368532d6"}
Nov 03 20:20:53.900165 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:53.900067    5570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a827ebd19189a5bfd6278c2c4835e6cab992e53e0a3c813254fd5a8c368532d6"
Nov 03 20:20:53.978104 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:53.978057    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aae4bf32-bc99-4177-aecb-4c7169a348c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a2067241-ntph9\" (UID: \"aae4bf32-bc99-4177-aecb-4c7169a348c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2067241-ntph9"
Nov 03 20:20:53.978317 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:53.978192    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrvr4\" (UniqueName: \"kubernetes.io/projected/aae4bf32-bc99-4177-aecb-4c7169a348c8-kube-api-access-lrvr4\") pod \"maas-api-revoke-sub-a2067241-ntph9\" (UID: \"aae4bf32-bc99-4177-aecb-4c7169a348c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2067241-ntph9"
Nov 03 20:20:53.978386 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:53.978366    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aae4bf32-bc99-4177-aecb-4c7169a348c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a2067241-ntph9\" (UID: \"aae4bf32-bc99-4177-aecb-4c7169a348c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2067241-ntph9"
Nov 03 20:20:53.978750 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:53.978720    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aae4bf32-bc99-4177-aecb-4c7169a348c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a2067241-ntph9\" (UID: \"aae4bf32-bc99-4177-aecb-4c7169a348c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2067241-ntph9"
Nov 03 20:20:53.980766 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:53.980741    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aae4bf32-bc99-4177-aecb-4c7169a348c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a2067241-ntph9\" (UID: \"aae4bf32-bc99-4177-aecb-4c7169a348c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2067241-ntph9"
Nov 03 20:20:53.990905 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:53.990877    5570 patch_prober.go:29] interesting pod/e2e-worker-w0-e40816-openshift-default-744c57967c-wj4gp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Nov 03 20:20:53.991321 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:53.990939    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-wj4gp" podUID="ce7b1d98-c21f-4412-97ce-9dca59ce2649" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Nov 03 20:20:53.999405 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:53.999377    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrvr4\" (UniqueName: \"kubernetes.io/projected/aae4bf32-bc99-4177-aecb-4c7169a348c8-kube-api-access-lrvr4\") pod \"maas-api-revoke-sub-a2067241-ntph9\" (UID: \"aae4bf32-bc99-4177-aecb-4c7169a348c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2067241-ntph9"
Nov 03 20:20:54.123292 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:54.123193    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2067241-ntph9"
Nov 03 20:20:54.467708 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:20:54.467671    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaae4bf32_bc99_4177_aecb_4c7169a348c8.slice/crio-e8a346fc574fc54792b24b126169bd0f739a0447719a91ad91f7a5193760447b WatchSource:0}: Error finding container e8a346fc574fc54792b24b126169bd0f739a0447719a91ad91f7a5193760447b: Status 404 returned error can't find the container with id e8a346fc574fc54792b24b126169bd0f739a0447719a91ad91f7a5193760447b
Nov 03 20:20:54.468817 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:54.468787    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2067241-ntph9"]
Nov 03 20:20:54.918290 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:54.918193    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2067241-ntph9" event={"ID":"aae4bf32-bc99-4177-aecb-4c7169a348c8","Type":"ContainerStarted","Data":"e8a346fc574fc54792b24b126169bd0f739a0447719a91ad91f7a5193760447b"}
Nov 03 20:20:54.990156 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:54.990116    5570 patch_prober.go:29] interesting pod/e2e-worker-w0-e40816-openshift-default-744c57967c-wj4gp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Nov 03 20:20:54.990361 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:54.990337    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-wj4gp" podUID="ce7b1d98-c21f-4412-97ce-9dca59ce2649" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Nov 03 20:20:55.931190 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:55.931087    5570 generic.go:356] "Generic (PLEG): container finished" podID="aae4bf32-bc99-4177-aecb-4c7169a348c8" containerID="79eba0ca017a8bc3978468618c78158e97c41c4ffed7d08e1a92e80fadf784ba" exitCode=0
Nov 03 20:20:55.931587 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:55.931169    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2067241-ntph9" event={"ID":"aae4bf32-bc99-4177-aecb-4c7169a348c8","Type":"ContainerDied","Data":"79eba0ca017a8bc3978468618c78158e97c41c4ffed7d08e1a92e80fadf784ba"}
Nov 03 20:20:55.991081 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:55.991043    5570 patch_prober.go:29] interesting pod/e2e-worker-w0-e40816-openshift-default-744c57967c-wj4gp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Nov 03 20:20:55.991268 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:55.991103    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-wj4gp" podUID="ce7b1d98-c21f-4412-97ce-9dca59ce2649" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Nov 03 20:20:57.191184 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:57.191157    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2067241-ntph9"
Nov 03 20:20:57.240630 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:20:57.240515    5570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8bwhz" containerName="extractor"
Nov 03 20:20:57.249661 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:57.248375    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-wj4gp"
Nov 03 20:20:57.249661 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:57.249430    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-wj4gp"
Nov 03 20:20:57.251385 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:57.251358    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-wj4gp"
Nov 03 20:20:57.353937 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:57.353902    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aae4bf32-bc99-4177-aecb-4c7169a348c8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aae4bf32-bc99-4177-aecb-4c7169a348c8-maas-api-openshift-ca\") pod \"aae4bf32-bc99-4177-aecb-4c7169a348c8\" (UID: \"aae4bf32-bc99-4177-aecb-4c7169a348c8\") "
Nov 03 20:20:57.354353 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:57.354176    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aae4bf32-bc99-4177-aecb-4c7169a348c8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aae4bf32-bc99-4177-aecb-4c7169a348c8-maas-api-xks-ca\") pod \"aae4bf32-bc99-4177-aecb-4c7169a348c8\" (UID: \"aae4bf32-bc99-4177-aecb-4c7169a348c8\") "
Nov 03 20:20:57.354353 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:57.354298    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aae4bf32-bc99-4177-aecb-4c7169a348c8-maas-api-openshift-ca" pod "aae4bf32-bc99-4177-aecb-4c7169a348c8" (UID: "aae4bf32-bc99-4177-aecb-4c7169a348c8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:20:57.354353 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:57.354311    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aae4bf32-bc99-4177-aecb-4c7169a348c8-kube-api-access-lrvr4\" (UniqueName: \"kubernetes.io/projected/aae4bf32-bc99-4177-aecb-4c7169a348c8-kube-api-access-lrvr4\") pod \"aae4bf32-bc99-4177-aecb-4c7169a348c8\" (UID: \"aae4bf32-bc99-4177-aecb-4c7169a348c8\") "
Nov 03 20:20:57.355411 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:57.355386    5570 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aae4bf32-bc99-4177-aecb-4c7169a348c8-maas-api-openshift-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:20:57.356679 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:57.356653    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aae4bf32-bc99-4177-aecb-4c7169a348c8-kube-api-access-lrvr4" pod "aae4bf32-bc99-4177-aecb-4c7169a348c8" (UID: "aae4bf32-bc99-4177-aecb-4c7169a348c8"). InnerVolumeSpecName "kube-api-access-lrvr4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:20:57.357197 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:57.357164    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aae4bf32-bc99-4177-aecb-4c7169a348c8-maas-api-xks-ca" pod "aae4bf32-bc99-4177-aecb-4c7169a348c8" (UID: "aae4bf32-bc99-4177-aecb-4c7169a348c8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:20:57.462818 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:57.460703    5570 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aae4bf32-bc99-4177-aecb-4c7169a348c8-maas-api-xks-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:20:57.462818 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:57.460747    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrvr4\" (UniqueName: \"kubernetes.io/projected/aae4bf32-bc99-4177-aecb-4c7169a348c8-kube-api-access-lrvr4\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:20:57.982006 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:57.981959    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2067241-ntph9" event={"ID":"aae4bf32-bc99-4177-aecb-4c7169a348c8","Type":"ContainerDied","Data":"e8a346fc574fc54792b24b126169bd0f739a0447719a91ad91f7a5193760447b"}
Nov 03 20:20:57.982006 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:57.981990    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2067241-ntph9"
Nov 03 20:20:57.982251 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:20:57.982035    5570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e8a346fc574fc54792b24b126169bd0f739a0447719a91ad91f7a5193760447b"
Nov 03 20:21:02.877605 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:02.877566    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9668dd6c-qhvqr"]
Nov 03 20:21:02.878327 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:02.878310    5570 cpu_manager.go:414] "containerMap: removing container" podUID="aae4bf32-bc99-4177-aecb-4c7169a348c8" containerName="revoke-keys"
Nov 03 20:21:02.878396 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:02.878328    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aae4bf32-bc99-4177-aecb-4c7169a348c8" containerName="revoke-keys"
Nov 03 20:21:02.878464 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:02.878454    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="aae4bf32-bc99-4177-aecb-4c7169a348c8" containerName="revoke-keys"
Nov 03 20:21:02.883550 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:02.883525    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9668dd6c-qhvqr"
Nov 03 20:21:02.899734 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:02.899706    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9668dd6c-qhvqr"]
Nov 03 20:21:02.906363 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:02.906332    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:21:02.906538 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:02.906514    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l7s5w\""
Nov 03 20:21:03.071462 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:03.071427    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdtfd\" (UniqueName: \"kubernetes.io/projected/c5d89a76-31b0-4631-8756-48d9f887ee21-kube-api-access-mdtfd\") pod \"maas-api-revoke-sub-9668dd6c-qhvqr\" (UID: \"c5d89a76-31b0-4631-8756-48d9f887ee21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9668dd6c-qhvqr"
Nov 03 20:21:03.071614 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:03.071551    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5d89a76-31b0-4631-8756-48d9f887ee21-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9668dd6c-qhvqr\" (UID: \"c5d89a76-31b0-4631-8756-48d9f887ee21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9668dd6c-qhvqr"
Nov 03 20:21:03.071651 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:03.071635    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5d89a76-31b0-4631-8756-48d9f887ee21-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9668dd6c-qhvqr\" (UID: \"c5d89a76-31b0-4631-8756-48d9f887ee21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9668dd6c-qhvqr"
Nov 03 20:21:03.173343 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:03.173251    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdtfd\" (UniqueName: \"kubernetes.io/projected/c5d89a76-31b0-4631-8756-48d9f887ee21-kube-api-access-mdtfd\") pod \"maas-api-revoke-sub-9668dd6c-qhvqr\" (UID: \"c5d89a76-31b0-4631-8756-48d9f887ee21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9668dd6c-qhvqr"
Nov 03 20:21:03.173496 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:03.173348    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5d89a76-31b0-4631-8756-48d9f887ee21-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9668dd6c-qhvqr\" (UID: \"c5d89a76-31b0-4631-8756-48d9f887ee21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9668dd6c-qhvqr"
Nov 03 20:21:03.173496 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:03.173454    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5d89a76-31b0-4631-8756-48d9f887ee21-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9668dd6c-qhvqr\" (UID: \"c5d89a76-31b0-4631-8756-48d9f887ee21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9668dd6c-qhvqr"
Nov 03 20:21:03.174299 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:03.174277    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5d89a76-31b0-4631-8756-48d9f887ee21-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9668dd6c-qhvqr\" (UID: \"c5d89a76-31b0-4631-8756-48d9f887ee21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9668dd6c-qhvqr"
Nov 03 20:21:03.176030 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:03.176001    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5d89a76-31b0-4631-8756-48d9f887ee21-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9668dd6c-qhvqr\" (UID: \"c5d89a76-31b0-4631-8756-48d9f887ee21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9668dd6c-qhvqr"
Nov 03 20:21:03.193314 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:03.193286    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdtfd\" (UniqueName: \"kubernetes.io/projected/c5d89a76-31b0-4631-8756-48d9f887ee21-kube-api-access-mdtfd\") pod \"maas-api-revoke-sub-9668dd6c-qhvqr\" (UID: \"c5d89a76-31b0-4631-8756-48d9f887ee21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9668dd6c-qhvqr"
Nov 03 20:21:03.202953 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:03.202919    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9668dd6c-qhvqr"
Nov 03 20:21:03.545476 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:03.545442    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9668dd6c-qhvqr"]
Nov 03 20:21:03.545764 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:21:03.545739    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5d89a76_31b0_4631_8756_48d9f887ee21.slice/crio-19b39c031eba2f4fc4c36f9dc7757921c787c91eb67e411cbed0b8c3b16ffa4b WatchSource:0}: Error finding container 19b39c031eba2f4fc4c36f9dc7757921c787c91eb67e411cbed0b8c3b16ffa4b: Status 404 returned error can't find the container with id 19b39c031eba2f4fc4c36f9dc7757921c787c91eb67e411cbed0b8c3b16ffa4b
Nov 03 20:21:04.097744 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:04.097635    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9668dd6c-qhvqr" event={"ID":"c5d89a76-31b0-4631-8756-48d9f887ee21","Type":"ContainerStarted","Data":"b1a97f151e6cfe91f285361e57a4d39552ecef5752a82463f243cd17408873bf"}
Nov 03 20:21:04.097744 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:04.097711    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9668dd6c-qhvqr" event={"ID":"c5d89a76-31b0-4631-8756-48d9f887ee21","Type":"ContainerStarted","Data":"19b39c031eba2f4fc4c36f9dc7757921c787c91eb67e411cbed0b8c3b16ffa4b"}
Nov 03 20:21:04.143317 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:04.143249    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9668dd6c-qhvqr" podStartSLOduration=2.143229326 podStartE2EDuration="2.143229326s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:21:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:21:04.13793245 +0000 UTC m=+998.290594338" watchObservedRunningTime="2026-11-03 20:21:04.143229326 +0000 UTC m=+998.295891223"
Nov 03 20:21:05.108039 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:05.107999    5570 generic.go:356] "Generic (PLEG): container finished" podID="c5d89a76-31b0-4631-8756-48d9f887ee21" containerID="b1a97f151e6cfe91f285361e57a4d39552ecef5752a82463f243cd17408873bf" exitCode=0
Nov 03 20:21:05.108576 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:05.108111    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9668dd6c-qhvqr" event={"ID":"c5d89a76-31b0-4631-8756-48d9f887ee21","Type":"ContainerDied","Data":"b1a97f151e6cfe91f285361e57a4d39552ecef5752a82463f243cd17408873bf"}
Nov 03 20:21:06.308331 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:06.308300    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9668dd6c-qhvqr"
Nov 03 20:21:06.355155 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:06.355121    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c5d89a76-31b0-4631-8756-48d9f887ee21-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5d89a76-31b0-4631-8756-48d9f887ee21-maas-api-xks-ca\") pod \"c5d89a76-31b0-4631-8756-48d9f887ee21\" (UID: \"c5d89a76-31b0-4631-8756-48d9f887ee21\") "
Nov 03 20:21:06.355333 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:06.355261    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5d89a76-31b0-4631-8756-48d9f887ee21-kube-api-access-mdtfd\" (UniqueName: \"kubernetes.io/projected/c5d89a76-31b0-4631-8756-48d9f887ee21-kube-api-access-mdtfd\") pod \"c5d89a76-31b0-4631-8756-48d9f887ee21\" (UID: \"c5d89a76-31b0-4631-8756-48d9f887ee21\") "
Nov 03 20:21:06.355467 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:06.355439    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5d89a76-31b0-4631-8756-48d9f887ee21-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5d89a76-31b0-4631-8756-48d9f887ee21-maas-api-openshift-ca\") pod \"c5d89a76-31b0-4631-8756-48d9f887ee21\" (UID: \"c5d89a76-31b0-4631-8756-48d9f887ee21\") "
Nov 03 20:21:06.355879 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:06.355845    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5d89a76-31b0-4631-8756-48d9f887ee21-maas-api-openshift-ca" pod "c5d89a76-31b0-4631-8756-48d9f887ee21" (UID: "c5d89a76-31b0-4631-8756-48d9f887ee21"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:21:06.356243 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:06.356217    5570 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5d89a76-31b0-4631-8756-48d9f887ee21-maas-api-openshift-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:21:06.357617 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:06.357592    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5d89a76-31b0-4631-8756-48d9f887ee21-maas-api-xks-ca" pod "c5d89a76-31b0-4631-8756-48d9f887ee21" (UID: "c5d89a76-31b0-4631-8756-48d9f887ee21"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:21:06.357914 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:06.357892    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5d89a76-31b0-4631-8756-48d9f887ee21-kube-api-access-mdtfd" pod "c5d89a76-31b0-4631-8756-48d9f887ee21" (UID: "c5d89a76-31b0-4631-8756-48d9f887ee21"). InnerVolumeSpecName "kube-api-access-mdtfd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:21:06.418037 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:06.418003    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-126d4c8e-f5fm4"]
Nov 03 20:21:06.419131 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:06.419107    5570 cpu_manager.go:414] "containerMap: removing container" podUID="c5d89a76-31b0-4631-8756-48d9f887ee21" containerName="revoke-keys"
Nov 03 20:21:06.419218 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:06.419134    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5d89a76-31b0-4631-8756-48d9f887ee21" containerName="revoke-keys"
Nov 03 20:21:06.419372 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:06.419356    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5d89a76-31b0-4631-8756-48d9f887ee21" containerName="revoke-keys"
Nov 03 20:21:06.423809 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:06.423786    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-126d4c8e-f5fm4"
Nov 03 20:21:06.436156 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:06.436128    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-126d4c8e-f5fm4"]
Nov 03 20:21:06.458540 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:06.458505    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mc62\" (UniqueName: \"kubernetes.io/projected/56932084-ddac-401e-87d3-8c75064e4498-kube-api-access-6mc62\") pod \"maas-api-revoke-sub-126d4c8e-f5fm4\" (UID: \"56932084-ddac-401e-87d3-8c75064e4498\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-126d4c8e-f5fm4"
Nov 03 20:21:06.458764 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:06.458738    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56932084-ddac-401e-87d3-8c75064e4498-maas-api-xks-ca\") pod \"maas-api-revoke-sub-126d4c8e-f5fm4\" (UID: \"56932084-ddac-401e-87d3-8c75064e4498\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-126d4c8e-f5fm4"
Nov 03 20:21:06.458919 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:06.458896    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56932084-ddac-401e-87d3-8c75064e4498-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-126d4c8e-f5fm4\" (UID: \"56932084-ddac-401e-87d3-8c75064e4498\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-126d4c8e-f5fm4"
Nov 03 20:21:06.459078 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:06.459063    5570 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5d89a76-31b0-4631-8756-48d9f887ee21-maas-api-xks-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:21:06.459126 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:06.459083    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdtfd\" (UniqueName: \"kubernetes.io/projected/c5d89a76-31b0-4631-8756-48d9f887ee21-kube-api-access-mdtfd\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:21:06.627848 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:06.626123    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56932084-ddac-401e-87d3-8c75064e4498-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-126d4c8e-f5fm4\" (UID: \"56932084-ddac-401e-87d3-8c75064e4498\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-126d4c8e-f5fm4"
Nov 03 20:21:06.627848 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:06.626460    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mc62\" (UniqueName: \"kubernetes.io/projected/56932084-ddac-401e-87d3-8c75064e4498-kube-api-access-6mc62\") pod \"maas-api-revoke-sub-126d4c8e-f5fm4\" (UID: \"56932084-ddac-401e-87d3-8c75064e4498\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-126d4c8e-f5fm4"
Nov 03 20:21:06.627848 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:06.626652    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56932084-ddac-401e-87d3-8c75064e4498-maas-api-xks-ca\") pod \"maas-api-revoke-sub-126d4c8e-f5fm4\" (UID: \"56932084-ddac-401e-87d3-8c75064e4498\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-126d4c8e-f5fm4"
Nov 03 20:21:06.631128 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:06.629593    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56932084-ddac-401e-87d3-8c75064e4498-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-126d4c8e-f5fm4\" (UID: \"56932084-ddac-401e-87d3-8c75064e4498\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-126d4c8e-f5fm4"
Nov 03 20:21:06.633312 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:06.633280    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56932084-ddac-401e-87d3-8c75064e4498-maas-api-xks-ca\") pod \"maas-api-revoke-sub-126d4c8e-f5fm4\" (UID: \"56932084-ddac-401e-87d3-8c75064e4498\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-126d4c8e-f5fm4"
Nov 03 20:21:06.653237 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:06.653210    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mc62\" (UniqueName: \"kubernetes.io/projected/56932084-ddac-401e-87d3-8c75064e4498-kube-api-access-6mc62\") pod \"maas-api-revoke-sub-126d4c8e-f5fm4\" (UID: \"56932084-ddac-401e-87d3-8c75064e4498\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-126d4c8e-f5fm4"
Nov 03 20:21:06.743470 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:06.743424    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-126d4c8e-f5fm4"
Nov 03 20:21:07.052898 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:07.051679    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-126d4c8e-f5fm4"]
Nov 03 20:21:07.055973 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:21:07.055945    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56932084_ddac_401e_87d3_8c75064e4498.slice/crio-99feb6d13e7c523f182e63adedc296f4ae8647f0e2e11f2afeba5aaac6b95505 WatchSource:0}: Error finding container 99feb6d13e7c523f182e63adedc296f4ae8647f0e2e11f2afeba5aaac6b95505: Status 404 returned error can't find the container with id 99feb6d13e7c523f182e63adedc296f4ae8647f0e2e11f2afeba5aaac6b95505
Nov 03 20:21:07.132466 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:07.132435    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9668dd6c-qhvqr"
Nov 03 20:21:07.132662 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:07.132510    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9668dd6c-qhvqr" event={"ID":"c5d89a76-31b0-4631-8756-48d9f887ee21","Type":"ContainerDied","Data":"19b39c031eba2f4fc4c36f9dc7757921c787c91eb67e411cbed0b8c3b16ffa4b"}
Nov 03 20:21:07.132662 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:07.132558    5570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="19b39c031eba2f4fc4c36f9dc7757921c787c91eb67e411cbed0b8c3b16ffa4b"
Nov 03 20:21:07.135246 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:07.135213    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-126d4c8e-f5fm4" event={"ID":"56932084-ddac-401e-87d3-8c75064e4498","Type":"ContainerStarted","Data":"99feb6d13e7c523f182e63adedc296f4ae8647f0e2e11f2afeba5aaac6b95505"}
Nov 03 20:21:08.290496 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:08.290400    5570 generic.go:356] "Generic (PLEG): container finished" podID="56932084-ddac-401e-87d3-8c75064e4498" containerID="9aefaf40501ee5dc0f0ba88ef0e2a1c17232ed3858d61c58eca8b6d0b6f39b4b" exitCode=0
Nov 03 20:21:08.290930 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:08.290599    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-126d4c8e-f5fm4" event={"ID":"56932084-ddac-401e-87d3-8c75064e4498","Type":"ContainerDied","Data":"9aefaf40501ee5dc0f0ba88ef0e2a1c17232ed3858d61c58eca8b6d0b6f39b4b"}
Nov 03 20:21:09.476021 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:09.475984    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-126d4c8e-f5fm4"
Nov 03 20:21:09.519607 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:09.519567    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56932084-ddac-401e-87d3-8c75064e4498-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56932084-ddac-401e-87d3-8c75064e4498-maas-api-openshift-ca\") pod \"56932084-ddac-401e-87d3-8c75064e4498\" (UID: \"56932084-ddac-401e-87d3-8c75064e4498\") "
Nov 03 20:21:09.520148 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:09.520017    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56932084-ddac-401e-87d3-8c75064e4498-maas-api-openshift-ca" pod "56932084-ddac-401e-87d3-8c75064e4498" (UID: "56932084-ddac-401e-87d3-8c75064e4498"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:21:09.520272 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:09.520148    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56932084-ddac-401e-87d3-8c75064e4498-kube-api-access-6mc62\" (UniqueName: \"kubernetes.io/projected/56932084-ddac-401e-87d3-8c75064e4498-kube-api-access-6mc62\") pod \"56932084-ddac-401e-87d3-8c75064e4498\" (UID: \"56932084-ddac-401e-87d3-8c75064e4498\") "
Nov 03 20:21:09.523789 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:09.520342    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/56932084-ddac-401e-87d3-8c75064e4498-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56932084-ddac-401e-87d3-8c75064e4498-maas-api-xks-ca\") pod \"56932084-ddac-401e-87d3-8c75064e4498\" (UID: \"56932084-ddac-401e-87d3-8c75064e4498\") "
Nov 03 20:21:09.527401 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:09.527370    5570 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56932084-ddac-401e-87d3-8c75064e4498-maas-api-openshift-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:21:09.527961 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:09.527927    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56932084-ddac-401e-87d3-8c75064e4498-maas-api-xks-ca" pod "56932084-ddac-401e-87d3-8c75064e4498" (UID: "56932084-ddac-401e-87d3-8c75064e4498"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:21:09.528614 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:09.528585    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56932084-ddac-401e-87d3-8c75064e4498-kube-api-access-6mc62" pod "56932084-ddac-401e-87d3-8c75064e4498" (UID: "56932084-ddac-401e-87d3-8c75064e4498"). InnerVolumeSpecName "kube-api-access-6mc62". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:21:09.629376 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:09.629282    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6mc62\" (UniqueName: \"kubernetes.io/projected/56932084-ddac-401e-87d3-8c75064e4498-kube-api-access-6mc62\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:21:09.629376 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:09.629313    5570 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56932084-ddac-401e-87d3-8c75064e4498-maas-api-xks-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:21:10.229061 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:10.228970    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fca5b006-j488g"]
Nov 03 20:21:10.229692 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:10.229676    5570 cpu_manager.go:414] "containerMap: removing container" podUID="56932084-ddac-401e-87d3-8c75064e4498" containerName="revoke-keys"
Nov 03 20:21:10.229732 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:10.229693    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56932084-ddac-401e-87d3-8c75064e4498" containerName="revoke-keys"
Nov 03 20:21:10.229857 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:10.229847    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="56932084-ddac-401e-87d3-8c75064e4498" containerName="revoke-keys"
Nov 03 20:21:10.233310 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:10.233293    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fca5b006-j488g"
Nov 03 20:21:10.240575 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:10.240543    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc403049-5b3e-45fa-80b8-6d1f53ab171c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fca5b006-j488g\" (UID: \"fc403049-5b3e-45fa-80b8-6d1f53ab171c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fca5b006-j488g"
Nov 03 20:21:10.240853 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:10.240700    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc403049-5b3e-45fa-80b8-6d1f53ab171c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fca5b006-j488g\" (UID: \"fc403049-5b3e-45fa-80b8-6d1f53ab171c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fca5b006-j488g"
Nov 03 20:21:10.240920 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:10.240888    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7h8cg\" (UniqueName: \"kubernetes.io/projected/fc403049-5b3e-45fa-80b8-6d1f53ab171c-kube-api-access-7h8cg\") pod \"maas-api-revoke-sub-fca5b006-j488g\" (UID: \"fc403049-5b3e-45fa-80b8-6d1f53ab171c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fca5b006-j488g"
Nov 03 20:21:10.250850 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:10.250817    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fca5b006-j488g"]
Nov 03 20:21:10.310282 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:10.310252    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-126d4c8e-f5fm4"
Nov 03 20:21:10.310694 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:10.310298    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-126d4c8e-f5fm4" event={"ID":"56932084-ddac-401e-87d3-8c75064e4498","Type":"ContainerDied","Data":"99feb6d13e7c523f182e63adedc296f4ae8647f0e2e11f2afeba5aaac6b95505"}
Nov 03 20:21:10.310694 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:10.310359    5570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="99feb6d13e7c523f182e63adedc296f4ae8647f0e2e11f2afeba5aaac6b95505"
Nov 03 20:21:10.343124 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:10.343091    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc403049-5b3e-45fa-80b8-6d1f53ab171c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fca5b006-j488g\" (UID: \"fc403049-5b3e-45fa-80b8-6d1f53ab171c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fca5b006-j488g"
Nov 03 20:21:10.343292 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:10.343272    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc403049-5b3e-45fa-80b8-6d1f53ab171c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fca5b006-j488g\" (UID: \"fc403049-5b3e-45fa-80b8-6d1f53ab171c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fca5b006-j488g"
Nov 03 20:21:10.343428 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:10.343410    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7h8cg\" (UniqueName: \"kubernetes.io/projected/fc403049-5b3e-45fa-80b8-6d1f53ab171c-kube-api-access-7h8cg\") pod \"maas-api-revoke-sub-fca5b006-j488g\" (UID: \"fc403049-5b3e-45fa-80b8-6d1f53ab171c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fca5b006-j488g"
Nov 03 20:21:10.343914 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:10.343890    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc403049-5b3e-45fa-80b8-6d1f53ab171c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fca5b006-j488g\" (UID: \"fc403049-5b3e-45fa-80b8-6d1f53ab171c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fca5b006-j488g"
Nov 03 20:21:10.345950 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:10.345926    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc403049-5b3e-45fa-80b8-6d1f53ab171c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fca5b006-j488g\" (UID: \"fc403049-5b3e-45fa-80b8-6d1f53ab171c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fca5b006-j488g"
Nov 03 20:21:10.364691 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:10.364664    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7h8cg\" (UniqueName: \"kubernetes.io/projected/fc403049-5b3e-45fa-80b8-6d1f53ab171c-kube-api-access-7h8cg\") pod \"maas-api-revoke-sub-fca5b006-j488g\" (UID: \"fc403049-5b3e-45fa-80b8-6d1f53ab171c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fca5b006-j488g"
Nov 03 20:21:10.549875 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:10.549764    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fca5b006-j488g"
Nov 03 20:21:10.855928 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:10.855890    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fca5b006-j488g"]
Nov 03 20:21:10.856533 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:21:10.856499    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc403049_5b3e_45fa_80b8_6d1f53ab171c.slice/crio-d6345650a032b7a6c800efb9efcbdbbd321606c2ceb562657aee5b0ab788161c WatchSource:0}: Error finding container d6345650a032b7a6c800efb9efcbdbbd321606c2ceb562657aee5b0ab788161c: Status 404 returned error can't find the container with id d6345650a032b7a6c800efb9efcbdbbd321606c2ceb562657aee5b0ab788161c
Nov 03 20:21:11.334525 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:11.334479    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fca5b006-j488g" event={"ID":"fc403049-5b3e-45fa-80b8-6d1f53ab171c","Type":"ContainerStarted","Data":"3fb0b1c4fb132283f1c575ea853e4c3e51f0937f24de0d2d5bb4eee799832e71"}
Nov 03 20:21:11.334827 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:11.334797    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fca5b006-j488g" event={"ID":"fc403049-5b3e-45fa-80b8-6d1f53ab171c","Type":"ContainerStarted","Data":"d6345650a032b7a6c800efb9efcbdbbd321606c2ceb562657aee5b0ab788161c"}
Nov 03 20:21:11.388258 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:11.388180    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fca5b006-j488g" podStartSLOduration=1.3881565120000001 podStartE2EDuration="1.388156512s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:21:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:21:11.382226561 +0000 UTC m=+1005.534888448" watchObservedRunningTime="2026-11-03 20:21:11.388156512 +0000 UTC m=+1005.540818403"
Nov 03 20:21:12.518644 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:12.512065    5570 generic.go:356] "Generic (PLEG): container finished" podID="fc403049-5b3e-45fa-80b8-6d1f53ab171c" containerID="3fb0b1c4fb132283f1c575ea853e4c3e51f0937f24de0d2d5bb4eee799832e71" exitCode=0
Nov 03 20:21:12.518644 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:12.512325    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fca5b006-j488g" event={"ID":"fc403049-5b3e-45fa-80b8-6d1f53ab171c","Type":"ContainerDied","Data":"3fb0b1c4fb132283f1c575ea853e4c3e51f0937f24de0d2d5bb4eee799832e71"}
Nov 03 20:21:13.517797 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:13.514889    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/distinct-w3-482722-kserve-d876967c-kd488" podUID="9a97f9cc-2261-471d-9053-51ee965850bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded"
Nov 03 20:21:13.517797 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:13.515978    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-2-w6-9e1bcb-kserve-7d96d9f6f6-4f7fl" podUID="56353b0f-08ee-4ada-9dc0-ac5afd5a59c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:13.522800 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:13.520921    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/premium-w1-6fc748-kserve-7474db9cd7-nf2xk" podUID="830e062e-83fc-4de7-91a7-5534dac1d46b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Nov 03 20:21:13.531799 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:13.529877    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz" podUID="3a715ef1-38e9-4386-9eec-e10e8df11a10" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:13.531799 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:13.530283    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/simulator-w0-e40816-kserve-688f79796f-2tvw5" podUID="c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Nov 03 20:21:13.634943 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:13.609884    5570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-e80203/premium-w4-e80203-kserve-5f9f445456-bms9d" podUID="6ad76efc-20c2-4951-b22a-c5a39cd39ed4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:13.634943 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:13.610157    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/premium-w2-9973d9-kserve-5c6dd4669c-5gv8x" podUID="cefc89ad-7ae1-433b-979d-6601678a6d98" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Nov 03 20:21:13.724799 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:13.723921    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/distinct-2-w0-e40816-kserve-79fb46654d-s242x" podUID="4ad9b048-925c-4d01-a839-a5f25d4dd02d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:13.750801 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:13.750203    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/premium-w4-e80203-kserve-5f9f445456-bms9d" podUID="6ad76efc-20c2-4951-b22a-c5a39cd39ed4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:13.968800 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:13.967896    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/distinct-2-w3-482722-kserve-74bf58cf7d-v9cjw" podUID="9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:14.135801 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:14.131947    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/premium-w3-482722-kserve-689dc7bd5d-x8rjp" podUID="369045c3-4d6a-49a7-9bc6-f2939883bee8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:14.135801 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:14.132388    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/distinct-2-w4-e80203-kserve-756f9b9bc6-69gxj" podUID="ccf5dc2f-f116-4917-bb1a-b136d8ed17f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded"
Nov 03 20:21:14.383999 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:14.383887    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh" podUID="2d3647f2-89e9-4908-a8ee-d84e527abeb0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:14.515521 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:14.515472    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/distinct-w3-482722-kserve-d876967c-kd488" podUID="9a97f9cc-2261-471d-9053-51ee965850bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded"
Nov 03 20:21:14.516432 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:14.516406    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-482722/distinct-w3-482722-kserve-d876967c-kd488"
Nov 03 20:21:14.517815 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:14.517126    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-2-w6-9e1bcb-kserve-7d96d9f6f6-4f7fl" podUID="56353b0f-08ee-4ada-9dc0-ac5afd5a59c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:14.518604 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:14.518584    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-2-w6-9e1bcb-kserve-7d96d9f6f6-4f7fl"
Nov 03 20:21:14.529934 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:14.529880    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/premium-w1-6fc748-kserve-7474db9cd7-nf2xk" podUID="830e062e-83fc-4de7-91a7-5534dac1d46b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:14.530932 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:14.530907    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6fc748/premium-w1-6fc748-kserve-7474db9cd7-nf2xk"
Nov 03 20:21:14.534306 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:14.534277    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/simulator-w0-e40816-kserve-688f79796f-2tvw5" podUID="c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:14.534585 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:14.534468    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz" podUID="3a715ef1-38e9-4386-9eec-e10e8df11a10" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": context deadline exceeded"
Nov 03 20:21:14.563894 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:14.549489    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e40816/simulator-w0-e40816-kserve-688f79796f-2tvw5"
Nov 03 20:21:14.563894 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:14.549562    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz"
Nov 03 20:21:14.621959 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:14.619858    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/premium-w2-9973d9-kserve-5c6dd4669c-5gv8x" podUID="cefc89ad-7ae1-433b-979d-6601678a6d98" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:14.621959 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:14.620451    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9973d9/premium-w2-9973d9-kserve-5c6dd4669c-5gv8x"
Nov 03 20:21:14.799797 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:14.797053    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/premium-w4-e80203-kserve-5f9f445456-bms9d" podUID="6ad76efc-20c2-4951-b22a-c5a39cd39ed4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:14.821865 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:14.819950    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/distinct-2-w0-e40816-kserve-79fb46654d-s242x" podUID="4ad9b048-925c-4d01-a839-a5f25d4dd02d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded"
Nov 03 20:21:14.821865 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:14.820815    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-e80203/premium-w4-e80203-kserve-5f9f445456-bms9d"
Nov 03 20:21:14.821865 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:14.820879    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e40816/distinct-2-w0-e40816-kserve-79fb46654d-s242x"
Nov 03 20:21:14.991815 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:14.990913    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/distinct-2-w3-482722-kserve-74bf58cf7d-v9cjw" podUID="9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:14.991815 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:14.991445    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-482722/distinct-2-w3-482722-kserve-74bf58cf7d-v9cjw"
Nov 03 20:21:15.035513 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:15.035470    5570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-9973d9/premium-w2-9973d9-kserve-5c6dd4669c-5gv8x" podUID="cefc89ad-7ae1-433b-979d-6601678a6d98" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:15.199931 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:15.199887    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/distinct-2-w4-e80203-kserve-756f9b9bc6-69gxj" podUID="ccf5dc2f-f116-4917-bb1a-b136d8ed17f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:15.225194 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:15.200602    5570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-482722/distinct-w3-482722-kserve-d876967c-kd488" podUID="9a97f9cc-2261-471d-9053-51ee965850bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/health\": context deadline exceeded"
Nov 03 20:21:15.225513 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:15.205316    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/premium-w3-482722-kserve-689dc7bd5d-x8rjp" podUID="369045c3-4d6a-49a7-9bc6-f2939883bee8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Nov 03 20:21:15.517571 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:15.479959    5570 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz" podUID="3a715ef1-38e9-4386-9eec-e10e8df11a10" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:15.517571 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:15.482373    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-e80203/distinct-2-w4-e80203-kserve-756f9b9bc6-69gxj"
Nov 03 20:21:15.517571 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:15.482551    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-482722/premium-w3-482722-kserve-689dc7bd5d-x8rjp"
Nov 03 20:21:15.517571 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:15.506171    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh" podUID="2d3647f2-89e9-4908-a8ee-d84e527abeb0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:15.550800 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:15.540100    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-2-w6-9e1bcb-kserve-7d96d9f6f6-4f7fl" podUID="56353b0f-08ee-4ada-9dc0-ac5afd5a59c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded"
Nov 03 20:21:15.579689 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:15.575081    5570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-482722/premium-w3-482722-kserve-689dc7bd5d-x8rjp" podUID="369045c3-4d6a-49a7-9bc6-f2939883bee8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:15.579689 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:15.577441    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/distinct-w3-482722-kserve-d876967c-kd488" podUID="9a97f9cc-2261-471d-9053-51ee965850bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:15.600800 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:15.589963    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz" podUID="3a715ef1-38e9-4386-9eec-e10e8df11a10" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": context deadline exceeded"
Nov 03 20:21:15.634797 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:15.620701    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/premium-w2-9973d9-kserve-5c6dd4669c-5gv8x" podUID="cefc89ad-7ae1-433b-979d-6601678a6d98" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:15.658159 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:15.658105    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/simulator-w0-e40816-kserve-688f79796f-2tvw5" podUID="c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:16.355799 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:16.355366    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/distinct-2-w3-482722-kserve-74bf58cf7d-v9cjw" podUID="9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded"
Nov 03 20:21:16.480903 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:16.453077    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/premium-w4-e80203-kserve-5f9f445456-bms9d" podUID="6ad76efc-20c2-4951-b22a-c5a39cd39ed4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Nov 03 20:21:16.492294 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:16.492241    5570 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87s5mf" podUID="b7306cb9-eeae-447b-b834-870c8d49a5aa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:16.562190 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:16.562086    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-2-w6-9e1bcb-kserve-7d96d9f6f6-4f7fl" podUID="56353b0f-08ee-4ada-9dc0-ac5afd5a59c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:16.670801 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:16.657580    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/distinct-2-w4-e80203-kserve-756f9b9bc6-69gxj" podUID="ccf5dc2f-f116-4917-bb1a-b136d8ed17f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:16.678861 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:16.492181    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/premium-w1-6fc748-kserve-7474db9cd7-nf2xk" podUID="830e062e-83fc-4de7-91a7-5534dac1d46b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Nov 03 20:21:16.706896 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:16.492010    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/premium-w3-482722-kserve-689dc7bd5d-x8rjp" podUID="369045c3-4d6a-49a7-9bc6-f2939883bee8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Nov 03 20:21:16.717900 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:16.717843    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/distinct-2-w0-e40816-kserve-79fb46654d-s242x" podUID="4ad9b048-925c-4d01-a839-a5f25d4dd02d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded"
Nov 03 20:21:16.753111 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:16.752970    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/simulator-w0-e40816-kserve-688f79796f-2tvw5" podUID="c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:16.779148 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:16.779091    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-cpfzl" podUID="30b21f09-fd9e-4836-9fc3-c19f818c9022" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:16.817915 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:16.577922    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/distinct-w3-482722-kserve-d876967c-kd488" podUID="9a97f9cc-2261-471d-9053-51ee965850bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:16.817915 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:16.801505    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-9e1bcb/premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh"
Nov 03 20:21:16.922812 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:16.887876    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/premium-w2-9973d9-kserve-5c6dd4669c-5gv8x" podUID="cefc89ad-7ae1-433b-979d-6601678a6d98" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:16.922812 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:16.608936    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz" podUID="3a715ef1-38e9-4386-9eec-e10e8df11a10" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:17.139388 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:17.139279    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh" podUID="2d3647f2-89e9-4908-a8ee-d84e527abeb0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:17.182964 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:17.182918    5570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-2-w6-9e1bcb-kserve-7d96d9f6f6-4f7fl" podUID="56353b0f-08ee-4ada-9dc0-ac5afd5a59c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:17.407859 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:17.382542    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fca5b006-j488g"
Nov 03 20:21:17.790037 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:17.747099    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc403049-5b3e-45fa-80b8-6d1f53ab171c-kube-api-access-7h8cg\" (UniqueName: \"kubernetes.io/projected/fc403049-5b3e-45fa-80b8-6d1f53ab171c-kube-api-access-7h8cg\") pod \"fc403049-5b3e-45fa-80b8-6d1f53ab171c\" (UID: \"fc403049-5b3e-45fa-80b8-6d1f53ab171c\") "
Nov 03 20:21:17.790037 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:17.747387    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc403049-5b3e-45fa-80b8-6d1f53ab171c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc403049-5b3e-45fa-80b8-6d1f53ab171c-maas-api-xks-ca\") pod \"fc403049-5b3e-45fa-80b8-6d1f53ab171c\" (UID: \"fc403049-5b3e-45fa-80b8-6d1f53ab171c\") "
Nov 03 20:21:17.790037 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:17.747638    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc403049-5b3e-45fa-80b8-6d1f53ab171c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc403049-5b3e-45fa-80b8-6d1f53ab171c-maas-api-openshift-ca\") pod \"fc403049-5b3e-45fa-80b8-6d1f53ab171c\" (UID: \"fc403049-5b3e-45fa-80b8-6d1f53ab171c\") "
Nov 03 20:21:17.807390 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:17.807346    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc403049-5b3e-45fa-80b8-6d1f53ab171c-maas-api-openshift-ca" pod "fc403049-5b3e-45fa-80b8-6d1f53ab171c" (UID: "fc403049-5b3e-45fa-80b8-6d1f53ab171c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:21:17.884233 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:17.845403    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/distinct-2-w3-482722-kserve-74bf58cf7d-v9cjw" podUID="9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:17.884233 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:17.846393    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/simulator-w0-e40816-kserve-688f79796f-2tvw5" podUID="c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:17.884233 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:17.846679    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/premium-w4-e80203-kserve-5f9f445456-bms9d" podUID="6ad76efc-20c2-4951-b22a-c5a39cd39ed4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Nov 03 20:21:17.884233 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:17.846971    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/distinct-2-w4-e80203-kserve-756f9b9bc6-69gxj" podUID="ccf5dc2f-f116-4917-bb1a-b136d8ed17f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded"
Nov 03 20:21:17.884233 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:17.847211    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/premium-w1-6fc748-kserve-7474db9cd7-nf2xk" podUID="830e062e-83fc-4de7-91a7-5534dac1d46b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Nov 03 20:21:17.884233 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:17.847438    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/premium-w3-482722-kserve-689dc7bd5d-x8rjp" podUID="369045c3-4d6a-49a7-9bc6-f2939883bee8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Nov 03 20:21:17.884233 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:17.847668    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/distinct-2-w0-e40816-kserve-79fb46654d-s242x" podUID="4ad9b048-925c-4d01-a839-a5f25d4dd02d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded"
Nov 03 20:21:17.884233 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:17.851754    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/distinct-w3-482722-kserve-d876967c-kd488" podUID="9a97f9cc-2261-471d-9053-51ee965850bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded"
Nov 03 20:21:17.884233 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:17.854602    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-2-w6-9e1bcb-kserve-7d96d9f6f6-4f7fl" podUID="56353b0f-08ee-4ada-9dc0-ac5afd5a59c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded"
Nov 03 20:21:17.884233 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:17.855831    5570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-9e1bcb/premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh" podUID="2d3647f2-89e9-4908-a8ee-d84e527abeb0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:17.884233 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:21:17.856637    5570 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.055s"
Nov 03 20:21:17.884233 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:17.856680    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fca5b006-j488g" event={"ID":"fc403049-5b3e-45fa-80b8-6d1f53ab171c","Type":"ContainerDied","Data":"d6345650a032b7a6c800efb9efcbdbbd321606c2ceb562657aee5b0ab788161c"}
Nov 03 20:21:17.884233 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:17.856727    5570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d6345650a032b7a6c800efb9efcbdbbd321606c2ceb562657aee5b0ab788161c"
Nov 03 20:21:17.884233 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:17.856926    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e036ad7-5g2sj"]
Nov 03 20:21:17.884233 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:17.857996    5570 cpu_manager.go:414] "containerMap: removing container" podUID="fc403049-5b3e-45fa-80b8-6d1f53ab171c" containerName="revoke-keys"
Nov 03 20:21:17.884233 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:17.858017    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc403049-5b3e-45fa-80b8-6d1f53ab171c" containerName="revoke-keys"
Nov 03 20:21:17.884233 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:17.858174    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc403049-5b3e-45fa-80b8-6d1f53ab171c" containerName="revoke-keys"
Nov 03 20:21:17.884233 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:17.880927    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc403049-5b3e-45fa-80b8-6d1f53ab171c-maas-api-xks-ca" pod "fc403049-5b3e-45fa-80b8-6d1f53ab171c" (UID: "fc403049-5b3e-45fa-80b8-6d1f53ab171c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:21:17.917799 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:17.891095    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/premium-w2-9973d9-kserve-5c6dd4669c-5gv8x" podUID="cefc89ad-7ae1-433b-979d-6601678a6d98" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Nov 03 20:21:17.917799 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:17.916331    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz" podUID="3a715ef1-38e9-4386-9eec-e10e8df11a10" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": context deadline exceeded"
Nov 03 20:21:17.919241 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:17.919206    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc403049-5b3e-45fa-80b8-6d1f53ab171c-kube-api-access-7h8cg" pod "fc403049-5b3e-45fa-80b8-6d1f53ab171c" (UID: "fc403049-5b3e-45fa-80b8-6d1f53ab171c"). InnerVolumeSpecName "kube-api-access-7h8cg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:21:18.001813 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:17.990077    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc403049-5b3e-45fa-80b8-6d1f53ab171c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc403049-5b3e-45fa-80b8-6d1f53ab171c-maas-api-xks-ca\") pod \"fc403049-5b3e-45fa-80b8-6d1f53ab171c\" (UID: \"fc403049-5b3e-45fa-80b8-6d1f53ab171c\") "
Nov 03 20:21:18.001813 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:17.990470    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc403049-5b3e-45fa-80b8-6d1f53ab171c-kube-api-access-7h8cg\" (UniqueName: \"kubernetes.io/projected/fc403049-5b3e-45fa-80b8-6d1f53ab171c-kube-api-access-7h8cg\") pod \"fc403049-5b3e-45fa-80b8-6d1f53ab171c\" (UID: \"fc403049-5b3e-45fa-80b8-6d1f53ab171c\") "
Nov 03 20:21:18.001813 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:17.991075    5570 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc403049-5b3e-45fa-80b8-6d1f53ab171c-maas-api-openshift-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:21:18.001813 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:21:17.991167    5570 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/fc403049-5b3e-45fa-80b8-6d1f53ab171c/volumes/kubernetes.io~projected/kube-api-access-7h8cg
Nov 03 20:21:18.001813 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:17.991224    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc403049-5b3e-45fa-80b8-6d1f53ab171c-kube-api-access-7h8cg" pod "fc403049-5b3e-45fa-80b8-6d1f53ab171c" (UID: "fc403049-5b3e-45fa-80b8-6d1f53ab171c"). InnerVolumeSpecName "kube-api-access-7h8cg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:21:18.001813 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:21:17.999139    5570 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/fc403049-5b3e-45fa-80b8-6d1f53ab171c/volumes/kubernetes.io~secret/maas-api-xks-ca
Nov 03 20:21:18.001813 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:17.999163    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc403049-5b3e-45fa-80b8-6d1f53ab171c-maas-api-xks-ca" pod "fc403049-5b3e-45fa-80b8-6d1f53ab171c" (UID: "fc403049-5b3e-45fa-80b8-6d1f53ab171c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:21:18.010194 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:18.010163    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e036ad7-5g2sj"
Nov 03 20:21:18.119315 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:18.119273    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf457154-f1db-4bf4-9f65-3604c92f6032-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8e036ad7-5g2sj\" (UID: \"bf457154-f1db-4bf4-9f65-3604c92f6032\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e036ad7-5g2sj"
Nov 03 20:21:18.119654 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:18.119629    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf457154-f1db-4bf4-9f65-3604c92f6032-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8e036ad7-5g2sj\" (UID: \"bf457154-f1db-4bf4-9f65-3604c92f6032\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e036ad7-5g2sj"
Nov 03 20:21:18.119793 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:18.119760    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htvtm\" (UniqueName: \"kubernetes.io/projected/bf457154-f1db-4bf4-9f65-3604c92f6032-kube-api-access-htvtm\") pod \"maas-api-revoke-sub-8e036ad7-5g2sj\" (UID: \"bf457154-f1db-4bf4-9f65-3604c92f6032\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e036ad7-5g2sj"
Nov 03 20:21:18.120359 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:18.120337    5570 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc403049-5b3e-45fa-80b8-6d1f53ab171c-maas-api-xks-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:21:18.134073 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:18.134036    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7h8cg\" (UniqueName: \"kubernetes.io/projected/fc403049-5b3e-45fa-80b8-6d1f53ab171c-kube-api-access-7h8cg\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:21:18.145162 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:18.144892    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh" podUID="2d3647f2-89e9-4908-a8ee-d84e527abeb0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:18.258492 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:18.258445    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf457154-f1db-4bf4-9f65-3604c92f6032-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8e036ad7-5g2sj\" (UID: \"bf457154-f1db-4bf4-9f65-3604c92f6032\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e036ad7-5g2sj"
Nov 03 20:21:18.258922 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:18.258895    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf457154-f1db-4bf4-9f65-3604c92f6032-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8e036ad7-5g2sj\" (UID: \"bf457154-f1db-4bf4-9f65-3604c92f6032\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e036ad7-5g2sj"
Nov 03 20:21:18.259096 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:18.259079    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htvtm\" (UniqueName: \"kubernetes.io/projected/bf457154-f1db-4bf4-9f65-3604c92f6032-kube-api-access-htvtm\") pod \"maas-api-revoke-sub-8e036ad7-5g2sj\" (UID: \"bf457154-f1db-4bf4-9f65-3604c92f6032\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e036ad7-5g2sj"
Nov 03 20:21:18.263809 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:18.260635    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf457154-f1db-4bf4-9f65-3604c92f6032-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8e036ad7-5g2sj\" (UID: \"bf457154-f1db-4bf4-9f65-3604c92f6032\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e036ad7-5g2sj"
Nov 03 20:21:18.367900 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:18.361833    5570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-e40816/simulator-w0-e40816-kserve-688f79796f-2tvw5" podUID="c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:18.405947 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:18.405835    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e036ad7-5g2sj"]
Nov 03 20:21:18.419822 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:18.418808    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-htvtm\" (UniqueName: \"kubernetes.io/projected/bf457154-f1db-4bf4-9f65-3604c92f6032-kube-api-access-htvtm\") pod \"maas-api-revoke-sub-8e036ad7-5g2sj\" (UID: \"bf457154-f1db-4bf4-9f65-3604c92f6032\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e036ad7-5g2sj"
Nov 03 20:21:18.494835 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:18.493404    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf457154-f1db-4bf4-9f65-3604c92f6032-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8e036ad7-5g2sj\" (UID: \"bf457154-f1db-4bf4-9f65-3604c92f6032\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e036ad7-5g2sj"
Nov 03 20:21:18.619482 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:18.619385    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fca5b006-j488g"
Nov 03 20:21:18.959842 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:18.758715    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e036ad7-5g2sj"
Nov 03 20:21:19.025810 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:19.023163    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/distinct-2-w3-482722-kserve-74bf58cf7d-v9cjw" podUID="9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:19.054135 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:19.054071    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/simulator-w0-e40816-kserve-688f79796f-2tvw5" podUID="c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Nov 03 20:21:19.336833 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:19.062466    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/premium-w4-e80203-kserve-5f9f445456-bms9d" podUID="6ad76efc-20c2-4951-b22a-c5a39cd39ed4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Nov 03 20:21:19.420754 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:19.062694    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/distinct-2-w4-e80203-kserve-756f9b9bc6-69gxj" podUID="ccf5dc2f-f116-4917-bb1a-b136d8ed17f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:19.420754 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:19.070981    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/premium-w1-6fc748-kserve-7474db9cd7-nf2xk" podUID="830e062e-83fc-4de7-91a7-5534dac1d46b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Nov 03 20:21:19.420754 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:19.071255    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/premium-w3-482722-kserve-689dc7bd5d-x8rjp" podUID="369045c3-4d6a-49a7-9bc6-f2939883bee8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Nov 03 20:21:19.420754 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:19.071464    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/distinct-2-w0-e40816-kserve-79fb46654d-s242x" podUID="4ad9b048-925c-4d01-a839-a5f25d4dd02d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded"
Nov 03 20:21:19.420754 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:19.077590    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/premium-w2-9973d9-kserve-5c6dd4669c-5gv8x" podUID="cefc89ad-7ae1-433b-979d-6601678a6d98" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:19.420754 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:19.138905    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/distinct-w3-482722-kserve-d876967c-kd488" podUID="9a97f9cc-2261-471d-9053-51ee965850bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:19.420754 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:19.138964    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-2-w6-9e1bcb-kserve-7d96d9f6f6-4f7fl" podUID="56353b0f-08ee-4ada-9dc0-ac5afd5a59c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded"
Nov 03 20:21:19.523245 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:19.523206    5570 patch_prober.go:29] interesting pod/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-z4kzq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:21:19.523541 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:19.523503    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-z4kzq" podUID="e185c296-cd84-4b0c-9316-ef10085cda0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:19.523729 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:19.166937    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh" podUID="2d3647f2-89e9-4908-a8ee-d84e527abeb0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:19.580369 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:19.142909    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz" podUID="3a715ef1-38e9-4386-9eec-e10e8df11a10" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:19.983808 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:19.638069    5570 patch_prober.go:29] interesting pod/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-b4zmq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:21:19.983808 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:19.962948    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-b4zmq" podUID="8f7ff673-520b-4dd1-a76d-0e7aa1053c3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:20.388075 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:20.387954    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/simulator-w0-e40816-kserve-688f79796f-2tvw5" podUID="c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Nov 03 20:21:20.422313 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:20.422265    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/distinct-w3-482722-kserve-d876967c-kd488" podUID="9a97f9cc-2261-471d-9053-51ee965850bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded"
Nov 03 20:21:20.499812 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:20.485353    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-2-w6-9e1bcb-kserve-7d96d9f6f6-4f7fl" podUID="56353b0f-08ee-4ada-9dc0-ac5afd5a59c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:20.511710 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:20.485056    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/distinct-2-w3-482722-kserve-74bf58cf7d-v9cjw" podUID="9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:20.707865 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:20.661763    5570 patch_prober.go:29] interesting pod/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-wv4cn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:21:20.707865 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:20.680078    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-wv4cn" podUID="c1e2eadf-ede0-4ea2-9eea-f25a39779795" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:20.707865 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:20.689812    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/premium-w4-e80203-kserve-5f9f445456-bms9d" podUID="6ad76efc-20c2-4951-b22a-c5a39cd39ed4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Nov 03 20:21:20.707865 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:20.690232    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/distinct-2-w4-e80203-kserve-756f9b9bc6-69gxj" podUID="ccf5dc2f-f116-4917-bb1a-b136d8ed17f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded"
Nov 03 20:21:20.707865 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:20.690555    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/premium-w1-6fc748-kserve-7474db9cd7-nf2xk" podUID="830e062e-83fc-4de7-91a7-5534dac1d46b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Nov 03 20:21:20.707865 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:20.690933    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/premium-w3-482722-kserve-689dc7bd5d-x8rjp" podUID="369045c3-4d6a-49a7-9bc6-f2939883bee8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Nov 03 20:21:20.707865 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:20.691242    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/distinct-2-w0-e40816-kserve-79fb46654d-s242x" podUID="4ad9b048-925c-4d01-a839-a5f25d4dd02d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded"
Nov 03 20:21:20.707865 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:20.691508    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/premium-w2-9973d9-kserve-5c6dd4669c-5gv8x" podUID="cefc89ad-7ae1-433b-979d-6601678a6d98" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Nov 03 20:21:20.709739 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:20.595926    5570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-e80203/distinct-2-w4-e80203-kserve-756f9b9bc6-69gxj" podUID="ccf5dc2f-f116-4917-bb1a-b136d8ed17f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/health\": context deadline exceeded"
Nov 03 20:21:20.993996 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:20.710006    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh" podUID="2d3647f2-89e9-4908-a8ee-d84e527abeb0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:21.003991 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:20.763844    5570 patch_prober.go:29] interesting pod/e2e-worker-w0-e40816-openshift-default-744c57967c-5bn7t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:21:21.004271 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:21.004236    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-5bn7t" podUID="26831c71-2091-4090-843d-f7268d257114" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:21.004440 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:20.766624    5570 patch_prober.go:29] interesting pod/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-wkmg8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:21:21.016095 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:21.016059    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-wkmg8" podUID="e314286e-4b2d-4fae-bc59-528809a8dbce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:21.021174 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:20.990873    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz" podUID="3a715ef1-38e9-4386-9eec-e10e8df11a10" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:21.369845 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:21.352887    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-6nh8m" podUID="b31639d5-0583-4706-b580-f5752fa502a4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:21.402596 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:21.402557    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/simulator-w0-e40816-kserve-688f79796f-2tvw5" podUID="c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:21.448792 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:21.437906    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/distinct-w3-482722-kserve-d876967c-kd488" podUID="9a97f9cc-2261-471d-9053-51ee965850bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:21.622138 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:21.610874    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-cpfzl" podUID="30b21f09-fd9e-4836-9fc3-c19f818c9022" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": context deadline exceeded"
Nov 03 20:21:21.622517 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:21.610914    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87s5mf" podUID="b7306cb9-eeae-447b-b834-870c8d49a5aa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": context deadline exceeded"
Nov 03 20:21:21.655018 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:21.654977    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/distinct-2-w3-482722-kserve-74bf58cf7d-v9cjw" podUID="9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:21.697160 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:21:21.697128    5570 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.177s"
Nov 03 20:21:21.697350 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:21.697208    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9973d9/premium-w2-9973d9-kserve-5c6dd4669c-5gv8x"
Nov 03 20:21:21.697350 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:21.697241    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz"
Nov 03 20:21:21.697350 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:21.697270    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-9e1bcb/premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh"
Nov 03 20:21:21.697350 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:21.697296    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-e80203/distinct-2-w4-e80203-kserve-756f9b9bc6-69gxj"
Nov 03 20:21:21.697350 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:21.697317    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6fc748/premium-w1-6fc748-kserve-7474db9cd7-nf2xk"
Nov 03 20:21:21.697350 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:21.697338    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-482722/premium-w3-482722-kserve-689dc7bd5d-x8rjp"
Nov 03 20:21:21.697705 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:21.697377    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-e80203/premium-w4-e80203-kserve-5f9f445456-bms9d"
Nov 03 20:21:21.723645 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:21.723605    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e40816/simulator-w0-e40816-kserve-688f79796f-2tvw5"
Nov 03 20:21:21.728514 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:21.728264    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-482722/distinct-w3-482722-kserve-d876967c-kd488"
Nov 03 20:21:21.728930 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:21.728889    5570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-482722/distinct-2-w3-482722-kserve-74bf58cf7d-v9cjw" podUID="9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:21.732336 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:21.732305    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/distinct-2-w0-e40816-kserve-79fb46654d-s242x" podUID="4ad9b048-925c-4d01-a839-a5f25d4dd02d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:21.735894 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:21.734223    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-2-w6-9e1bcb-kserve-7d96d9f6f6-4f7fl"
Nov 03 20:21:21.735894 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:21.735886    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-482722/distinct-2-w3-482722-kserve-74bf58cf7d-v9cjw"
Nov 03 20:21:21.782344 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:21.782312    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e40816/distinct-2-w0-e40816-kserve-79fb46654d-s242x"
Nov 03 20:21:21.979253 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:21.979212    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e036ad7-5g2sj"]
Nov 03 20:21:22.688590 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:22.688547    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e036ad7-5g2sj" event={"ID":"bf457154-f1db-4bf4-9f65-3604c92f6032","Type":"ContainerStarted","Data":"126130e48b1ef8613d48e9bc21b7c9cf2466320cbd66e077de2689975c62af81"}
Nov 03 20:21:22.689070 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:22.688665    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e036ad7-5g2sj" event={"ID":"bf457154-f1db-4bf4-9f65-3604c92f6032","Type":"ContainerStarted","Data":"3183da987ae335637e4e010ff5a3f7aef66b0b716c0ddf72f5950253e02e4b3f"}
Nov 03 20:21:22.737818 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:22.737753    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e036ad7-5g2sj" podStartSLOduration=7.737738541 podStartE2EDuration="7.737738541s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:21:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:21:22.733672245 +0000 UTC m=+1016.886334135" watchObservedRunningTime="2026-11-03 20:21:22.737738541 +0000 UTC m=+1016.890400459"
Nov 03 20:21:23.698609 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:23.698577    5570 generic.go:356] "Generic (PLEG): container finished" podID="bf457154-f1db-4bf4-9f65-3604c92f6032" containerID="126130e48b1ef8613d48e9bc21b7c9cf2466320cbd66e077de2689975c62af81" exitCode=0
Nov 03 20:21:23.699028 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:23.698662    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e036ad7-5g2sj" event={"ID":"bf457154-f1db-4bf4-9f65-3604c92f6032","Type":"ContainerDied","Data":"126130e48b1ef8613d48e9bc21b7c9cf2466320cbd66e077de2689975c62af81"}
Nov 03 20:21:24.895175 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:24.895147    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e036ad7-5g2sj"
Nov 03 20:21:25.043756 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:25.043665    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf457154-f1db-4bf4-9f65-3604c92f6032-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf457154-f1db-4bf4-9f65-3604c92f6032-maas-api-openshift-ca\") pod \"bf457154-f1db-4bf4-9f65-3604c92f6032\" (UID: \"bf457154-f1db-4bf4-9f65-3604c92f6032\") "
Nov 03 20:21:25.043756 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:25.043715    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf457154-f1db-4bf4-9f65-3604c92f6032-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf457154-f1db-4bf4-9f65-3604c92f6032-maas-api-xks-ca\") pod \"bf457154-f1db-4bf4-9f65-3604c92f6032\" (UID: \"bf457154-f1db-4bf4-9f65-3604c92f6032\") "
Nov 03 20:21:25.044009 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:25.043895    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf457154-f1db-4bf4-9f65-3604c92f6032-kube-api-access-htvtm\" (UniqueName: \"kubernetes.io/projected/bf457154-f1db-4bf4-9f65-3604c92f6032-kube-api-access-htvtm\") pod \"bf457154-f1db-4bf4-9f65-3604c92f6032\" (UID: \"bf457154-f1db-4bf4-9f65-3604c92f6032\") "
Nov 03 20:21:25.044192 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:25.044164    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf457154-f1db-4bf4-9f65-3604c92f6032-maas-api-openshift-ca" pod "bf457154-f1db-4bf4-9f65-3604c92f6032" (UID: "bf457154-f1db-4bf4-9f65-3604c92f6032"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:21:25.044559 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:25.044538    5570 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf457154-f1db-4bf4-9f65-3604c92f6032-maas-api-openshift-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:21:25.045984 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:25.045961    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf457154-f1db-4bf4-9f65-3604c92f6032-maas-api-xks-ca" pod "bf457154-f1db-4bf4-9f65-3604c92f6032" (UID: "bf457154-f1db-4bf4-9f65-3604c92f6032"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:21:25.046140 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:25.046118    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf457154-f1db-4bf4-9f65-3604c92f6032-kube-api-access-htvtm" pod "bf457154-f1db-4bf4-9f65-3604c92f6032" (UID: "bf457154-f1db-4bf4-9f65-3604c92f6032"). InnerVolumeSpecName "kube-api-access-htvtm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:21:25.146155 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:25.146119    5570 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf457154-f1db-4bf4-9f65-3604c92f6032-maas-api-xks-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:21:25.146155 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:25.146148    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-htvtm\" (UniqueName: \"kubernetes.io/projected/bf457154-f1db-4bf4-9f65-3604c92f6032-kube-api-access-htvtm\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:21:25.715913 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:25.715883    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e036ad7-5g2sj"
Nov 03 20:21:25.716095 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:25.715918    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e036ad7-5g2sj" event={"ID":"bf457154-f1db-4bf4-9f65-3604c92f6032","Type":"ContainerDied","Data":"3183da987ae335637e4e010ff5a3f7aef66b0b716c0ddf72f5950253e02e4b3f"}
Nov 03 20:21:25.716095 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:25.716002    5570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3183da987ae335637e4e010ff5a3f7aef66b0b716c0ddf72f5950253e02e4b3f"
Nov 03 20:21:27.229555 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:21:27.229511    5570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8bwhz" containerName="extractor"
Nov 03 20:21:30.264495 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:30.264457    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-852e3ec1-6vq7t"]
Nov 03 20:21:30.266180 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:30.266148    5570 cpu_manager.go:414] "containerMap: removing container" podUID="bf457154-f1db-4bf4-9f65-3604c92f6032" containerName="revoke-keys"
Nov 03 20:21:30.266347 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:30.266332    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf457154-f1db-4bf4-9f65-3604c92f6032" containerName="revoke-keys"
Nov 03 20:21:30.266643 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:30.266626    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf457154-f1db-4bf4-9f65-3604c92f6032" containerName="revoke-keys"
Nov 03 20:21:30.272622 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:30.272596    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-852e3ec1-6vq7t"
Nov 03 20:21:30.288917 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:30.288881    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-852e3ec1-6vq7t"]
Nov 03 20:21:30.309457 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:30.291269    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:21:30.309719 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:30.291834    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l7s5w\""
Nov 03 20:21:30.373148 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:30.373114    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10788aba-4e6e-44dd-b3c0-eacc930678f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-852e3ec1-6vq7t\" (UID: \"10788aba-4e6e-44dd-b3c0-eacc930678f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-852e3ec1-6vq7t"
Nov 03 20:21:30.373371 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:30.373346    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjgbv\" (UniqueName: \"kubernetes.io/projected/10788aba-4e6e-44dd-b3c0-eacc930678f9-kube-api-access-jjgbv\") pod \"maas-api-revoke-sub-852e3ec1-6vq7t\" (UID: \"10788aba-4e6e-44dd-b3c0-eacc930678f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-852e3ec1-6vq7t"
Nov 03 20:21:30.373548 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:30.373522    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10788aba-4e6e-44dd-b3c0-eacc930678f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-852e3ec1-6vq7t\" (UID: \"10788aba-4e6e-44dd-b3c0-eacc930678f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-852e3ec1-6vq7t"
Nov 03 20:21:30.475027 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:30.474991    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjgbv\" (UniqueName: \"kubernetes.io/projected/10788aba-4e6e-44dd-b3c0-eacc930678f9-kube-api-access-jjgbv\") pod \"maas-api-revoke-sub-852e3ec1-6vq7t\" (UID: \"10788aba-4e6e-44dd-b3c0-eacc930678f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-852e3ec1-6vq7t"
Nov 03 20:21:30.475215 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:30.475122    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10788aba-4e6e-44dd-b3c0-eacc930678f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-852e3ec1-6vq7t\" (UID: \"10788aba-4e6e-44dd-b3c0-eacc930678f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-852e3ec1-6vq7t"
Nov 03 20:21:30.475275 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:30.475262    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10788aba-4e6e-44dd-b3c0-eacc930678f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-852e3ec1-6vq7t\" (UID: \"10788aba-4e6e-44dd-b3c0-eacc930678f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-852e3ec1-6vq7t"
Nov 03 20:21:30.475993 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:30.475958    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10788aba-4e6e-44dd-b3c0-eacc930678f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-852e3ec1-6vq7t\" (UID: \"10788aba-4e6e-44dd-b3c0-eacc930678f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-852e3ec1-6vq7t"
Nov 03 20:21:30.477516 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:30.477489    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10788aba-4e6e-44dd-b3c0-eacc930678f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-852e3ec1-6vq7t\" (UID: \"10788aba-4e6e-44dd-b3c0-eacc930678f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-852e3ec1-6vq7t"
Nov 03 20:21:30.494894 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:30.494859    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjgbv\" (UniqueName: \"kubernetes.io/projected/10788aba-4e6e-44dd-b3c0-eacc930678f9-kube-api-access-jjgbv\") pod \"maas-api-revoke-sub-852e3ec1-6vq7t\" (UID: \"10788aba-4e6e-44dd-b3c0-eacc930678f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-852e3ec1-6vq7t"
Nov 03 20:21:30.634070 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:30.633995    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-852e3ec1-6vq7t"
Nov 03 20:21:31.060661 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:31.060122    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-852e3ec1-6vq7t"]
Nov 03 20:21:31.786362 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:31.786269    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-852e3ec1-6vq7t" event={"ID":"10788aba-4e6e-44dd-b3c0-eacc930678f9","Type":"ContainerStarted","Data":"218686cf266f775f69ae8515a4e96ff410b158739543489796ea28e215b5d176"}
Nov 03 20:21:31.786362 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:31.786330    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-852e3ec1-6vq7t" event={"ID":"10788aba-4e6e-44dd-b3c0-eacc930678f9","Type":"ContainerStarted","Data":"667edbe70091e6ce5615c17c8e3dace51f8134266f86025dcd671c0df502dc2a"}
Nov 03 20:21:31.831183 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:31.831110    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-852e3ec1-6vq7t" podStartSLOduration=1.831089837 podStartE2EDuration="1.831089837s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:21:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:21:31.826027172 +0000 UTC m=+1025.978689042" watchObservedRunningTime="2026-11-03 20:21:31.831089837 +0000 UTC m=+1025.983751730"
Nov 03 20:21:32.804918 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:32.804751    5570 generic.go:356] "Generic (PLEG): container finished" podID="10788aba-4e6e-44dd-b3c0-eacc930678f9" containerID="218686cf266f775f69ae8515a4e96ff410b158739543489796ea28e215b5d176" exitCode=0
Nov 03 20:21:32.805699 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:32.805668    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-852e3ec1-6vq7t" event={"ID":"10788aba-4e6e-44dd-b3c0-eacc930678f9","Type":"ContainerDied","Data":"218686cf266f775f69ae8515a4e96ff410b158739543489796ea28e215b5d176"}
Nov 03 20:21:34.024023 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:34.023987    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-852e3ec1-6vq7t"
Nov 03 20:21:34.148578 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:34.148494    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/10788aba-4e6e-44dd-b3c0-eacc930678f9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10788aba-4e6e-44dd-b3c0-eacc930678f9-maas-api-xks-ca\") pod \"10788aba-4e6e-44dd-b3c0-eacc930678f9\" (UID: \"10788aba-4e6e-44dd-b3c0-eacc930678f9\") "
Nov 03 20:21:34.148578 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:34.148552    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10788aba-4e6e-44dd-b3c0-eacc930678f9-kube-api-access-jjgbv\" (UniqueName: \"kubernetes.io/projected/10788aba-4e6e-44dd-b3c0-eacc930678f9-kube-api-access-jjgbv\") pod \"10788aba-4e6e-44dd-b3c0-eacc930678f9\" (UID: \"10788aba-4e6e-44dd-b3c0-eacc930678f9\") "
Nov 03 20:21:34.148937 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:34.148917    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10788aba-4e6e-44dd-b3c0-eacc930678f9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10788aba-4e6e-44dd-b3c0-eacc930678f9-maas-api-openshift-ca\") pod \"10788aba-4e6e-44dd-b3c0-eacc930678f9\" (UID: \"10788aba-4e6e-44dd-b3c0-eacc930678f9\") "
Nov 03 20:21:34.149357 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:34.149312    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10788aba-4e6e-44dd-b3c0-eacc930678f9-maas-api-openshift-ca" pod "10788aba-4e6e-44dd-b3c0-eacc930678f9" (UID: "10788aba-4e6e-44dd-b3c0-eacc930678f9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:21:34.149652 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:34.149625    5570 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10788aba-4e6e-44dd-b3c0-eacc930678f9-maas-api-openshift-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:21:34.150952 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:34.150922    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10788aba-4e6e-44dd-b3c0-eacc930678f9-maas-api-xks-ca" pod "10788aba-4e6e-44dd-b3c0-eacc930678f9" (UID: "10788aba-4e6e-44dd-b3c0-eacc930678f9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:21:34.151054 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:34.150947    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10788aba-4e6e-44dd-b3c0-eacc930678f9-kube-api-access-jjgbv" pod "10788aba-4e6e-44dd-b3c0-eacc930678f9" (UID: "10788aba-4e6e-44dd-b3c0-eacc930678f9"). InnerVolumeSpecName "kube-api-access-jjgbv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:21:34.251325 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:34.251295    5570 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10788aba-4e6e-44dd-b3c0-eacc930678f9-maas-api-xks-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:21:34.251325 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:34.251323    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjgbv\" (UniqueName: \"kubernetes.io/projected/10788aba-4e6e-44dd-b3c0-eacc930678f9-kube-api-access-jjgbv\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:21:34.595059 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:34.595026    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-9798ad8e-gw-openshift-default-575b7f8f9c-xg9nd"]
Nov 03 20:21:34.596063 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:34.596022    5570 cpu_manager.go:414] "containerMap: removing container" podUID="10788aba-4e6e-44dd-b3c0-eacc930678f9" containerName="revoke-keys"
Nov 03 20:21:34.596063 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:34.596049    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10788aba-4e6e-44dd-b3c0-eacc930678f9" containerName="revoke-keys"
Nov 03 20:21:34.596568 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:34.596287    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="10788aba-4e6e-44dd-b3c0-eacc930678f9" containerName="revoke-keys"
Nov 03 20:21:34.600291 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:34.600269    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-9798ad8e-gw-openshift-default-575b7f8f9c-xg9nd"
Nov 03 20:21:34.624743 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:34.624713    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-9798ad8e-gw-openshift-default-575b7f8f9c-xg9nd"]
Nov 03 20:21:34.656877 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:34.656848    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-9798ad8e-gw-openshift-default-dockercfg-c6kdd\""
Nov 03 20:21:34.665696 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:34.665658    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-baa8c991-tz94q"]
Nov 03 20:21:34.670708 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:34.670683    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-baa8c991-tz94q"
Nov 03 20:21:34.701834 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:34.694145    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-baa8c991-tz94q"]
Nov 03 20:21:34.787795 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:34.780162    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v67wt\" (UniqueName: \"kubernetes.io/projected/b9bd3e2e-35ec-482d-8dc2-1a18f255c220-kube-api-access-v67wt\") pod \"e2e-migrate-9798ad8e-gw-openshift-default-575b7f8f9c-xg9nd\" (UID: \"b9bd3e2e-35ec-482d-8dc2-1a18f255c220\") " pod="openshift-ingress/e2e-migrate-9798ad8e-gw-openshift-default-575b7f8f9c-xg9nd"
Nov 03 20:21:34.787795 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:34.780512    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9bd3e2e-35ec-482d-8dc2-1a18f255c220-istio-data\") pod \"e2e-migrate-9798ad8e-gw-openshift-default-575b7f8f9c-xg9nd\" (UID: \"b9bd3e2e-35ec-482d-8dc2-1a18f255c220\") " pod="openshift-ingress/e2e-migrate-9798ad8e-gw-openshift-default-575b7f8f9c-xg9nd"
Nov 03 20:21:34.787795 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:34.780609    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmt6f\" (UniqueName: \"kubernetes.io/projected/8f41cff1-9c8c-48a8-a719-6bb52f8095a7-kube-api-access-xmt6f\") pod \"maas-api-revoke-sub-baa8c991-tz94q\" (UID: \"8f41cff1-9c8c-48a8-a719-6bb52f8095a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-baa8c991-tz94q"
Nov 03 20:21:34.787795 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:34.780753    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9bd3e2e-35ec-482d-8dc2-1a18f255c220-istiod-ca-cert\") pod \"e2e-migrate-9798ad8e-gw-openshift-default-575b7f8f9c-xg9nd\" (UID: \"b9bd3e2e-35ec-482d-8dc2-1a18f255c220\") " pod="openshift-ingress/e2e-migrate-9798ad8e-gw-openshift-default-575b7f8f9c-xg9nd"
Nov 03 20:21:34.787795 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:34.780900    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f41cff1-9c8c-48a8-a719-6bb52f8095a7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-baa8c991-tz94q\" (UID: \"8f41cff1-9c8c-48a8-a719-6bb52f8095a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-baa8c991-tz94q"
Nov 03 20:21:34.787795 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:34.780978    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9bd3e2e-35ec-482d-8dc2-1a18f255c220-workload-socket\") pod \"e2e-migrate-9798ad8e-gw-openshift-default-575b7f8f9c-xg9nd\" (UID: \"b9bd3e2e-35ec-482d-8dc2-1a18f255c220\") " pod="openshift-ingress/e2e-migrate-9798ad8e-gw-openshift-default-575b7f8f9c-xg9nd"
Nov 03 20:21:34.787795 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:34.781051    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9bd3e2e-35ec-482d-8dc2-1a18f255c220-istio-token\") pod \"e2e-migrate-9798ad8e-gw-openshift-default-575b7f8f9c-xg9nd\" (UID: \"b9bd3e2e-35ec-482d-8dc2-1a18f255c220\") " pod="openshift-ingress/e2e-migrate-9798ad8e-gw-openshift-default-575b7f8f9c-xg9nd"
Nov 03 20:21:34.787795 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:34.781132    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f41cff1-9c8c-48a8-a719-6bb52f8095a7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-baa8c991-tz94q\" (UID: \"8f41cff1-9c8c-48a8-a719-6bb52f8095a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-baa8c991-tz94q"
Nov 03 20:21:34.787795 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:34.781172    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9bd3e2e-35ec-482d-8dc2-1a18f255c220-istio-envoy\") pod \"e2e-migrate-9798ad8e-gw-openshift-default-575b7f8f9c-xg9nd\" (UID: \"b9bd3e2e-35ec-482d-8dc2-1a18f255c220\") " pod="openshift-ingress/e2e-migrate-9798ad8e-gw-openshift-default-575b7f8f9c-xg9nd"
Nov 03 20:21:34.787795 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:34.781203    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9bd3e2e-35ec-482d-8dc2-1a18f255c220-workload-certs\") pod \"e2e-migrate-9798ad8e-gw-openshift-default-575b7f8f9c-xg9nd\" (UID: \"b9bd3e2e-35ec-482d-8dc2-1a18f255c220\") " pod="openshift-ingress/e2e-migrate-9798ad8e-gw-openshift-default-575b7f8f9c-xg9nd"
Nov 03 20:21:34.787795 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:34.781289    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9bd3e2e-35ec-482d-8dc2-1a18f255c220-istio-podinfo\") pod \"e2e-migrate-9798ad8e-gw-openshift-default-575b7f8f9c-xg9nd\" (UID: \"b9bd3e2e-35ec-482d-8dc2-1a18f255c220\") " pod="openshift-ingress/e2e-migrate-9798ad8e-gw-openshift-default-575b7f8f9c-xg9nd"
Nov 03 20:21:34.787795 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:34.781345    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9bd3e2e-35ec-482d-8dc2-1a18f255c220-credential-socket\") pod \"e2e-migrate-9798ad8e-gw-openshift-default-575b7f8f9c-xg9nd\" (UID: \"b9bd3e2e-35ec-482d-8dc2-1a18f255c220\") " pod="openshift-ingress/e2e-migrate-9798ad8e-gw-openshift-default-575b7f8f9c-xg9nd"
Nov 03 20:21:35.008615 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:34.990548    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmt6f\" (UniqueName: \"kubernetes.io/projected/8f41cff1-9c8c-48a8-a719-6bb52f8095a7-kube-api-access-xmt6f\") pod \"maas-api-revoke-sub-baa8c991-tz94q\" (UID: \"8f41cff1-9c8c-48a8-a719-6bb52f8095a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-baa8c991-tz94q"
Nov 03 20:21:35.008615 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:34.990707    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9bd3e2e-35ec-482d-8dc2-1a18f255c220-istiod-ca-cert\") pod \"e2e-migrate-9798ad8e-gw-openshift-default-575b7f8f9c-xg9nd\" (UID: \"b9bd3e2e-35ec-482d-8dc2-1a18f255c220\") " pod="openshift-ingress/e2e-migrate-9798ad8e-gw-openshift-default-575b7f8f9c-xg9nd"
Nov 03 20:21:35.008615 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:34.990768    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f41cff1-9c8c-48a8-a719-6bb52f8095a7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-baa8c991-tz94q\" (UID: \"8f41cff1-9c8c-48a8-a719-6bb52f8095a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-baa8c991-tz94q"
Nov 03 20:21:35.008615 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:34.990865    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9bd3e2e-35ec-482d-8dc2-1a18f255c220-workload-socket\") pod \"e2e-migrate-9798ad8e-gw-openshift-default-575b7f8f9c-xg9nd\" (UID: \"b9bd3e2e-35ec-482d-8dc2-1a18f255c220\") " pod="openshift-ingress/e2e-migrate-9798ad8e-gw-openshift-default-575b7f8f9c-xg9nd"
Nov 03 20:21:35.008615 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:34.990953    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9bd3e2e-35ec-482d-8dc2-1a18f255c220-istio-token\") pod \"e2e-migrate-9798ad8e-gw-openshift-default-575b7f8f9c-xg9nd\" (UID: \"b9bd3e2e-35ec-482d-8dc2-1a18f255c220\") " pod="openshift-ingress/e2e-migrate-9798ad8e-gw-openshift-default-575b7f8f9c-xg9nd"
Nov 03 20:21:35.008615 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:34.991032    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f41cff1-9c8c-48a8-a719-6bb52f8095a7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-baa8c991-tz94q\" (UID: \"8f41cff1-9c8c-48a8-a719-6bb52f8095a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-baa8c991-tz94q"
Nov 03 20:21:35.008615 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:34.991069    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9bd3e2e-35ec-482d-8dc2-1a18f255c220-istio-envoy\") pod \"e2e-migrate-9798ad8e-gw-openshift-default-575b7f8f9c-xg9nd\" (UID: \"b9bd3e2e-35ec-482d-8dc2-1a18f255c220\") " pod="openshift-ingress/e2e-migrate-9798ad8e-gw-openshift-default-575b7f8f9c-xg9nd"
Nov 03 20:21:35.008615 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:34.991096    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9bd3e2e-35ec-482d-8dc2-1a18f255c220-workload-certs\") pod \"e2e-migrate-9798ad8e-gw-openshift-default-575b7f8f9c-xg9nd\" (UID: \"b9bd3e2e-35ec-482d-8dc2-1a18f255c220\") " pod="openshift-ingress/e2e-migrate-9798ad8e-gw-openshift-default-575b7f8f9c-xg9nd"
Nov 03 20:21:35.008615 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:34.991177    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9bd3e2e-35ec-482d-8dc2-1a18f255c220-istio-podinfo\") pod \"e2e-migrate-9798ad8e-gw-openshift-default-575b7f8f9c-xg9nd\" (UID: \"b9bd3e2e-35ec-482d-8dc2-1a18f255c220\") " pod="openshift-ingress/e2e-migrate-9798ad8e-gw-openshift-default-575b7f8f9c-xg9nd"
Nov 03 20:21:35.008615 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:34.991231    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9bd3e2e-35ec-482d-8dc2-1a18f255c220-credential-socket\") pod \"e2e-migrate-9798ad8e-gw-openshift-default-575b7f8f9c-xg9nd\" (UID: \"b9bd3e2e-35ec-482d-8dc2-1a18f255c220\") " pod="openshift-ingress/e2e-migrate-9798ad8e-gw-openshift-default-575b7f8f9c-xg9nd"
Nov 03 20:21:35.008615 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:34.991315    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v67wt\" (UniqueName: \"kubernetes.io/projected/b9bd3e2e-35ec-482d-8dc2-1a18f255c220-kube-api-access-v67wt\") pod \"e2e-migrate-9798ad8e-gw-openshift-default-575b7f8f9c-xg9nd\" (UID: \"b9bd3e2e-35ec-482d-8dc2-1a18f255c220\") " pod="openshift-ingress/e2e-migrate-9798ad8e-gw-openshift-default-575b7f8f9c-xg9nd"
Nov 03 20:21:35.008615 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:34.991348    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9bd3e2e-35ec-482d-8dc2-1a18f255c220-istio-data\") pod \"e2e-migrate-9798ad8e-gw-openshift-default-575b7f8f9c-xg9nd\" (UID: \"b9bd3e2e-35ec-482d-8dc2-1a18f255c220\") " pod="openshift-ingress/e2e-migrate-9798ad8e-gw-openshift-default-575b7f8f9c-xg9nd"
Nov 03 20:21:35.008615 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:34.994827    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9bd3e2e-35ec-482d-8dc2-1a18f255c220-istiod-ca-cert\") pod \"e2e-migrate-9798ad8e-gw-openshift-default-575b7f8f9c-xg9nd\" (UID: \"b9bd3e2e-35ec-482d-8dc2-1a18f255c220\") " pod="openshift-ingress/e2e-migrate-9798ad8e-gw-openshift-default-575b7f8f9c-xg9nd"
Nov 03 20:21:35.011465 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:35.011432    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f41cff1-9c8c-48a8-a719-6bb52f8095a7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-baa8c991-tz94q\" (UID: \"8f41cff1-9c8c-48a8-a719-6bb52f8095a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-baa8c991-tz94q"
Nov 03 20:21:35.023903 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:35.021431    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9bd3e2e-35ec-482d-8dc2-1a18f255c220-istio-data\") pod \"e2e-migrate-9798ad8e-gw-openshift-default-575b7f8f9c-xg9nd\" (UID: \"b9bd3e2e-35ec-482d-8dc2-1a18f255c220\") " pod="openshift-ingress/e2e-migrate-9798ad8e-gw-openshift-default-575b7f8f9c-xg9nd"
Nov 03 20:21:35.024447 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:35.024418    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9bd3e2e-35ec-482d-8dc2-1a18f255c220-workload-certs\") pod \"e2e-migrate-9798ad8e-gw-openshift-default-575b7f8f9c-xg9nd\" (UID: \"b9bd3e2e-35ec-482d-8dc2-1a18f255c220\") " pod="openshift-ingress/e2e-migrate-9798ad8e-gw-openshift-default-575b7f8f9c-xg9nd"
Nov 03 20:21:35.057462 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:35.057414    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9bd3e2e-35ec-482d-8dc2-1a18f255c220-credential-socket\") pod \"e2e-migrate-9798ad8e-gw-openshift-default-575b7f8f9c-xg9nd\" (UID: \"b9bd3e2e-35ec-482d-8dc2-1a18f255c220\") " pod="openshift-ingress/e2e-migrate-9798ad8e-gw-openshift-default-575b7f8f9c-xg9nd"
Nov 03 20:21:35.064539 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:35.063368    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-852e3ec1-6vq7t" event={"ID":"10788aba-4e6e-44dd-b3c0-eacc930678f9","Type":"ContainerDied","Data":"667edbe70091e6ce5615c17c8e3dace51f8134266f86025dcd671c0df502dc2a"}
Nov 03 20:21:35.064797 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:35.064760    5570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="667edbe70091e6ce5615c17c8e3dace51f8134266f86025dcd671c0df502dc2a"
Nov 03 20:21:35.064903 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:35.063442    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-852e3ec1-6vq7t"
Nov 03 20:21:35.073371 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:35.073325    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9bd3e2e-35ec-482d-8dc2-1a18f255c220-istio-envoy\") pod \"e2e-migrate-9798ad8e-gw-openshift-default-575b7f8f9c-xg9nd\" (UID: \"b9bd3e2e-35ec-482d-8dc2-1a18f255c220\") " pod="openshift-ingress/e2e-migrate-9798ad8e-gw-openshift-default-575b7f8f9c-xg9nd"
Nov 03 20:21:35.074082 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:35.074052    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmt6f\" (UniqueName: \"kubernetes.io/projected/8f41cff1-9c8c-48a8-a719-6bb52f8095a7-kube-api-access-xmt6f\") pod \"maas-api-revoke-sub-baa8c991-tz94q\" (UID: \"8f41cff1-9c8c-48a8-a719-6bb52f8095a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-baa8c991-tz94q"
Nov 03 20:21:35.074222 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:35.074155    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9bd3e2e-35ec-482d-8dc2-1a18f255c220-workload-socket\") pod \"e2e-migrate-9798ad8e-gw-openshift-default-575b7f8f9c-xg9nd\" (UID: \"b9bd3e2e-35ec-482d-8dc2-1a18f255c220\") " pod="openshift-ingress/e2e-migrate-9798ad8e-gw-openshift-default-575b7f8f9c-xg9nd"
Nov 03 20:21:35.115041 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:35.107705    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9bd3e2e-35ec-482d-8dc2-1a18f255c220-istio-token\") pod \"e2e-migrate-9798ad8e-gw-openshift-default-575b7f8f9c-xg9nd\" (UID: \"b9bd3e2e-35ec-482d-8dc2-1a18f255c220\") " pod="openshift-ingress/e2e-migrate-9798ad8e-gw-openshift-default-575b7f8f9c-xg9nd"
Nov 03 20:21:35.174360 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:35.174311    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f41cff1-9c8c-48a8-a719-6bb52f8095a7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-baa8c991-tz94q\" (UID: \"8f41cff1-9c8c-48a8-a719-6bb52f8095a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-baa8c991-tz94q"
Nov 03 20:21:35.178703 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:35.178635    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v67wt\" (UniqueName: \"kubernetes.io/projected/b9bd3e2e-35ec-482d-8dc2-1a18f255c220-kube-api-access-v67wt\") pod \"e2e-migrate-9798ad8e-gw-openshift-default-575b7f8f9c-xg9nd\" (UID: \"b9bd3e2e-35ec-482d-8dc2-1a18f255c220\") " pod="openshift-ingress/e2e-migrate-9798ad8e-gw-openshift-default-575b7f8f9c-xg9nd"
Nov 03 20:21:35.179415 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:35.179385    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9bd3e2e-35ec-482d-8dc2-1a18f255c220-istio-podinfo\") pod \"e2e-migrate-9798ad8e-gw-openshift-default-575b7f8f9c-xg9nd\" (UID: \"b9bd3e2e-35ec-482d-8dc2-1a18f255c220\") " pod="openshift-ingress/e2e-migrate-9798ad8e-gw-openshift-default-575b7f8f9c-xg9nd"
Nov 03 20:21:35.258931 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:35.255145    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-9798ad8e-gw-openshift-default-575b7f8f9c-xg9nd"
Nov 03 20:21:35.409083 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:35.409048    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-baa8c991-tz94q"
Nov 03 20:21:35.650323 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:35.650279    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-9798ad8e-gw-openshift-default-575b7f8f9c-xg9nd"]
Nov 03 20:21:35.654027 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:21:35.653673    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9bd3e2e_35ec_482d_8dc2_1a18f255c220.slice/crio-7439411a98f60fc0301865ae8060d5ac52dd2f1a5b1624a27702e41e9194cf2b WatchSource:0}: Error finding container 7439411a98f60fc0301865ae8060d5ac52dd2f1a5b1624a27702e41e9194cf2b: Status 404 returned error can't find the container with id 7439411a98f60fc0301865ae8060d5ac52dd2f1a5b1624a27702e41e9194cf2b
Nov 03 20:21:35.657478 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:35.657445    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:21:35.657567 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:35.657520    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:21:35.657567 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:35.657549    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:21:35.734221 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:21:35.734194    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f41cff1_9c8c_48a8_a719_6bb52f8095a7.slice/crio-eaef904850ed32824f920325fcc62421aef0cf1cb33fe8fc74010a541ec37fa5 WatchSource:0}: Error finding container eaef904850ed32824f920325fcc62421aef0cf1cb33fe8fc74010a541ec37fa5: Status 404 returned error can't find the container with id eaef904850ed32824f920325fcc62421aef0cf1cb33fe8fc74010a541ec37fa5
Nov 03 20:21:35.737568 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:35.737544    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-baa8c991-tz94q"]
Nov 03 20:21:36.080618 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:36.080575    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-baa8c991-tz94q" event={"ID":"8f41cff1-9c8c-48a8-a719-6bb52f8095a7","Type":"ContainerStarted","Data":"eaef904850ed32824f920325fcc62421aef0cf1cb33fe8fc74010a541ec37fa5"}
Nov 03 20:21:36.082919 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:36.082889    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-9798ad8e-gw-openshift-default-575b7f8f9c-xg9nd" event={"ID":"b9bd3e2e-35ec-482d-8dc2-1a18f255c220","Type":"ContainerStarted","Data":"e3bed704031bf8816c6564d3549a7c19e2f619f6ff9c7ec362d1fda9e7309df2"}
Nov 03 20:21:36.083063 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:36.082945    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-9798ad8e-gw-openshift-default-575b7f8f9c-xg9nd" event={"ID":"b9bd3e2e-35ec-482d-8dc2-1a18f255c220","Type":"ContainerStarted","Data":"7439411a98f60fc0301865ae8060d5ac52dd2f1a5b1624a27702e41e9194cf2b"}
Nov 03 20:21:36.146374 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:36.146316    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-9798ad8e-gw-openshift-default-575b7f8f9c-xg9nd" podStartSLOduration=2.146295728 podStartE2EDuration="2.146295728s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:21:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:21:36.139587268 +0000 UTC m=+1030.292249156" watchObservedRunningTime="2026-11-03 20:21:36.146295728 +0000 UTC m=+1030.298957618"
Nov 03 20:21:36.269855 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:36.265958    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-9798ad8e-gw-openshift-default-575b7f8f9c-xg9nd"
Nov 03 20:21:36.288820 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:36.281932    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-9798ad8e-gw-openshift-default-575b7f8f9c-xg9nd"
Nov 03 20:21:37.092865 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:37.092742    5570 generic.go:356] "Generic (PLEG): container finished" podID="8f41cff1-9c8c-48a8-a719-6bb52f8095a7" containerID="170be724fae6bc1478f780b1bf297ccd3787593ede151862c62b912871839dca" exitCode=0
Nov 03 20:21:37.092865 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:37.092830    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-baa8c991-tz94q" event={"ID":"8f41cff1-9c8c-48a8-a719-6bb52f8095a7","Type":"ContainerDied","Data":"170be724fae6bc1478f780b1bf297ccd3787593ede151862c62b912871839dca"}
Nov 03 20:21:37.093360 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:37.093311    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-9798ad8e-gw-openshift-default-575b7f8f9c-xg9nd"
Nov 03 20:21:37.094243 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:37.094221    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-9798ad8e-gw-openshift-default-575b7f8f9c-xg9nd"
Nov 03 20:21:38.282391 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:38.282363    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-baa8c991-tz94q"
Nov 03 20:21:38.368258 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:38.368164    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f41cff1-9c8c-48a8-a719-6bb52f8095a7-kube-api-access-xmt6f\" (UniqueName: \"kubernetes.io/projected/8f41cff1-9c8c-48a8-a719-6bb52f8095a7-kube-api-access-xmt6f\") pod \"8f41cff1-9c8c-48a8-a719-6bb52f8095a7\" (UID: \"8f41cff1-9c8c-48a8-a719-6bb52f8095a7\") "
Nov 03 20:21:38.368258 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:38.368250    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f41cff1-9c8c-48a8-a719-6bb52f8095a7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f41cff1-9c8c-48a8-a719-6bb52f8095a7-maas-api-openshift-ca\") pod \"8f41cff1-9c8c-48a8-a719-6bb52f8095a7\" (UID: \"8f41cff1-9c8c-48a8-a719-6bb52f8095a7\") "
Nov 03 20:21:38.368491 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:38.368287    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f41cff1-9c8c-48a8-a719-6bb52f8095a7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f41cff1-9c8c-48a8-a719-6bb52f8095a7-maas-api-xks-ca\") pod \"8f41cff1-9c8c-48a8-a719-6bb52f8095a7\" (UID: \"8f41cff1-9c8c-48a8-a719-6bb52f8095a7\") "
Nov 03 20:21:38.368643 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:38.368611    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f41cff1-9c8c-48a8-a719-6bb52f8095a7-maas-api-openshift-ca" pod "8f41cff1-9c8c-48a8-a719-6bb52f8095a7" (UID: "8f41cff1-9c8c-48a8-a719-6bb52f8095a7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:21:38.369040 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:38.369007    5570 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f41cff1-9c8c-48a8-a719-6bb52f8095a7-maas-api-openshift-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:21:38.370465 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:38.370444    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f41cff1-9c8c-48a8-a719-6bb52f8095a7-maas-api-xks-ca" pod "8f41cff1-9c8c-48a8-a719-6bb52f8095a7" (UID: "8f41cff1-9c8c-48a8-a719-6bb52f8095a7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:21:38.370857 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:38.370836    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f41cff1-9c8c-48a8-a719-6bb52f8095a7-kube-api-access-xmt6f" pod "8f41cff1-9c8c-48a8-a719-6bb52f8095a7" (UID: "8f41cff1-9c8c-48a8-a719-6bb52f8095a7"). InnerVolumeSpecName "kube-api-access-xmt6f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:21:38.470500 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:38.470467    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmt6f\" (UniqueName: \"kubernetes.io/projected/8f41cff1-9c8c-48a8-a719-6bb52f8095a7-kube-api-access-xmt6f\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:21:38.470500 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:38.470497    5570 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f41cff1-9c8c-48a8-a719-6bb52f8095a7-maas-api-xks-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:21:39.109358 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:39.109332    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-baa8c991-tz94q"
Nov 03 20:21:39.109545 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:39.109373    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-baa8c991-tz94q" event={"ID":"8f41cff1-9c8c-48a8-a719-6bb52f8095a7","Type":"ContainerDied","Data":"eaef904850ed32824f920325fcc62421aef0cf1cb33fe8fc74010a541ec37fa5"}
Nov 03 20:21:39.109545 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:39.109423    5570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eaef904850ed32824f920325fcc62421aef0cf1cb33fe8fc74010a541ec37fa5"
Nov 03 20:21:41.615049 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:41.615015    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aafe6908-shqjq"]
Nov 03 20:21:41.615726 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:41.615711    5570 cpu_manager.go:414] "containerMap: removing container" podUID="8f41cff1-9c8c-48a8-a719-6bb52f8095a7" containerName="revoke-keys"
Nov 03 20:21:41.615808 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:41.615728    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f41cff1-9c8c-48a8-a719-6bb52f8095a7" containerName="revoke-keys"
Nov 03 20:21:41.615899 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:41.615890    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f41cff1-9c8c-48a8-a719-6bb52f8095a7" containerName="revoke-keys"
Nov 03 20:21:41.619608 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:41.619567    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aafe6908-shqjq"
Nov 03 20:21:41.655803 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:41.648140    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l7s5w\""
Nov 03 20:21:41.655803 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:41.648239    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:21:41.898212 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:41.870421    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g69f8"]
Nov 03 20:21:41.910032 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:41.880713    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5mhx\" (UniqueName: \"kubernetes.io/projected/1aec61be-2b12-40cb-ada3-88c05f790cfb-kube-api-access-p5mhx\") pod \"maas-api-revoke-sub-aafe6908-shqjq\" (UID: \"1aec61be-2b12-40cb-ada3-88c05f790cfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aafe6908-shqjq"
Nov 03 20:21:41.913926 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:41.910453    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1aec61be-2b12-40cb-ada3-88c05f790cfb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aafe6908-shqjq\" (UID: \"1aec61be-2b12-40cb-ada3-88c05f790cfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aafe6908-shqjq"
Nov 03 20:21:41.914352 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:41.914297    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1aec61be-2b12-40cb-ada3-88c05f790cfb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aafe6908-shqjq\" (UID: \"1aec61be-2b12-40cb-ada3-88c05f790cfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aafe6908-shqjq"
Nov 03 20:21:42.108548 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:42.108363    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1aec61be-2b12-40cb-ada3-88c05f790cfb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aafe6908-shqjq\" (UID: \"1aec61be-2b12-40cb-ada3-88c05f790cfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aafe6908-shqjq"
Nov 03 20:21:42.108964 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:42.108930    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1aec61be-2b12-40cb-ada3-88c05f790cfb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aafe6908-shqjq\" (UID: \"1aec61be-2b12-40cb-ada3-88c05f790cfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aafe6908-shqjq"
Nov 03 20:21:42.109433 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:42.109394    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5mhx\" (UniqueName: \"kubernetes.io/projected/1aec61be-2b12-40cb-ada3-88c05f790cfb-kube-api-access-p5mhx\") pod \"maas-api-revoke-sub-aafe6908-shqjq\" (UID: \"1aec61be-2b12-40cb-ada3-88c05f790cfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aafe6908-shqjq"
Nov 03 20:21:42.119964 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:42.119166    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1aec61be-2b12-40cb-ada3-88c05f790cfb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aafe6908-shqjq\" (UID: \"1aec61be-2b12-40cb-ada3-88c05f790cfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aafe6908-shqjq"
Nov 03 20:21:42.282921 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:42.282865    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aafe6908-shqjq"]
Nov 03 20:21:42.284827 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:42.284252    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5mhx\" (UniqueName: \"kubernetes.io/projected/1aec61be-2b12-40cb-ada3-88c05f790cfb-kube-api-access-p5mhx\") pod \"maas-api-revoke-sub-aafe6908-shqjq\" (UID: \"1aec61be-2b12-40cb-ada3-88c05f790cfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aafe6908-shqjq"
Nov 03 20:21:42.311899 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:42.249384    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1aec61be-2b12-40cb-ada3-88c05f790cfb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aafe6908-shqjq\" (UID: \"1aec61be-2b12-40cb-ada3-88c05f790cfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aafe6908-shqjq"
Nov 03 20:21:42.338894 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:42.338392    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-kmqsw"]
Nov 03 20:21:42.343719 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:42.343689    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-kmqsw"]
Nov 03 20:21:42.343935 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:42.343841    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-kmqsw"
Nov 03 20:21:42.493273 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:42.493220    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28aada86-170c-4935-80ed-e8c5ee503c14-istiod-ca-cert\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-kmqsw\" (UID: \"28aada86-170c-4935-80ed-e8c5ee503c14\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-kmqsw"
Nov 03 20:21:42.493453 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:42.493432    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28aada86-170c-4935-80ed-e8c5ee503c14-credential-socket\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-kmqsw\" (UID: \"28aada86-170c-4935-80ed-e8c5ee503c14\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-kmqsw"
Nov 03 20:21:42.493529 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:42.493463    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28aada86-170c-4935-80ed-e8c5ee503c14-istio-token\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-kmqsw\" (UID: \"28aada86-170c-4935-80ed-e8c5ee503c14\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-kmqsw"
Nov 03 20:21:42.493529 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:42.493510    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28aada86-170c-4935-80ed-e8c5ee503c14-istio-envoy\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-kmqsw\" (UID: \"28aada86-170c-4935-80ed-e8c5ee503c14\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-kmqsw"
Nov 03 20:21:42.493729 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:42.493694    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28aada86-170c-4935-80ed-e8c5ee503c14-workload-certs\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-kmqsw\" (UID: \"28aada86-170c-4935-80ed-e8c5ee503c14\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-kmqsw"
Nov 03 20:21:42.493902 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:42.493753    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28aada86-170c-4935-80ed-e8c5ee503c14-istio-podinfo\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-kmqsw\" (UID: \"28aada86-170c-4935-80ed-e8c5ee503c14\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-kmqsw"
Nov 03 20:21:42.493902 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:42.493896    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28aada86-170c-4935-80ed-e8c5ee503c14-workload-socket\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-kmqsw\" (UID: \"28aada86-170c-4935-80ed-e8c5ee503c14\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-kmqsw"
Nov 03 20:21:42.494041 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:42.494025    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28aada86-170c-4935-80ed-e8c5ee503c14-istio-data\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-kmqsw\" (UID: \"28aada86-170c-4935-80ed-e8c5ee503c14\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-kmqsw"
Nov 03 20:21:42.494087 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:42.494071    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8n6x7\" (UniqueName: \"kubernetes.io/projected/28aada86-170c-4935-80ed-e8c5ee503c14-kube-api-access-8n6x7\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-kmqsw\" (UID: \"28aada86-170c-4935-80ed-e8c5ee503c14\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-kmqsw"
Nov 03 20:21:42.596246 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:42.596155    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28aada86-170c-4935-80ed-e8c5ee503c14-istio-data\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-kmqsw\" (UID: \"28aada86-170c-4935-80ed-e8c5ee503c14\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-kmqsw"
Nov 03 20:21:42.596246 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:42.596202    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8n6x7\" (UniqueName: \"kubernetes.io/projected/28aada86-170c-4935-80ed-e8c5ee503c14-kube-api-access-8n6x7\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-kmqsw\" (UID: \"28aada86-170c-4935-80ed-e8c5ee503c14\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-kmqsw"
Nov 03 20:21:42.596462 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:42.596259    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28aada86-170c-4935-80ed-e8c5ee503c14-istiod-ca-cert\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-kmqsw\" (UID: \"28aada86-170c-4935-80ed-e8c5ee503c14\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-kmqsw"
Nov 03 20:21:42.596462 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:42.596353    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28aada86-170c-4935-80ed-e8c5ee503c14-credential-socket\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-kmqsw\" (UID: \"28aada86-170c-4935-80ed-e8c5ee503c14\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-kmqsw"
Nov 03 20:21:42.596462 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:42.596380    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28aada86-170c-4935-80ed-e8c5ee503c14-istio-token\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-kmqsw\" (UID: \"28aada86-170c-4935-80ed-e8c5ee503c14\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-kmqsw"
Nov 03 20:21:42.596462 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:42.596429    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28aada86-170c-4935-80ed-e8c5ee503c14-istio-envoy\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-kmqsw\" (UID: \"28aada86-170c-4935-80ed-e8c5ee503c14\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-kmqsw"
Nov 03 20:21:42.596676 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:42.596484    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28aada86-170c-4935-80ed-e8c5ee503c14-workload-certs\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-kmqsw\" (UID: \"28aada86-170c-4935-80ed-e8c5ee503c14\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-kmqsw"
Nov 03 20:21:42.596676 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:42.596528    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28aada86-170c-4935-80ed-e8c5ee503c14-istio-podinfo\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-kmqsw\" (UID: \"28aada86-170c-4935-80ed-e8c5ee503c14\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-kmqsw"
Nov 03 20:21:42.596676 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:42.596594    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28aada86-170c-4935-80ed-e8c5ee503c14-istio-data\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-kmqsw\" (UID: \"28aada86-170c-4935-80ed-e8c5ee503c14\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-kmqsw"
Nov 03 20:21:42.596676 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:42.596627    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28aada86-170c-4935-80ed-e8c5ee503c14-workload-socket\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-kmqsw\" (UID: \"28aada86-170c-4935-80ed-e8c5ee503c14\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-kmqsw"
Nov 03 20:21:42.597073 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:42.597048    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28aada86-170c-4935-80ed-e8c5ee503c14-workload-certs\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-kmqsw\" (UID: \"28aada86-170c-4935-80ed-e8c5ee503c14\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-kmqsw"
Nov 03 20:21:42.597073 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:42.597060    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28aada86-170c-4935-80ed-e8c5ee503c14-workload-socket\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-kmqsw\" (UID: \"28aada86-170c-4935-80ed-e8c5ee503c14\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-kmqsw"
Nov 03 20:21:42.597258 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:42.597197    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28aada86-170c-4935-80ed-e8c5ee503c14-credential-socket\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-kmqsw\" (UID: \"28aada86-170c-4935-80ed-e8c5ee503c14\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-kmqsw"
Nov 03 20:21:42.597369 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:42.597351    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28aada86-170c-4935-80ed-e8c5ee503c14-istiod-ca-cert\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-kmqsw\" (UID: \"28aada86-170c-4935-80ed-e8c5ee503c14\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-kmqsw"
Nov 03 20:21:42.598918 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:42.598894    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28aada86-170c-4935-80ed-e8c5ee503c14-istio-envoy\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-kmqsw\" (UID: \"28aada86-170c-4935-80ed-e8c5ee503c14\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-kmqsw"
Nov 03 20:21:42.599321 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:42.599295    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28aada86-170c-4935-80ed-e8c5ee503c14-istio-podinfo\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-kmqsw\" (UID: \"28aada86-170c-4935-80ed-e8c5ee503c14\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-kmqsw"
Nov 03 20:21:42.599470 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:42.599456    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aafe6908-shqjq"
Nov 03 20:21:42.621509 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:42.621481    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28aada86-170c-4935-80ed-e8c5ee503c14-istio-token\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-kmqsw\" (UID: \"28aada86-170c-4935-80ed-e8c5ee503c14\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-kmqsw"
Nov 03 20:21:42.621878 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:42.621505    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8n6x7\" (UniqueName: \"kubernetes.io/projected/28aada86-170c-4935-80ed-e8c5ee503c14-kube-api-access-8n6x7\") pod \"e2e-worker-w2-9973d9-openshift-default-ffb6475-kmqsw\" (UID: \"28aada86-170c-4935-80ed-e8c5ee503c14\") " pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-kmqsw"
Nov 03 20:21:42.662356 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:42.662319    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-kmqsw"
Nov 03 20:21:43.397802 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:43.351254    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aafe6908-shqjq"]
Nov 03 20:21:43.898330 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:43.854874    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/simulator-w0-e40816-kserve-688f79796f-2tvw5" podUID="c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:43.898330 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:43.855651    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/distinct-2-w3-482722-kserve-74bf58cf7d-v9cjw" podUID="9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:43.898330 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:43.855872    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/premium-w4-e80203-kserve-5f9f445456-bms9d" podUID="6ad76efc-20c2-4951-b22a-c5a39cd39ed4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:43.898330 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:43.870912    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-2-w6-9e1bcb-kserve-7d96d9f6f6-4f7fl" podUID="56353b0f-08ee-4ada-9dc0-ac5afd5a59c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:44.186947 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:44.186900    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh" podUID="2d3647f2-89e9-4908-a8ee-d84e527abeb0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Nov 03 20:21:44.598804 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:44.581747    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/distinct-w3-482722-kserve-d876967c-kd488" podUID="9a97f9cc-2261-471d-9053-51ee965850bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:44.691097 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:44.690957    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/distinct-2-w4-e80203-kserve-756f9b9bc6-69gxj" podUID="ccf5dc2f-f116-4917-bb1a-b136d8ed17f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded"
Nov 03 20:21:44.775839 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:44.748324    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/distinct-2-w0-e40816-kserve-79fb46654d-s242x" podUID="4ad9b048-925c-4d01-a839-a5f25d4dd02d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:44.775839 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:44.768175    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/premium-w1-6fc748-kserve-7474db9cd7-nf2xk" podUID="830e062e-83fc-4de7-91a7-5534dac1d46b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:44.820914 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:44.820866    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/premium-w2-9973d9-kserve-5c6dd4669c-5gv8x" podUID="cefc89ad-7ae1-433b-979d-6601678a6d98" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:44.882147 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:44.857543    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/premium-w4-e80203-kserve-5f9f445456-bms9d" podUID="6ad76efc-20c2-4951-b22a-c5a39cd39ed4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:44.897800 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:44.883073    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/simulator-w0-e40816-kserve-688f79796f-2tvw5" podUID="c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Nov 03 20:21:44.897800 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:44.884234    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-2-w6-9e1bcb-kserve-7d96d9f6f6-4f7fl" podUID="56353b0f-08ee-4ada-9dc0-ac5afd5a59c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:44.897800 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:44.884299    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/premium-w3-482722-kserve-689dc7bd5d-x8rjp" podUID="369045c3-4d6a-49a7-9bc6-f2939883bee8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Nov 03 20:21:44.897800 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:44.885115    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/distinct-2-w3-482722-kserve-74bf58cf7d-v9cjw" podUID="9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded"
Nov 03 20:21:44.923832 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:44.912031    5570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-9973d9/premium-w2-9973d9-kserve-5c6dd4669c-5gv8x" podUID="cefc89ad-7ae1-433b-979d-6601678a6d98" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:44.923832 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:44.918233    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-e80203/premium-w4-e80203-kserve-5f9f445456-bms9d"
Nov 03 20:21:44.923832 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:44.918555    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e40816/simulator-w0-e40816-kserve-688f79796f-2tvw5"
Nov 03 20:21:44.923832 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:44.918814    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-2-w6-9e1bcb-kserve-7d96d9f6f6-4f7fl"
Nov 03 20:21:44.923832 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:44.918980    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-482722/distinct-2-w3-482722-kserve-74bf58cf7d-v9cjw"
Nov 03 20:21:44.947027 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:44.938891    5570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-482722/distinct-w3-482722-kserve-d876967c-kd488" podUID="9a97f9cc-2261-471d-9053-51ee965850bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:45.170822 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:45.159828    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-kmqsw"]
Nov 03 20:21:45.170822 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:45.166105    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:21:45.170822 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:45.166199    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:21:45.170822 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:45.166246    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:21:45.262534 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:45.229871    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh" podUID="2d3647f2-89e9-4908-a8ee-d84e527abeb0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Nov 03 20:21:45.262534 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:45.237498    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-9e1bcb/premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh"
Nov 03 20:21:45.522830 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:45.497318    5570 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz" podUID="3a715ef1-38e9-4386-9eec-e10e8df11a10" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:45.673807 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:45.671111    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz" podUID="3a715ef1-38e9-4386-9eec-e10e8df11a10" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": context deadline exceeded"
Nov 03 20:21:45.673807 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:45.673298    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/distinct-w3-482722-kserve-d876967c-kd488" podUID="9a97f9cc-2261-471d-9053-51ee965850bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:45.727877 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:45.691660    5570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-482722/premium-w3-482722-kserve-689dc7bd5d-x8rjp" podUID="369045c3-4d6a-49a7-9bc6-f2939883bee8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": context deadline exceeded"
Nov 03 20:21:45.727877 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:45.692341    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-482722/distinct-w3-482722-kserve-d876967c-kd488"
Nov 03 20:21:45.868799 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:45.767259    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/distinct-2-w4-e80203-kserve-756f9b9bc6-69gxj" podUID="ccf5dc2f-f116-4917-bb1a-b136d8ed17f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:46.003872 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:45.798510    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/premium-w1-6fc748-kserve-7474db9cd7-nf2xk" podUID="830e062e-83fc-4de7-91a7-5534dac1d46b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:46.056934 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:45.806135    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/distinct-2-w0-e40816-kserve-79fb46654d-s242x" podUID="4ad9b048-925c-4d01-a839-a5f25d4dd02d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:46.146876 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:45.869542    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/premium-w4-e80203-kserve-5f9f445456-bms9d" podUID="6ad76efc-20c2-4951-b22a-c5a39cd39ed4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Nov 03 20:21:46.146876 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:46.024220    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/premium-w2-9973d9-kserve-5c6dd4669c-5gv8x" podUID="cefc89ad-7ae1-433b-979d-6601678a6d98" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Nov 03 20:21:46.153179 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:46.052160    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-e80203/distinct-2-w4-e80203-kserve-756f9b9bc6-69gxj"
Nov 03 20:21:46.241901 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:46.054091    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/distinct-2-w3-482722-kserve-74bf58cf7d-v9cjw" podUID="9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:46.241901 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:46.056164    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-2-w6-9e1bcb-kserve-7d96d9f6f6-4f7fl" podUID="56353b0f-08ee-4ada-9dc0-ac5afd5a59c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:46.345267 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:46.345211    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh" podUID="2d3647f2-89e9-4908-a8ee-d84e527abeb0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Nov 03 20:21:46.406803 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:46.393859    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/premium-w3-482722-kserve-689dc7bd5d-x8rjp" podUID="369045c3-4d6a-49a7-9bc6-f2939883bee8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:46.408449 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:46.405939    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/simulator-w0-e40816-kserve-688f79796f-2tvw5" podUID="c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Nov 03 20:21:46.702842 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:46.689748    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/distinct-w3-482722-kserve-d876967c-kd488" podUID="9a97f9cc-2261-471d-9053-51ee965850bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded"
Nov 03 20:21:46.715451 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:46.715397    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz" podUID="3a715ef1-38e9-4386-9eec-e10e8df11a10" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:46.910868 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:46.903464    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/distinct-2-w4-e80203-kserve-756f9b9bc6-69gxj" podUID="ccf5dc2f-f116-4917-bb1a-b136d8ed17f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded"
Nov 03 20:21:47.062173 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:47.062127    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/distinct-2-w0-e40816-kserve-79fb46654d-s242x" podUID="4ad9b048-925c-4d01-a839-a5f25d4dd02d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded"
Nov 03 20:21:47.064980 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:47.064680    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/premium-w1-6fc748-kserve-7474db9cd7-nf2xk" podUID="830e062e-83fc-4de7-91a7-5534dac1d46b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Nov 03 20:21:47.156060 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:47.156019    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6fc748/premium-w1-6fc748-kserve-7474db9cd7-nf2xk"
Nov 03 20:21:47.265795 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:47.240311    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/premium-w4-e80203-kserve-5f9f445456-bms9d" podUID="6ad76efc-20c2-4951-b22a-c5a39cd39ed4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:47.794590 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:47.247348    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/premium-w2-9973d9-kserve-5c6dd4669c-5gv8x" podUID="cefc89ad-7ae1-433b-979d-6601678a6d98" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Nov 03 20:21:47.794590 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:47.293883    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e40816/distinct-2-w0-e40816-kserve-79fb46654d-s242x"
Nov 03 20:21:47.904895 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:47.904847    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-kmqsw" event={"ID":"28aada86-170c-4935-80ed-e8c5ee503c14","Type":"ContainerStarted","Data":"5969452b5e187cb57129ed9a6482c4a56325d5ef9259c66fbae67df3976f53e5"}
Nov 03 20:21:47.905175 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:47.905147    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aafe6908-shqjq" event={"ID":"1aec61be-2b12-40cb-ada3-88c05f790cfb","Type":"ContainerStarted","Data":"826700c5a0e302d779ffe0d88475c81cf7690a22aadcc3679b996da2279ae71a"}
Nov 03 20:21:47.914051 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:47.914012    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9973d9/premium-w2-9973d9-kserve-5c6dd4669c-5gv8x"
Nov 03 20:21:47.914324 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:47.914307    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-482722/premium-w3-482722-kserve-689dc7bd5d-x8rjp"
Nov 03 20:21:47.914468 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:47.914455    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz"
Nov 03 20:21:48.266037 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:48.257887    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/distinct-2-w3-482722-kserve-74bf58cf7d-v9cjw" podUID="9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:48.266037 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:48.258298    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-2-w6-9e1bcb-kserve-7d96d9f6f6-4f7fl" podUID="56353b0f-08ee-4ada-9dc0-ac5afd5a59c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded"
Nov 03 20:21:48.266037 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:48.258609    5570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-2-w6-9e1bcb-kserve-7d96d9f6f6-4f7fl" podUID="56353b0f-08ee-4ada-9dc0-ac5afd5a59c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": context deadline exceeded"
Nov 03 20:21:48.309814 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:48.305199    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh" podUID="2d3647f2-89e9-4908-a8ee-d84e527abeb0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Nov 03 20:21:48.309814 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:48.305618    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/premium-w3-482722-kserve-689dc7bd5d-x8rjp" podUID="369045c3-4d6a-49a7-9bc6-f2939883bee8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Nov 03 20:21:48.309814 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:48.305959    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/simulator-w0-e40816-kserve-688f79796f-2tvw5" podUID="c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Nov 03 20:21:48.333051 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:48.321405    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/distinct-w3-482722-kserve-d876967c-kd488" podUID="9a97f9cc-2261-471d-9053-51ee965850bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded"
Nov 03 20:21:48.333051 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:48.327131    5570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-9e1bcb/premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh" podUID="2d3647f2-89e9-4908-a8ee-d84e527abeb0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": context deadline exceeded"
Nov 03 20:21:48.333051 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:48.327213    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz" podUID="3a715ef1-38e9-4386-9eec-e10e8df11a10" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": context deadline exceeded"
Nov 03 20:21:48.333051 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:48.327537    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/distinct-2-w4-e80203-kserve-756f9b9bc6-69gxj" podUID="ccf5dc2f-f116-4917-bb1a-b136d8ed17f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded"
Nov 03 20:21:48.333051 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:48.327842    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/distinct-2-w0-e40816-kserve-79fb46654d-s242x" podUID="4ad9b048-925c-4d01-a839-a5f25d4dd02d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded"
Nov 03 20:21:48.333051 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:48.328135    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/premium-w1-6fc748-kserve-7474db9cd7-nf2xk" podUID="830e062e-83fc-4de7-91a7-5534dac1d46b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Nov 03 20:21:48.403805 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:48.401849    5570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-e40816/simulator-w0-e40816-kserve-688f79796f-2tvw5" podUID="c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:48.504922 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:48.451912    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/premium-w2-9973d9-kserve-5c6dd4669c-5gv8x" podUID="cefc89ad-7ae1-433b-979d-6601678a6d98" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:48.549112 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:48.548253    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/premium-w4-e80203-kserve-5f9f445456-bms9d" podUID="6ad76efc-20c2-4951-b22a-c5a39cd39ed4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:49.155810 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:49.148321    5570 generic.go:356] "Generic (PLEG): container finished" podID="1aec61be-2b12-40cb-ada3-88c05f790cfb" containerID="aa7c85a41a5902ad05b98dc85d3ca3f9fea44ffd1a1413cd179f7c3edbece58e" exitCode=0
Nov 03 20:21:49.230802 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:49.150525    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aafe6908-shqjq" event={"ID":"1aec61be-2b12-40cb-ada3-88c05f790cfb","Type":"ContainerDied","Data":"aa7c85a41a5902ad05b98dc85d3ca3f9fea44ffd1a1413cd179f7c3edbece58e"}
Nov 03 20:21:49.297015 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:49.296965    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-kmqsw" event={"ID":"28aada86-170c-4935-80ed-e8c5ee503c14","Type":"ContainerStarted","Data":"f8dd680f3a7881f299314139130aacee319f496bfe291e09b3d07a6d33658515"}
Nov 03 20:21:49.315822 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:49.312099    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/distinct-2-w3-482722-kserve-74bf58cf7d-v9cjw" podUID="9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded"
Nov 03 20:21:49.333419 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:49.333350    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/distinct-w3-482722-kserve-d876967c-kd488" podUID="9a97f9cc-2261-471d-9053-51ee965850bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:49.333890 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:49.333839    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/premium-w1-6fc748-kserve-7474db9cd7-nf2xk" podUID="830e062e-83fc-4de7-91a7-5534dac1d46b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:49.364798 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:49.363042    5570 patch_prober.go:29] interesting pod/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-z4kzq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:21:49.364798 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:49.363104    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-z4kzq" podUID="e185c296-cd84-4b0c-9316-ef10085cda0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:49.364798 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:49.316217    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/simulator-w0-e40816-kserve-688f79796f-2tvw5" podUID="c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:49.364798 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:49.363580    5570 patch_prober.go:29] interesting pod/e2e-worker-w1-6fc748-openshift-default-665546959f-jbtqd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:21:49.364798 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:49.363624    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-jbtqd" podUID="baacee7f-b912-4964-9855-be2beb0094cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:49.364798 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:49.316105    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-2-w6-9e1bcb-kserve-7d96d9f6f6-4f7fl" podUID="56353b0f-08ee-4ada-9dc0-ac5afd5a59c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded"
Nov 03 20:21:49.372798 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:49.316171    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh" podUID="2d3647f2-89e9-4908-a8ee-d84e527abeb0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Nov 03 20:21:49.372798 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:49.367325    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/distinct-2-w0-e40816-kserve-79fb46654d-s242x" podUID="4ad9b048-925c-4d01-a839-a5f25d4dd02d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded"
Nov 03 20:21:49.372798 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:49.367760    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz" podUID="3a715ef1-38e9-4386-9eec-e10e8df11a10" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": context deadline exceeded"
Nov 03 20:21:49.372798 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:49.368097    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/distinct-2-w4-e80203-kserve-756f9b9bc6-69gxj" podUID="ccf5dc2f-f116-4917-bb1a-b136d8ed17f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded"
Nov 03 20:21:49.372798 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:49.316193    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/premium-w3-482722-kserve-689dc7bd5d-x8rjp" podUID="369045c3-4d6a-49a7-9bc6-f2939883bee8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Nov 03 20:21:49.530687 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:49.528365    5570 patch_prober.go:29] interesting pod/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-b4zmq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:21:49.538907 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:49.538858    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-b4zmq" podUID="8f7ff673-520b-4dd1-a76d-0e7aa1053c3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:49.557625 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:49.557576    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/premium-w4-e80203-kserve-5f9f445456-bms9d" podUID="6ad76efc-20c2-4951-b22a-c5a39cd39ed4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:49.558248 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:49.558215    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/premium-w2-9973d9-kserve-5c6dd4669c-5gv8x" podUID="cefc89ad-7ae1-433b-979d-6601678a6d98" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Nov 03 20:21:49.659209 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:49.659138    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-kmqsw" podStartSLOduration=8.659116139 podStartE2EDuration="8.659116139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:21:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:21:49.391859165 +0000 UTC m=+1043.544521054" watchObservedRunningTime="2026-11-03 20:21:49.659116139 +0000 UTC m=+1043.811778051"
Nov 03 20:21:49.681055 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:49.681009    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-kmqsw"
Nov 03 20:21:49.855660 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:49.855566    5570 patch_prober.go:29] interesting pod/e2e-worker-w2-9973d9-openshift-default-ffb6475-kmqsw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Nov 03 20:21:49.855968 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:49.855935    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-kmqsw" podUID="28aada86-170c-4935-80ed-e8c5ee503c14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Nov 03 20:21:49.915213 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:49.915175    5570 patch_prober.go:29] interesting pod/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sf2lg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:21:49.915510 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:49.915476    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sf2lg" podUID="07093a96-d60c-490f-910b-734f41752c17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:50.190864 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:50.166285    5570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-e80203/distinct-2-w4-e80203-kserve-756f9b9bc6-69gxj" podUID="ccf5dc2f-f116-4917-bb1a-b136d8ed17f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:50.275331 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:50.275294    5570 patch_prober.go:29] interesting pod/e2e-worker-w1-6fc748-openshift-default-665546959f-2zvr4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:21:50.275643 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:50.275596    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-2zvr4" podUID="c8edff27-9b0c-40e5-9a72-4a85d7efd3e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:50.299851 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:50.295954    5570 patch_prober.go:29] interesting pod/e2e-worker-w1-6fc748-openshift-default-665546959f-9gvtf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:21:50.299851 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:50.296020    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-9gvtf" podUID="5e7f17ab-0f1d-4a59-b415-fe844ae7c1e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:50.342477 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:50.342432    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/distinct-w3-482722-kserve-d876967c-kd488" podUID="9a97f9cc-2261-471d-9053-51ee965850bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:50.343441 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:50.343406    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/distinct-2-w3-482722-kserve-74bf58cf7d-v9cjw" podUID="9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:50.350803 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:50.344675    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/premium-w1-6fc748-kserve-7474db9cd7-nf2xk" podUID="830e062e-83fc-4de7-91a7-5534dac1d46b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Nov 03 20:21:50.364306 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:50.364260    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/simulator-w0-e40816-kserve-688f79796f-2tvw5" podUID="c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:50.365754 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:50.365719    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-2-w6-9e1bcb-kserve-7d96d9f6f6-4f7fl" podUID="56353b0f-08ee-4ada-9dc0-ac5afd5a59c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:50.372833 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:50.366103    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh" podUID="2d3647f2-89e9-4908-a8ee-d84e527abeb0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:50.377832 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:50.367580    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/distinct-2-w0-e40816-kserve-79fb46654d-s242x" podUID="4ad9b048-925c-4d01-a839-a5f25d4dd02d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:50.396619 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:50.372132    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/distinct-2-w4-e80203-kserve-756f9b9bc6-69gxj" podUID="ccf5dc2f-f116-4917-bb1a-b136d8ed17f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded"
Nov 03 20:21:50.452697 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:50.372160    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz" podUID="3a715ef1-38e9-4386-9eec-e10e8df11a10" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:50.452697 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:50.372193    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/premium-w3-482722-kserve-689dc7bd5d-x8rjp" podUID="369045c3-4d6a-49a7-9bc6-f2939883bee8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:50.453563 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:50.453532    5570 patch_prober.go:29] interesting pod/e2e-worker-w0-e40816-openshift-default-744c57967c-5bn7t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:21:50.453797 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:50.453751    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-5bn7t" podUID="26831c71-2091-4090-843d-f7268d257114" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:50.501807 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:50.501256    5570 patch_prober.go:29] interesting pod/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-wkmg8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:21:50.501807 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:50.501321    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-wkmg8" podUID="e314286e-4b2d-4fae-bc59-528809a8dbce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:50.704979 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:50.704860    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/premium-w4-e80203-kserve-5f9f445456-bms9d" podUID="6ad76efc-20c2-4951-b22a-c5a39cd39ed4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Nov 03 20:21:50.869870 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:50.706907    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/premium-w2-9973d9-kserve-5c6dd4669c-5gv8x" podUID="cefc89ad-7ae1-433b-979d-6601678a6d98" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:50.881490 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:50.881448    5570 patch_prober.go:29] interesting pod/e2e-worker-w2-9973d9-openshift-default-ffb6475-kmqsw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Nov 03 20:21:50.973868 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:50.936912    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-kmqsw" podUID="28aada86-170c-4935-80ed-e8c5ee503c14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Nov 03 20:21:50.973868 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:50.936575    5570 patch_prober.go:29] interesting pod/e2e-worker-w3-482722-openshift-default-7645487df-c9kvc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:21:50.973868 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:50.937449    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-c9kvc" podUID="1351662d-6ca4-430b-8bbe-0ca1429df833" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:50.973868 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:50.936845    5570 patch_prober.go:29] interesting pod/e2e-worker-w3-482722-openshift-default-7645487df-fqjnh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:21:50.973868 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:50.937530    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-fqjnh" podUID="51c1be09-aa2e-47bd-a863-97f7daf94ddb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:51.290874 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:51.284565    5570 patch_prober.go:29] interesting pod/e2e-worker-w0-e40816-openshift-default-744c57967c-xt4z5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:21:51.290874 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:51.284653    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-xt4z5" podUID="e026bfa1-f899-46e8-9309-be16eea226bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:51.473799 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:51.449378    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/premium-w3-482722-kserve-689dc7bd5d-x8rjp" podUID="369045c3-4d6a-49a7-9bc6-f2939883bee8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:51.473799 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:51.449555    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz" podUID="3a715ef1-38e9-4386-9eec-e10e8df11a10" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": context deadline exceeded"
Nov 03 20:21:51.473799 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:51.449927    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-2-w6-9e1bcb-kserve-7d96d9f6f6-4f7fl" podUID="56353b0f-08ee-4ada-9dc0-ac5afd5a59c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded"
Nov 03 20:21:51.473799 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:51.450396    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/distinct-2-w4-e80203-kserve-756f9b9bc6-69gxj" podUID="ccf5dc2f-f116-4917-bb1a-b136d8ed17f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:51.473799 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:51.452362    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/distinct-2-w0-e40816-kserve-79fb46654d-s242x" podUID="4ad9b048-925c-4d01-a839-a5f25d4dd02d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded"
Nov 03 20:21:51.473799 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:51.452600    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/distinct-w3-482722-kserve-d876967c-kd488" podUID="9a97f9cc-2261-471d-9053-51ee965850bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded"
Nov 03 20:21:51.473799 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:51.456123    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/distinct-2-w3-482722-kserve-74bf58cf7d-v9cjw" podUID="9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded"
Nov 03 20:21:51.473799 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:51.459935    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh" podUID="2d3647f2-89e9-4908-a8ee-d84e527abeb0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Nov 03 20:21:51.473799 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:51.462198    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/premium-w1-6fc748-kserve-7474db9cd7-nf2xk" podUID="830e062e-83fc-4de7-91a7-5534dac1d46b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Nov 03 20:21:51.473799 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:51.462411    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/simulator-w0-e40816-kserve-688f79796f-2tvw5" podUID="c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Nov 03 20:21:51.473799 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:51.472939    5570 patch_prober.go:29] interesting pod/e2e-worker-w2-9973d9-openshift-default-ffb6475-qskf7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:21:51.473799 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:51.473023    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-qskf7" podUID="469fc281-f7ee-4584-b3b6-85c6a669f806" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:51.611859 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:51.601393    5570 patch_prober.go:29] interesting pod/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmkq7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:21:51.611859 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:51.601487    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmkq7" podUID="157f80a0-a704-4bed-b544-205c3f2a52b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:51.624804 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:51.620885    5570 patch_prober.go:29] interesting pod/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-vzfw9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:21:51.697800 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:51.658149    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-vzfw9" podUID="a739707b-9c5a-4c9b-90df-501c5dd0c98d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:51.698383 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:51.676147    5570 patch_prober.go:29] interesting pod/e2e-worker-w2-9973d9-openshift-default-ffb6475-kmqsw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Nov 03 20:21:51.698601 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:51.698568    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-kmqsw" podUID="28aada86-170c-4935-80ed-e8c5ee503c14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Nov 03 20:21:51.743803 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:51.742991    5570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-482722/distinct-2-w3-482722-kserve-74bf58cf7d-v9cjw" podUID="9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:51.862035 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:51.861904    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/premium-w4-e80203-kserve-5f9f445456-bms9d" podUID="6ad76efc-20c2-4951-b22a-c5a39cd39ed4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Nov 03 20:21:51.878576 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:51.876952    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/premium-w2-9973d9-kserve-5c6dd4669c-5gv8x" podUID="cefc89ad-7ae1-433b-979d-6601678a6d98" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:52.455922 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:52.455862    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/distinct-w3-482722-kserve-d876967c-kd488" podUID="9a97f9cc-2261-471d-9053-51ee965850bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:52.459799 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:52.456283    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/distinct-2-w3-482722-kserve-74bf58cf7d-v9cjw" podUID="9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:52.459799 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:52.456575    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/distinct-2-w0-e40816-kserve-79fb46654d-s242x" podUID="4ad9b048-925c-4d01-a839-a5f25d4dd02d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:52.487802 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:52.460571    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz" podUID="3a715ef1-38e9-4386-9eec-e10e8df11a10" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:52.491793 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:52.488536    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/distinct-2-w4-e80203-kserve-756f9b9bc6-69gxj" podUID="ccf5dc2f-f116-4917-bb1a-b136d8ed17f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:52.491793 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:52.488967    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/premium-w3-482722-kserve-689dc7bd5d-x8rjp" podUID="369045c3-4d6a-49a7-9bc6-f2939883bee8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Nov 03 20:21:52.491793 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:52.489242    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-2-w6-9e1bcb-kserve-7d96d9f6f6-4f7fl" podUID="56353b0f-08ee-4ada-9dc0-ac5afd5a59c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded"
Nov 03 20:21:52.492796 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:52.492184    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh" podUID="2d3647f2-89e9-4908-a8ee-d84e527abeb0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Nov 03 20:21:52.492796 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:52.492484    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/premium-w1-6fc748-kserve-7474db9cd7-nf2xk" podUID="830e062e-83fc-4de7-91a7-5534dac1d46b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Nov 03 20:21:52.492796 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:52.492725    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/simulator-w0-e40816-kserve-688f79796f-2tvw5" podUID="c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Nov 03 20:21:52.534979 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:52.534921    5570 patch_prober.go:29] interesting pod/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-7458z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:21:52.535353 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:52.535321    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-7458z" podUID="3c3d9f2d-dcbb-4fb3-a54f-ed9cdef4118e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:52.631112 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:52.630860    5570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-e40816/distinct-2-w0-e40816-kserve-79fb46654d-s242x" podUID="4ad9b048-925c-4d01-a839-a5f25d4dd02d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:52.666662 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:52.666166    5570 patch_prober.go:29] interesting pod/e2e-worker-w2-9973d9-openshift-default-ffb6475-kmqsw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Nov 03 20:21:52.666662 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:52.666232    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-kmqsw" podUID="28aada86-170c-4935-80ed-e8c5ee503c14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Nov 03 20:21:52.864334 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:52.864201    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/premium-w4-e80203-kserve-5f9f445456-bms9d" podUID="6ad76efc-20c2-4951-b22a-c5a39cd39ed4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:52.895550 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:52.895500    5570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-6fc748/premium-w1-6fc748-kserve-7474db9cd7-nf2xk" podUID="830e062e-83fc-4de7-91a7-5534dac1d46b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:52.895903 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:52.895852    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/premium-w2-9973d9-kserve-5c6dd4669c-5gv8x" podUID="cefc89ad-7ae1-433b-979d-6601678a6d98" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Nov 03 20:21:52.937104 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:52.937065    5570 patch_prober.go:29] interesting pod/e2e-worker-w1-6fc748-openshift-default-665546959f-wg864 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:21:52.937429 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:52.937382    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-wg864" podUID="6a1b3d29-b647-46e1-9005-e540e937f85f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:53.373207 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:53.373160    5570 patch_prober.go:29] interesting pod/e2e-worker-w3-482722-openshift-default-7645487df-5c82t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:21:53.407442 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:53.407057    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-5c82t" podUID="6029825e-e953-4c43-9f4d-d19b5ca38623" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:53.540553 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:53.540494    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/distinct-2-w4-e80203-kserve-756f9b9bc6-69gxj" podUID="ccf5dc2f-f116-4917-bb1a-b136d8ed17f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:53.541812 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:53.541094    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/premium-w3-482722-kserve-689dc7bd5d-x8rjp" podUID="369045c3-4d6a-49a7-9bc6-f2939883bee8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Nov 03 20:21:53.567224 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:53.566919    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-2-w6-9e1bcb-kserve-7d96d9f6f6-4f7fl" podUID="56353b0f-08ee-4ada-9dc0-ac5afd5a59c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:53.567987 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:53.567953    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz" podUID="3a715ef1-38e9-4386-9eec-e10e8df11a10" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:53.687769 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:53.567572    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/distinct-w3-482722-kserve-d876967c-kd488" podUID="9a97f9cc-2261-471d-9053-51ee965850bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:53.710969 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:53.609633    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/premium-w1-6fc748-kserve-7474db9cd7-nf2xk" podUID="830e062e-83fc-4de7-91a7-5534dac1d46b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Nov 03 20:21:53.711958 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:53.609665    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/simulator-w0-e40816-kserve-688f79796f-2tvw5" podUID="c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:53.712439 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:53.609477    5570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-e80203/premium-w4-e80203-kserve-5f9f445456-bms9d" podUID="6ad76efc-20c2-4951-b22a-c5a39cd39ed4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:53.712948 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:53.584862    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh" podUID="2d3647f2-89e9-4908-a8ee-d84e527abeb0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:53.713329 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:53.624430    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/distinct-2-w3-482722-kserve-74bf58cf7d-v9cjw" podUID="9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded"
Nov 03 20:21:53.774833 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:53.624473    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/distinct-2-w0-e40816-kserve-79fb46654d-s242x" podUID="4ad9b048-925c-4d01-a839-a5f25d4dd02d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded"
Nov 03 20:21:53.869645 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:53.869594    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/premium-w4-e80203-kserve-5f9f445456-bms9d" podUID="6ad76efc-20c2-4951-b22a-c5a39cd39ed4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:54.023900 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:54.006716    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/premium-w2-9973d9-kserve-5c6dd4669c-5gv8x" podUID="cefc89ad-7ae1-433b-979d-6601678a6d98" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:54.284092 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:54.236673    5570 patch_prober.go:29] interesting pod/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmqs6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:21:54.284092 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:54.236753    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmqs6" podUID="516f80b2-26e4-44d2-b97a-6e740d0b53e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:54.535446 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:54.524128    5570 patch_prober.go:29] interesting pod/e2e-worker-w2-9973d9-openshift-default-ffb6475-kmqsw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Nov 03 20:21:54.539907 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:54.535752    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-kmqsw" podUID="28aada86-170c-4935-80ed-e8c5ee503c14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Nov 03 20:21:54.686898 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:54.616147    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/distinct-2-w4-e80203-kserve-756f9b9bc6-69gxj" podUID="ccf5dc2f-f116-4917-bb1a-b136d8ed17f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded"
Nov 03 20:21:54.686898 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:54.644416    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/premium-w3-482722-kserve-689dc7bd5d-x8rjp" podUID="369045c3-4d6a-49a7-9bc6-f2939883bee8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:54.688674 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:54.688626    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/distinct-w3-482722-kserve-d876967c-kd488" podUID="9a97f9cc-2261-471d-9053-51ee965850bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:54.708849 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:54.704312    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz" podUID="3a715ef1-38e9-4386-9eec-e10e8df11a10" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": context deadline exceeded"
Nov 03 20:21:54.774798 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:54.772094    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/premium-w1-6fc748-kserve-7474db9cd7-nf2xk" podUID="830e062e-83fc-4de7-91a7-5534dac1d46b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Nov 03 20:21:54.784602 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:54.778915    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/simulator-w0-e40816-kserve-688f79796f-2tvw5" podUID="c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Nov 03 20:21:54.799694 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:54.798911    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh" podUID="2d3647f2-89e9-4908-a8ee-d84e527abeb0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Nov 03 20:21:54.830915 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:54.804299    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/distinct-2-w3-482722-kserve-74bf58cf7d-v9cjw" podUID="9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded"
Nov 03 20:21:54.907837 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:54.902200    5570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-9973d9/premium-w2-9973d9-kserve-5c6dd4669c-5gv8x" podUID="cefc89ad-7ae1-433b-979d-6601678a6d98" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:54.910100 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:54.908158    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/distinct-2-w0-e40816-kserve-79fb46654d-s242x" podUID="4ad9b048-925c-4d01-a839-a5f25d4dd02d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded"
Nov 03 20:21:54.924744 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:54.923831    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-2-w6-9e1bcb-kserve-7d96d9f6f6-4f7fl" podUID="56353b0f-08ee-4ada-9dc0-ac5afd5a59c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded"
Nov 03 20:21:54.936815 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:54.926684    5570 patch_prober.go:29] interesting pod/e2e-worker-w2-9973d9-openshift-default-ffb6475-kmqsw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Nov 03 20:21:54.937151 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:54.937112    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-kmqsw" podUID="28aada86-170c-4935-80ed-e8c5ee503c14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Nov 03 20:21:54.995800 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:54.971824    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/premium-w4-e80203-kserve-5f9f445456-bms9d" podUID="6ad76efc-20c2-4951-b22a-c5a39cd39ed4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Nov 03 20:21:55.100869 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:55.100758    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aafe6908-shqjq"
Nov 03 20:21:55.253798 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:55.253321    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/premium-w2-9973d9-kserve-5c6dd4669c-5gv8x" podUID="cefc89ad-7ae1-433b-979d-6601678a6d98" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:55.331797 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:55.329002    5570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-482722/distinct-w3-482722-kserve-d876967c-kd488" podUID="9a97f9cc-2261-471d-9053-51ee965850bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:55.504250 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:21:55.501216    5570 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.511s"
Nov 03 20:21:55.625793 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:55.624066    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1aec61be-2b12-40cb-ada3-88c05f790cfb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1aec61be-2b12-40cb-ada3-88c05f790cfb-maas-api-openshift-ca\") pod \"1aec61be-2b12-40cb-ada3-88c05f790cfb\" (UID: \"1aec61be-2b12-40cb-ada3-88c05f790cfb\") "
Nov 03 20:21:55.625793 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:55.624212    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1aec61be-2b12-40cb-ada3-88c05f790cfb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1aec61be-2b12-40cb-ada3-88c05f790cfb-maas-api-xks-ca\") pod \"1aec61be-2b12-40cb-ada3-88c05f790cfb\" (UID: \"1aec61be-2b12-40cb-ada3-88c05f790cfb\") "
Nov 03 20:21:55.625793 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:55.624270    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1aec61be-2b12-40cb-ada3-88c05f790cfb-kube-api-access-p5mhx\" (UniqueName: \"kubernetes.io/projected/1aec61be-2b12-40cb-ada3-88c05f790cfb-kube-api-access-p5mhx\") pod \"1aec61be-2b12-40cb-ada3-88c05f790cfb\" (UID: \"1aec61be-2b12-40cb-ada3-88c05f790cfb\") "
Nov 03 20:21:55.646607 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:55.646553    5570 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz" podUID="3a715ef1-38e9-4386-9eec-e10e8df11a10" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/health\": context deadline exceeded"
Nov 03 20:21:55.708802 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:55.702240    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/distinct-2-w4-e80203-kserve-756f9b9bc6-69gxj" podUID="ccf5dc2f-f116-4917-bb1a-b136d8ed17f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded"
Nov 03 20:21:55.738647 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:55.738600    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/premium-w3-482722-kserve-689dc7bd5d-x8rjp" podUID="369045c3-4d6a-49a7-9bc6-f2939883bee8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:55.827260 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:55.827130    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1aec61be-2b12-40cb-ada3-88c05f790cfb-maas-api-openshift-ca" pod "1aec61be-2b12-40cb-ada3-88c05f790cfb" (UID: "1aec61be-2b12-40cb-ada3-88c05f790cfb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:21:55.873001 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:55.872957    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/distinct-w3-482722-kserve-d876967c-kd488" podUID="9a97f9cc-2261-471d-9053-51ee965850bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded"
Nov 03 20:21:55.986800 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:55.899964    5570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-482722/premium-w3-482722-kserve-689dc7bd5d-x8rjp" podUID="369045c3-4d6a-49a7-9bc6-f2939883bee8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:55.986800 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:55.955849    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/simulator-w0-e40816-kserve-688f79796f-2tvw5" podUID="c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Nov 03 20:21:55.986800 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:55.977707    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/premium-w4-e80203-kserve-5f9f445456-bms9d" podUID="6ad76efc-20c2-4951-b22a-c5a39cd39ed4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:56.017864 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:55.960242    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh" podUID="2d3647f2-89e9-4908-a8ee-d84e527abeb0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Nov 03 20:21:56.216865 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:56.215985    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/distinct-2-w0-e40816-kserve-79fb46654d-s242x" podUID="4ad9b048-925c-4d01-a839-a5f25d4dd02d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:56.216865 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:56.021947    5570 patch_prober.go:29] interesting pod/e2e-worker-w2-9973d9-openshift-default-ffb6475-nz52s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:21:56.216865 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:56.216162    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-nz52s" podUID="974badc1-41ad-461c-8719-e1f3f836ccbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:56.216865 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:56.023733    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz" podUID="3a715ef1-38e9-4386-9eec-e10e8df11a10" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:56.216865 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:56.021980    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/premium-w1-6fc748-kserve-7474db9cd7-nf2xk" podUID="830e062e-83fc-4de7-91a7-5534dac1d46b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:56.216865 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:56.046348    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1aec61be-2b12-40cb-ada3-88c05f790cfb-maas-api-xks-ca" pod "1aec61be-2b12-40cb-ada3-88c05f790cfb" (UID: "1aec61be-2b12-40cb-ada3-88c05f790cfb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:21:56.216865 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:56.077396    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-2-w6-9e1bcb-kserve-7d96d9f6f6-4f7fl" podUID="56353b0f-08ee-4ada-9dc0-ac5afd5a59c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:56.216865 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:56.130422    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aafe6908-shqjq" event={"ID":"1aec61be-2b12-40cb-ada3-88c05f790cfb","Type":"ContainerDied","Data":"826700c5a0e302d779ffe0d88475c81cf7690a22aadcc3679b996da2279ae71a"}
Nov 03 20:21:56.216865 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:56.216395    5570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="826700c5a0e302d779ffe0d88475c81cf7690a22aadcc3679b996da2279ae71a"
Nov 03 20:21:56.248804 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:56.021863    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/distinct-2-w3-482722-kserve-74bf58cf7d-v9cjw" podUID="9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded"
Nov 03 20:21:56.248804 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:56.228757    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1aec61be-2b12-40cb-ada3-88c05f790cfb-kube-api-access-p5mhx" pod "1aec61be-2b12-40cb-ada3-88c05f790cfb" (UID: "1aec61be-2b12-40cb-ada3-88c05f790cfb"). InnerVolumeSpecName "kube-api-access-p5mhx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:21:56.248804 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:56.148381    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aafe6908-shqjq"
Nov 03 20:21:56.248804 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:56.242908    5570 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1aec61be-2b12-40cb-ada3-88c05f790cfb-maas-api-openshift-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:21:56.248804 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:56.242935    5570 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1aec61be-2b12-40cb-ada3-88c05f790cfb-maas-api-xks-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:21:56.248804 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:56.242951    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5mhx\" (UniqueName: \"kubernetes.io/projected/1aec61be-2b12-40cb-ada3-88c05f790cfb-kube-api-access-p5mhx\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:21:56.248804 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:56.243348    5570 patch_prober.go:29] interesting pod/e2e-worker-w2-9973d9-openshift-default-ffb6475-kmqsw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Nov 03 20:21:56.248804 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:56.243393    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-kmqsw" podUID="28aada86-170c-4935-80ed-e8c5ee503c14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Nov 03 20:21:56.296896 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:56.272865    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/premium-w2-9973d9-kserve-5c6dd4669c-5gv8x" podUID="cefc89ad-7ae1-433b-979d-6601678a6d98" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:56.552178 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:56.552077    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6fc748/premium-w1-6fc748-kserve-7474db9cd7-nf2xk"
Nov 03 20:21:56.619669 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:56.618181    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-9e1bcb/premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh"
Nov 03 20:21:56.649859 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:56.647880    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-482722/distinct-w3-482722-kserve-d876967c-kd488"
Nov 03 20:21:56.661765 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:56.661739    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-2-w6-9e1bcb-kserve-7d96d9f6f6-4f7fl"
Nov 03 20:21:56.662906 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:56.662886    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-482722/distinct-2-w3-482722-kserve-74bf58cf7d-v9cjw"
Nov 03 20:21:56.740705 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:56.740664    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/distinct-2-w4-e80203-kserve-756f9b9bc6-69gxj" podUID="ccf5dc2f-f116-4917-bb1a-b136d8ed17f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:56.769799 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:56.769332    5570 patch_prober.go:29] interesting pod/e2e-worker-w2-9973d9-openshift-default-ffb6475-kmqsw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Nov 03 20:21:56.769799 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:56.769394    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-kmqsw" podUID="28aada86-170c-4935-80ed-e8c5ee503c14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Nov 03 20:21:56.795155 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:56.779673    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/premium-w3-482722-kserve-689dc7bd5d-x8rjp" podUID="369045c3-4d6a-49a7-9bc6-f2939883bee8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:56.810656 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:56.810575    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-e80203/premium-w4-e80203-kserve-5f9f445456-bms9d"
Nov 03 20:21:56.840811 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:56.837245    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9973d9/premium-w2-9973d9-kserve-5c6dd4669c-5gv8x"
Nov 03 20:21:56.978451 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:56.978419    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz"
Nov 03 20:21:57.002243 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:56.991012    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-e80203/distinct-2-w4-e80203-kserve-756f9b9bc6-69gxj"
Nov 03 20:21:57.002243 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:56.992912    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/simulator-w0-e40816-kserve-688f79796f-2tvw5" podUID="c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:57.008807 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:57.008756    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-482722/premium-w3-482722-kserve-689dc7bd5d-x8rjp"
Nov 03 20:21:57.119066 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:57.118980    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e40816/distinct-2-w0-e40816-kserve-79fb46654d-s242x"
Nov 03 20:21:57.196240 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:57.196205    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e40816/simulator-w0-e40816-kserve-688f79796f-2tvw5"
Nov 03 20:21:57.493157 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:21:57.484924    5570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8bwhz" containerName="extractor"
Nov 03 20:21:57.664809 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:57.663684    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-kmqsw"
Nov 03 20:21:57.674291 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:57.674250    5570 patch_prober.go:29] interesting pod/e2e-worker-w2-9973d9-openshift-default-ffb6475-kmqsw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Nov 03 20:21:57.674506 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:57.674310    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-kmqsw" podUID="28aada86-170c-4935-80ed-e8c5ee503c14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Nov 03 20:21:58.663553 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:58.663512    5570 patch_prober.go:29] interesting pod/e2e-worker-w2-9973d9-openshift-default-ffb6475-kmqsw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Nov 03 20:21:58.664050 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:58.663580    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-kmqsw" podUID="28aada86-170c-4935-80ed-e8c5ee503c14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Nov 03 20:21:58.693829 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:58.693796    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-11a7d765-58f7bcf694-bwl46"]
Nov 03 20:21:58.694519 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:58.694496    5570 cpu_manager.go:414] "containerMap: removing container" podUID="1aec61be-2b12-40cb-ada3-88c05f790cfb" containerName="revoke-keys"
Nov 03 20:21:58.694519 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:58.694512    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1aec61be-2b12-40cb-ada3-88c05f790cfb" containerName="revoke-keys"
Nov 03 20:21:58.694636 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:58.694627    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="1aec61be-2b12-40cb-ada3-88c05f790cfb" containerName="revoke-keys"
Nov 03 20:21:58.698738 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:58.698720    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-11a7d765-58f7bcf694-bwl46"
Nov 03 20:21:58.711424 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:58.711396    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30787f8a-2875-48eb-b4d8-f4b2fa97d29d-plugins-config-volume\") pod \"payload-processing-e2e-ait-11a7d765-58f7bcf694-bwl46\" (UID: \"30787f8a-2875-48eb-b4d8-f4b2fa97d29d\") " pod="openshift-ingress/payload-processing-e2e-ait-11a7d765-58f7bcf694-bwl46"
Nov 03 20:21:58.711625 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:58.711604    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tq75\" (UniqueName: \"kubernetes.io/projected/30787f8a-2875-48eb-b4d8-f4b2fa97d29d-kube-api-access-2tq75\") pod \"payload-processing-e2e-ait-11a7d765-58f7bcf694-bwl46\" (UID: \"30787f8a-2875-48eb-b4d8-f4b2fa97d29d\") " pod="openshift-ingress/payload-processing-e2e-ait-11a7d765-58f7bcf694-bwl46"
Nov 03 20:21:58.713068 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:58.713050    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-11a7d765-58f7bcf694-bwl46"]
Nov 03 20:21:58.718101 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:58.718083    5570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-11a7d765\""
Nov 03 20:21:58.721422 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:58.721404    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-11a7d765-dockercfg-hkttc\""
Nov 03 20:21:58.862797 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:58.839046    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tq75\" (UniqueName: \"kubernetes.io/projected/30787f8a-2875-48eb-b4d8-f4b2fa97d29d-kube-api-access-2tq75\") pod \"payload-processing-e2e-ait-11a7d765-58f7bcf694-bwl46\" (UID: \"30787f8a-2875-48eb-b4d8-f4b2fa97d29d\") " pod="openshift-ingress/payload-processing-e2e-ait-11a7d765-58f7bcf694-bwl46"
Nov 03 20:21:58.863450 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:58.863409    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30787f8a-2875-48eb-b4d8-f4b2fa97d29d-plugins-config-volume\") pod \"payload-processing-e2e-ait-11a7d765-58f7bcf694-bwl46\" (UID: \"30787f8a-2875-48eb-b4d8-f4b2fa97d29d\") " pod="openshift-ingress/payload-processing-e2e-ait-11a7d765-58f7bcf694-bwl46"
Nov 03 20:21:58.884205 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:58.884168    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30787f8a-2875-48eb-b4d8-f4b2fa97d29d-plugins-config-volume\") pod \"payload-processing-e2e-ait-11a7d765-58f7bcf694-bwl46\" (UID: \"30787f8a-2875-48eb-b4d8-f4b2fa97d29d\") " pod="openshift-ingress/payload-processing-e2e-ait-11a7d765-58f7bcf694-bwl46"
Nov 03 20:21:58.891698 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:58.891664    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tq75\" (UniqueName: \"kubernetes.io/projected/30787f8a-2875-48eb-b4d8-f4b2fa97d29d-kube-api-access-2tq75\") pod \"payload-processing-e2e-ait-11a7d765-58f7bcf694-bwl46\" (UID: \"30787f8a-2875-48eb-b4d8-f4b2fa97d29d\") " pod="openshift-ingress/payload-processing-e2e-ait-11a7d765-58f7bcf694-bwl46"
Nov 03 20:21:59.020290 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:59.020251    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-11a7d765-58f7bcf694-bwl46"
Nov 03 20:21:59.628901 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:59.628842    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-11a7d765-58f7bcf694-bwl46"]
Nov 03 20:21:59.629119 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:21:59.629097    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30787f8a_2875_48eb_b4d8_f4b2fa97d29d.slice/crio-a986609b60a4d20bf45299a99ee756331909d33d759410078ccc8507c045fe16 WatchSource:0}: Error finding container a986609b60a4d20bf45299a99ee756331909d33d759410078ccc8507c045fe16: Status 404 returned error can't find the container with id a986609b60a4d20bf45299a99ee756331909d33d759410078ccc8507c045fe16
Nov 03 20:21:59.663083 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:59.663048    5570 patch_prober.go:29] interesting pod/e2e-worker-w2-9973d9-openshift-default-ffb6475-kmqsw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Nov 03 20:21:59.663258 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:21:59.663108    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-kmqsw" podUID="28aada86-170c-4935-80ed-e8c5ee503c14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Nov 03 20:22:00.580791 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:00.579566    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-11a7d765-58f7bcf694-bwl46" event={"ID":"30787f8a-2875-48eb-b4d8-f4b2fa97d29d","Type":"ContainerStarted","Data":"d63387c45ba1613f69aef56cb0d7dda6b838c84ebc1fff7aea0a35fe56967590"}
Nov 03 20:22:00.580791 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:00.579623    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-11a7d765-58f7bcf694-bwl46" event={"ID":"30787f8a-2875-48eb-b4d8-f4b2fa97d29d","Type":"ContainerStarted","Data":"a986609b60a4d20bf45299a99ee756331909d33d759410078ccc8507c045fe16"}
Nov 03 20:22:00.580791 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:00.579719    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-11a7d765-58f7bcf694-bwl46"
Nov 03 20:22:00.622815 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:00.622743    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-11a7d765-58f7bcf694-bwl46" podStartSLOduration=2.622725799 podStartE2EDuration="2.622725799s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:21:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:22:00.622421664 +0000 UTC m=+1054.775083570" watchObservedRunningTime="2026-11-03 20:22:00.622725799 +0000 UTC m=+1054.775387688"
Nov 03 20:22:00.663512 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:00.663464    5570 patch_prober.go:29] interesting pod/e2e-worker-w2-9973d9-openshift-default-ffb6475-kmqsw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Nov 03 20:22:00.663723 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:00.663538    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-kmqsw" podUID="28aada86-170c-4935-80ed-e8c5ee503c14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Nov 03 20:22:01.663175 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:01.663133    5570 patch_prober.go:29] interesting pod/e2e-worker-w2-9973d9-openshift-default-ffb6475-kmqsw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Nov 03 20:22:01.663751 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:01.663190    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-kmqsw" podUID="28aada86-170c-4935-80ed-e8c5ee503c14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Nov 03 20:22:03.048584 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:03.048538    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-kmqsw"
Nov 03 20:22:03.049900 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:03.049875    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-kmqsw"
Nov 03 20:22:09.554796 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:09.548170    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1b67df53-86998f8f74-df6c6"]
Nov 03 20:22:09.706600 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:09.687283    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-1b67df53-86998f8f74-df6c6"
Nov 03 20:22:09.774316 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:09.774284    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-1b67df53-dockercfg-hz5hg\""
Nov 03 20:22:09.791895 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:09.789651    5570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-1b67df53\""
Nov 03 20:22:09.806972 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:09.806880    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1b67df53-86998f8f74-df6c6"]
Nov 03 20:22:09.863184 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:09.863139    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwwd5\" (UniqueName: \"kubernetes.io/projected/848f4854-6df4-4ab9-9baf-b0710fd158d2-kube-api-access-vwwd5\") pod \"payload-processing-e2e-ait-1b67df53-86998f8f74-df6c6\" (UID: \"848f4854-6df4-4ab9-9baf-b0710fd158d2\") " pod="openshift-ingress/payload-processing-e2e-ait-1b67df53-86998f8f74-df6c6"
Nov 03 20:22:09.864036 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:09.864003    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/848f4854-6df4-4ab9-9baf-b0710fd158d2-plugins-config-volume\") pod \"payload-processing-e2e-ait-1b67df53-86998f8f74-df6c6\" (UID: \"848f4854-6df4-4ab9-9baf-b0710fd158d2\") " pod="openshift-ingress/payload-processing-e2e-ait-1b67df53-86998f8f74-df6c6"
Nov 03 20:22:09.965646 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:09.965613    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwwd5\" (UniqueName: \"kubernetes.io/projected/848f4854-6df4-4ab9-9baf-b0710fd158d2-kube-api-access-vwwd5\") pod \"payload-processing-e2e-ait-1b67df53-86998f8f74-df6c6\" (UID: \"848f4854-6df4-4ab9-9baf-b0710fd158d2\") " pod="openshift-ingress/payload-processing-e2e-ait-1b67df53-86998f8f74-df6c6"
Nov 03 20:22:09.965938 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:09.965729    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/848f4854-6df4-4ab9-9baf-b0710fd158d2-plugins-config-volume\") pod \"payload-processing-e2e-ait-1b67df53-86998f8f74-df6c6\" (UID: \"848f4854-6df4-4ab9-9baf-b0710fd158d2\") " pod="openshift-ingress/payload-processing-e2e-ait-1b67df53-86998f8f74-df6c6"
Nov 03 20:22:09.966510 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:09.966474    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/848f4854-6df4-4ab9-9baf-b0710fd158d2-plugins-config-volume\") pod \"payload-processing-e2e-ait-1b67df53-86998f8f74-df6c6\" (UID: \"848f4854-6df4-4ab9-9baf-b0710fd158d2\") " pod="openshift-ingress/payload-processing-e2e-ait-1b67df53-86998f8f74-df6c6"
Nov 03 20:22:09.988495 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:09.988464    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwwd5\" (UniqueName: \"kubernetes.io/projected/848f4854-6df4-4ab9-9baf-b0710fd158d2-kube-api-access-vwwd5\") pod \"payload-processing-e2e-ait-1b67df53-86998f8f74-df6c6\" (UID: \"848f4854-6df4-4ab9-9baf-b0710fd158d2\") " pod="openshift-ingress/payload-processing-e2e-ait-1b67df53-86998f8f74-df6c6"
Nov 03 20:22:10.167030 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:10.166938    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-1b67df53-86998f8f74-df6c6"
Nov 03 20:22:10.477940 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:22:10.477906    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod848f4854_6df4_4ab9_9baf_b0710fd158d2.slice/crio-5e7f857b71665638c9d9b15d407fcad7e0942c9d08bc859f8d1a54e63f0136c1 WatchSource:0}: Error finding container 5e7f857b71665638c9d9b15d407fcad7e0942c9d08bc859f8d1a54e63f0136c1: Status 404 returned error can't find the container with id 5e7f857b71665638c9d9b15d407fcad7e0942c9d08bc859f8d1a54e63f0136c1
Nov 03 20:22:10.478860 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:10.478837    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1b67df53-86998f8f74-df6c6"]
Nov 03 20:22:11.137700 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:11.137600    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-1b67df53-86998f8f74-df6c6" event={"ID":"848f4854-6df4-4ab9-9baf-b0710fd158d2","Type":"ContainerStarted","Data":"d32c0d456877777ed0f51aa4e5276f82ce45c53e91a7e19d5c916e8ebd319c09"}
Nov 03 20:22:11.137700 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:11.137685    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-1b67df53-86998f8f74-df6c6" event={"ID":"848f4854-6df4-4ab9-9baf-b0710fd158d2","Type":"ContainerStarted","Data":"5e7f857b71665638c9d9b15d407fcad7e0942c9d08bc859f8d1a54e63f0136c1"}
Nov 03 20:22:11.138304 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:11.137760    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-1b67df53-86998f8f74-df6c6"
Nov 03 20:22:11.223320 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:11.222701    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-1b67df53-86998f8f74-df6c6" podStartSLOduration=2.222674333 podStartE2EDuration="2.222674333s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:22:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:22:11.209084313 +0000 UTC m=+1065.361746201" watchObservedRunningTime="2026-11-03 20:22:11.222674333 +0000 UTC m=+1065.375336244"
Nov 03 20:22:11.594781 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:11.594735    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-11a7d765-58f7bcf694-bwl46"
Nov 03 20:22:19.310979 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:19.310927    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/premium-w3-482722-kserve-689dc7bd5d-x8rjp" podUID="369045c3-4d6a-49a7-9bc6-f2939883bee8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:22:19.312854 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:19.312820    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/simulator-w0-e40816-kserve-688f79796f-2tvw5" podUID="c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Nov 03 20:22:19.335800 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:19.330910    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/distinct-2-w4-e80203-kserve-756f9b9bc6-69gxj" podUID="ccf5dc2f-f116-4917-bb1a-b136d8ed17f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:22:19.335800 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:19.332483    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz" podUID="3a715ef1-38e9-4386-9eec-e10e8df11a10" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": context deadline exceeded"
Nov 03 20:22:19.335800 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:19.333929    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/distinct-w3-482722-kserve-d876967c-kd488" podUID="9a97f9cc-2261-471d-9053-51ee965850bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:22:19.373052 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:19.372983    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/premium-w1-6fc748-kserve-7474db9cd7-nf2xk" podUID="830e062e-83fc-4de7-91a7-5534dac1d46b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:22:19.375454 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:19.374938    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/distinct-2-w3-482722-kserve-74bf58cf7d-v9cjw" podUID="9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:22:19.393601 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:19.393550    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/distinct-2-w0-e40816-kserve-79fb46654d-s242x" podUID="4ad9b048-925c-4d01-a839-a5f25d4dd02d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:22:19.396652 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:19.393879    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh" podUID="2d3647f2-89e9-4908-a8ee-d84e527abeb0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Nov 03 20:22:19.397755 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:19.393908    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-2-w6-9e1bcb-kserve-7d96d9f6f6-4f7fl" podUID="56353b0f-08ee-4ada-9dc0-ac5afd5a59c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded"
Nov 03 20:22:19.491799 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:19.463415    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/premium-w2-9973d9-kserve-5c6dd4669c-5gv8x" podUID="cefc89ad-7ae1-433b-979d-6601678a6d98" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Nov 03 20:22:19.865948 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:19.865888    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/premium-w4-e80203-kserve-5f9f445456-bms9d" podUID="6ad76efc-20c2-4951-b22a-c5a39cd39ed4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Nov 03 20:22:19.875972 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:19.875929    5570 patch_prober.go:29] interesting pod/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sf2lg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:22:19.876273 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:19.876237    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sf2lg" podUID="07093a96-d60c-490f-910b-734f41752c17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:22:20.182968 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:20.182858    5570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-e80203/distinct-2-w4-e80203-kserve-756f9b9bc6-69gxj" podUID="ccf5dc2f-f116-4917-bb1a-b136d8ed17f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:22:20.311983 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:20.311924    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/premium-w3-482722-kserve-689dc7bd5d-x8rjp" podUID="369045c3-4d6a-49a7-9bc6-f2939883bee8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:22:20.313311 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:20.313286    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-482722/premium-w3-482722-kserve-689dc7bd5d-x8rjp"
Nov 03 20:22:20.325010 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:20.314194    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/simulator-w0-e40816-kserve-688f79796f-2tvw5" podUID="c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:22:20.326020 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:20.325985    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e40816/simulator-w0-e40816-kserve-688f79796f-2tvw5"
Nov 03 20:22:20.333917 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:20.333865    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz" podUID="3a715ef1-38e9-4386-9eec-e10e8df11a10" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:22:20.334292 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:20.334258    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/distinct-w3-482722-kserve-d876967c-kd488" podUID="9a97f9cc-2261-471d-9053-51ee965850bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:22:20.334793 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:20.334750    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/distinct-2-w4-e80203-kserve-756f9b9bc6-69gxj" podUID="ccf5dc2f-f116-4917-bb1a-b136d8ed17f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded"
Nov 03 20:22:20.335556 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:20.335533    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz"
Nov 03 20:22:20.335744 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:20.335728    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-482722/distinct-w3-482722-kserve-d876967c-kd488"
Nov 03 20:22:20.335928 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:20.335914    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-e80203/distinct-2-w4-e80203-kserve-756f9b9bc6-69gxj"
Nov 03 20:22:20.382801 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:20.376834    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/distinct-2-w3-482722-kserve-74bf58cf7d-v9cjw" podUID="9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:22:20.382801 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:20.377290    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/premium-w1-6fc748-kserve-7474db9cd7-nf2xk" podUID="830e062e-83fc-4de7-91a7-5534dac1d46b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Nov 03 20:22:20.382801 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:20.377877    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-482722/distinct-2-w3-482722-kserve-74bf58cf7d-v9cjw"
Nov 03 20:22:20.382801 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:20.377936    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6fc748/premium-w1-6fc748-kserve-7474db9cd7-nf2xk"
Nov 03 20:22:20.414987 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:20.414939    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh" podUID="2d3647f2-89e9-4908-a8ee-d84e527abeb0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:22:20.415667 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:20.415634    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/distinct-2-w0-e40816-kserve-79fb46654d-s242x" podUID="4ad9b048-925c-4d01-a839-a5f25d4dd02d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded"
Nov 03 20:22:20.420071 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:20.417095    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-9e1bcb/premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh"
Nov 03 20:22:20.420297 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:20.420280    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e40816/distinct-2-w0-e40816-kserve-79fb46654d-s242x"
Nov 03 20:22:20.444266 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:20.417952    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-2-w6-9e1bcb-kserve-7d96d9f6f6-4f7fl" podUID="56353b0f-08ee-4ada-9dc0-ac5afd5a59c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:22:20.473802 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:20.472133    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-2-w6-9e1bcb-kserve-7d96d9f6f6-4f7fl"
Nov 03 20:22:20.474016 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:20.473975    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/premium-w2-9973d9-kserve-5c6dd4669c-5gv8x" podUID="cefc89ad-7ae1-433b-979d-6601678a6d98" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:22:20.536818 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:20.482408    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9973d9/premium-w2-9973d9-kserve-5c6dd4669c-5gv8x"
Nov 03 20:22:20.918146 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:20.917411    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/premium-w4-e80203-kserve-5f9f445456-bms9d" podUID="6ad76efc-20c2-4951-b22a-c5a39cd39ed4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Nov 03 20:22:20.923268 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:20.923232    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-e80203/premium-w4-e80203-kserve-5f9f445456-bms9d"
Nov 03 20:22:21.316746 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:21.313227    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/premium-w3-482722-kserve-689dc7bd5d-x8rjp" podUID="369045c3-4d6a-49a7-9bc6-f2939883bee8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:22:21.375229 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:21.352944    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/distinct-2-w4-e80203-kserve-756f9b9bc6-69gxj" podUID="ccf5dc2f-f116-4917-bb1a-b136d8ed17f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:22:21.455800 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:21.407996    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/simulator-w0-e40816-kserve-688f79796f-2tvw5" podUID="c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Nov 03 20:22:21.468002 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:21.467939    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz" podUID="3a715ef1-38e9-4386-9eec-e10e8df11a10" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:22:21.500853 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:21.468755    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh" podUID="2d3647f2-89e9-4908-a8ee-d84e527abeb0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:22:21.502683 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:21.502634    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/distinct-2-w3-482722-kserve-74bf58cf7d-v9cjw" podUID="9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded"
Nov 03 20:22:21.552801 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:21.469001    5570 patch_prober.go:29] interesting pod/e2e-worker-w2-9973d9-openshift-default-ffb6475-qskf7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:22:21.553130 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:21.553089    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-qskf7" podUID="469fc281-f7ee-4584-b3b6-85c6a669f806" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:22:21.627898 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:21.468275    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/distinct-w3-482722-kserve-d876967c-kd488" podUID="9a97f9cc-2261-471d-9053-51ee965850bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:22:21.627898 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:21.536947    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/premium-w1-6fc748-kserve-7474db9cd7-nf2xk" podUID="830e062e-83fc-4de7-91a7-5534dac1d46b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Nov 03 20:22:21.627898 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:21.537504    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-2-w6-9e1bcb-kserve-7d96d9f6f6-4f7fl" podUID="56353b0f-08ee-4ada-9dc0-ac5afd5a59c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:22:21.627898 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:21.537530    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/distinct-2-w0-e40816-kserve-79fb46654d-s242x" podUID="4ad9b048-925c-4d01-a839-a5f25d4dd02d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded"
Nov 03 20:22:21.627898 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:21.536992    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/premium-w2-9973d9-kserve-5c6dd4669c-5gv8x" podUID="cefc89ad-7ae1-433b-979d-6601678a6d98" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:22:21.743802 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:21.733177    5570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-482722/distinct-2-w3-482722-kserve-74bf58cf7d-v9cjw" podUID="9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:22:21.756299 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:21.756258    5570 patch_prober.go:29] interesting pod/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-vzfw9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:22:21.779912 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:21.779852    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-vzfw9" podUID="a739707b-9c5a-4c9b-90df-501c5dd0c98d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:22:21.925801 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:21.925678    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/premium-w4-e80203-kserve-5f9f445456-bms9d" podUID="6ad76efc-20c2-4951-b22a-c5a39cd39ed4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Nov 03 20:22:22.314999 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:22.314919    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/premium-w3-482722-kserve-689dc7bd5d-x8rjp" podUID="369045c3-4d6a-49a7-9bc6-f2939883bee8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:22:22.371093 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:22.371043    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/distinct-2-w4-e80203-kserve-756f9b9bc6-69gxj" podUID="ccf5dc2f-f116-4917-bb1a-b136d8ed17f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:22:22.450452 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:22.450403    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/simulator-w0-e40816-kserve-688f79796f-2tvw5" podUID="c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:22:22.614858 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:22.610382    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/distinct-2-w3-482722-kserve-74bf58cf7d-v9cjw" podUID="9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded"
Nov 03 20:22:22.662505 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:22.662456    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh" podUID="2d3647f2-89e9-4908-a8ee-d84e527abeb0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Nov 03 20:22:22.779830 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:22.695890    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/distinct-2-w0-e40816-kserve-79fb46654d-s242x" podUID="4ad9b048-925c-4d01-a839-a5f25d4dd02d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:22:22.798222 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:22.695931    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/premium-w1-6fc748-kserve-7474db9cd7-nf2xk" podUID="830e062e-83fc-4de7-91a7-5534dac1d46b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Nov 03 20:22:22.798222 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:22.695956    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-2-w6-9e1bcb-kserve-7d96d9f6f6-4f7fl" podUID="56353b0f-08ee-4ada-9dc0-ac5afd5a59c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded"
Nov 03 20:22:22.798222 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:22.695998    5570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-e40816/distinct-2-w0-e40816-kserve-79fb46654d-s242x" podUID="4ad9b048-925c-4d01-a839-a5f25d4dd02d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/health\": context deadline exceeded"
Nov 03 20:22:22.798222 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:22.696028    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/premium-w2-9973d9-kserve-5c6dd4669c-5gv8x" podUID="cefc89ad-7ae1-433b-979d-6601678a6d98" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:22:22.798222 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:22.696050    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz" podUID="3a715ef1-38e9-4386-9eec-e10e8df11a10" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": context deadline exceeded"
Nov 03 20:22:22.798222 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:22.696068    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/distinct-w3-482722-kserve-d876967c-kd488" podUID="9a97f9cc-2261-471d-9053-51ee965850bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded"
Nov 03 20:22:22.798222 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:22.751670    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-1b67df53-86998f8f74-df6c6"
Nov 03 20:22:23.094806 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:23.091528    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/premium-w4-e80203-kserve-5f9f445456-bms9d" podUID="6ad76efc-20c2-4951-b22a-c5a39cd39ed4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:22:23.212799 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:23.209079    5570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-6fc748/premium-w1-6fc748-kserve-7474db9cd7-nf2xk" podUID="830e062e-83fc-4de7-91a7-5534dac1d46b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": context deadline exceeded"
Nov 03 20:22:23.502806 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:23.471652    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/premium-w3-482722-kserve-689dc7bd5d-x8rjp" podUID="369045c3-4d6a-49a7-9bc6-f2939883bee8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Nov 03 20:22:23.726798 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:23.716460    5570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-e80203/premium-w4-e80203-kserve-5f9f445456-bms9d" podUID="6ad76efc-20c2-4951-b22a-c5a39cd39ed4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:22:23.726798 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:23.591641    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/simulator-w0-e40816-kserve-688f79796f-2tvw5" podUID="c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Nov 03 20:22:23.934865 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:23.740922    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/distinct-2-w3-482722-kserve-74bf58cf7d-v9cjw" podUID="9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:22:23.934865 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:23.623186    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-cpfzl" podUID="30b21f09-fd9e-4836-9fc3-c19f818c9022" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:22:23.934865 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:23.591581    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/distinct-2-w4-e80203-kserve-756f9b9bc6-69gxj" podUID="ccf5dc2f-f116-4917-bb1a-b136d8ed17f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded"
Nov 03 20:22:23.934865 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:23.904985    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/distinct-2-w0-e40816-kserve-79fb46654d-s242x" podUID="4ad9b048-925c-4d01-a839-a5f25d4dd02d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded"
Nov 03 20:22:23.935560 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:23.935524    5570 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-657vf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:22:23.935751 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:23.935725    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-657vf" podUID="f089dd20-d6a6-4c63-88a5-3fbe9a2ff315" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:22:23.997350 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:23.941377    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/distinct-w3-482722-kserve-d876967c-kd488" podUID="9a97f9cc-2261-471d-9053-51ee965850bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:22:24.124803 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:24.114204    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/premium-w1-6fc748-kserve-7474db9cd7-nf2xk" podUID="830e062e-83fc-4de7-91a7-5534dac1d46b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Nov 03 20:22:24.146799 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:24.142843    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/premium-w2-9973d9-kserve-5c6dd4669c-5gv8x" podUID="cefc89ad-7ae1-433b-979d-6601678a6d98" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Nov 03 20:22:24.150805 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:24.147644    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-2-w6-9e1bcb-kserve-7d96d9f6f6-4f7fl" podUID="56353b0f-08ee-4ada-9dc0-ac5afd5a59c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded"
Nov 03 20:22:24.193556 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:24.193495    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz" podUID="3a715ef1-38e9-4386-9eec-e10e8df11a10" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": context deadline exceeded"
Nov 03 20:22:24.235320 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:24.235215    5570 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-vdrfs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:22:24.235320 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:24.235299    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-vdrfs" podUID="1b70a87b-76a6-40e6-9ced-037c3ceee5cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:22:24.344952 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:24.168712    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/premium-w4-e80203-kserve-5f9f445456-bms9d" podUID="6ad76efc-20c2-4951-b22a-c5a39cd39ed4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:22:24.413969 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:24.413916    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87s5mf" podUID="b7306cb9-eeae-447b-b834-870c8d49a5aa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": context deadline exceeded"
Nov 03 20:22:24.479929 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:24.453936    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh" podUID="2d3647f2-89e9-4908-a8ee-d84e527abeb0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Nov 03 20:22:24.731850 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:24.731723    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/simulator-w0-e40816-kserve-688f79796f-2tvw5" podUID="c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:22:24.768352 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:24.768301    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/distinct-2-w4-e80203-kserve-756f9b9bc6-69gxj" podUID="ccf5dc2f-f116-4917-bb1a-b136d8ed17f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:22:24.771975 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:24.771937    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/premium-w3-482722-kserve-689dc7bd5d-x8rjp" podUID="369045c3-4d6a-49a7-9bc6-f2939883bee8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:22:24.793298 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:24.792898    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/distinct-2-w3-482722-kserve-74bf58cf7d-v9cjw" podUID="9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:22:24.909099 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:24.907882    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/distinct-2-w0-e40816-kserve-79fb46654d-s242x" podUID="4ad9b048-925c-4d01-a839-a5f25d4dd02d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:22:24.909099 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:24.908299    5570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-482722/distinct-w3-482722-kserve-d876967c-kd488" podUID="9a97f9cc-2261-471d-9053-51ee965850bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/health\": context deadline exceeded"
Nov 03 20:22:24.927857 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:24.927807    5570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-9973d9/premium-w2-9973d9-kserve-5c6dd4669c-5gv8x" podUID="cefc89ad-7ae1-433b-979d-6601678a6d98" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:22:25.129858 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:25.129371    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/distinct-w3-482722-kserve-d876967c-kd488" podUID="9a97f9cc-2261-471d-9053-51ee965850bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:22:25.133797 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:25.130345    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/premium-w1-6fc748-kserve-7474db9cd7-nf2xk" podUID="830e062e-83fc-4de7-91a7-5534dac1d46b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:22:25.151801 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:25.150892    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/premium-w2-9973d9-kserve-5c6dd4669c-5gv8x" podUID="cefc89ad-7ae1-433b-979d-6601678a6d98" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Nov 03 20:22:25.197977 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:25.197920    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz" podUID="3a715ef1-38e9-4386-9eec-e10e8df11a10" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:22:25.199521 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:25.198846    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-2-w6-9e1bcb-kserve-7d96d9f6f6-4f7fl" podUID="56353b0f-08ee-4ada-9dc0-ac5afd5a59c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:22:25.441878 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:25.441836    5570 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz" podUID="3a715ef1-38e9-4386-9eec-e10e8df11a10" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:22:25.451573 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:25.451537    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/premium-w4-e80203-kserve-5f9f445456-bms9d" podUID="6ad76efc-20c2-4951-b22a-c5a39cd39ed4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:22:25.455530 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:25.455497    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh" podUID="2d3647f2-89e9-4908-a8ee-d84e527abeb0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Nov 03 20:22:25.603522 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:25.603467    5570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-482722/premium-w3-482722-kserve-689dc7bd5d-x8rjp" podUID="369045c3-4d6a-49a7-9bc6-f2939883bee8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:22:25.737869 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:25.735645    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/simulator-w0-e40816-kserve-688f79796f-2tvw5" podUID="c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:22:25.790798 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:25.772212    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/distinct-2-w4-e80203-kserve-756f9b9bc6-69gxj" podUID="ccf5dc2f-f116-4917-bb1a-b136d8ed17f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:22:25.790798 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:25.773836    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/premium-w3-482722-kserve-689dc7bd5d-x8rjp" podUID="369045c3-4d6a-49a7-9bc6-f2939883bee8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:22:25.823344 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:25.823303    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/distinct-2-w3-482722-kserve-74bf58cf7d-v9cjw" podUID="9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:22:25.955798 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:25.935556    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/distinct-2-w0-e40816-kserve-79fb46654d-s242x" podUID="4ad9b048-925c-4d01-a839-a5f25d4dd02d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded"
Nov 03 20:22:26.142885 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:26.130474    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/premium-w1-6fc748-kserve-7474db9cd7-nf2xk" podUID="830e062e-83fc-4de7-91a7-5534dac1d46b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:22:26.183799 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:26.148950    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/distinct-w3-482722-kserve-d876967c-kd488" podUID="9a97f9cc-2261-471d-9053-51ee965850bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:22:26.184426 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:26.153540    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/premium-w2-9973d9-kserve-5c6dd4669c-5gv8x" podUID="cefc89ad-7ae1-433b-979d-6601678a6d98" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:22:26.230820 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:26.229015    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz" podUID="3a715ef1-38e9-4386-9eec-e10e8df11a10" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:22:26.270053 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:26.270018    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-482722/distinct-2-w3-482722-kserve-74bf58cf7d-v9cjw"
Nov 03 20:22:26.270369 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:26.270350    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-2-w6-9e1bcb-kserve-7d96d9f6f6-4f7fl"
Nov 03 20:22:26.275809 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:26.270528    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-482722/premium-w3-482722-kserve-689dc7bd5d-x8rjp"
Nov 03 20:22:26.502793 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:26.499155    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/premium-w4-e80203-kserve-5f9f445456-bms9d" podUID="6ad76efc-20c2-4951-b22a-c5a39cd39ed4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Nov 03 20:22:26.537811 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:26.513406    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh" podUID="2d3647f2-89e9-4908-a8ee-d84e527abeb0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:22:26.703253 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:26.703224    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz"
Nov 03 20:22:26.708205 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:26.707459    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-482722/distinct-w3-482722-kserve-d876967c-kd488"
Nov 03 20:22:26.721070 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:26.718803    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e40816/simulator-w0-e40816-kserve-688f79796f-2tvw5"
Nov 03 20:22:26.721070 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:26.718861    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-e80203/distinct-2-w4-e80203-kserve-756f9b9bc6-69gxj"
Nov 03 20:22:26.730793 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:26.726302    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6fc748/premium-w1-6fc748-kserve-7474db9cd7-nf2xk"
Nov 03 20:22:26.737797 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:26.734606    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-e80203/premium-w4-e80203-kserve-5f9f445456-bms9d"
Nov 03 20:22:26.737797 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:26.736083    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-9e1bcb/premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh"
Nov 03 20:22:26.801234 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:26.789195    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9973d9/premium-w2-9973d9-kserve-5c6dd4669c-5gv8x"
Nov 03 20:22:27.015212 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:27.015173    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/distinct-2-w0-e40816-kserve-79fb46654d-s242x" podUID="4ad9b048-925c-4d01-a839-a5f25d4dd02d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:22:27.050732 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:27.050206    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e40816/distinct-2-w0-e40816-kserve-79fb46654d-s242x"
Nov 03 20:22:27.388236 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:22:27.385034    5570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8bwhz" containerName="extractor"
Nov 03 20:22:35.973099 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:35.973064    5570 generic.go:356] "Generic (PLEG): container finished" podID="b9bd3e2e-35ec-482d-8dc2-1a18f255c220" containerID="e3bed704031bf8816c6564d3549a7c19e2f619f6ff9c7ec362d1fda9e7309df2" exitCode=0
Nov 03 20:22:35.973536 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:35.973140    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-9798ad8e-gw-openshift-default-575b7f8f9c-xg9nd" event={"ID":"b9bd3e2e-35ec-482d-8dc2-1a18f255c220","Type":"ContainerDied","Data":"e3bed704031bf8816c6564d3549a7c19e2f619f6ff9c7ec362d1fda9e7309df2"}
Nov 03 20:22:35.973586 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:35.973547    5570 scope.go:122] "RemoveContainer" containerID="e3bed704031bf8816c6564d3549a7c19e2f619f6ff9c7ec362d1fda9e7309df2"
Nov 03 20:22:35.975150 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:35.975124    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:22:35.975235 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:35.975188    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:22:35.975235 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:35.975219    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:22:36.983683 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:36.983606    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-9798ad8e-gw-openshift-default-575b7f8f9c-xg9nd" event={"ID":"b9bd3e2e-35ec-482d-8dc2-1a18f255c220","Type":"ContainerStarted","Data":"b9803342d64166210f8684fc1ff7ac901d162d929fb81e72f946ffd6c7fffb2c"}
Nov 03 20:22:37.030636 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:37.030601    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-649609db-pqmqh"]
Nov 03 20:22:37.035152 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:37.035125    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-649609db-pqmqh"]
Nov 03 20:22:37.093907 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:37.093874    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-9798ad8e-gw-openshift-default-575b7f8f9c-xg9nd"
Nov 03 20:22:37.262000 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:37.261915    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-9798ad8e-gw-openshift-default-575b7f8f9c-xg9nd"
Nov 03 20:22:37.262754 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:37.262720    5570 patch_prober.go:29] interesting pod/e2e-migrate-9798ad8e-gw-openshift-default-575b7f8f9c-xg9nd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body=
Nov 03 20:22:37.263004 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:37.262974    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-9798ad8e-gw-openshift-default-575b7f8f9c-xg9nd" podUID="b9bd3e2e-35ec-482d-8dc2-1a18f255c220" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused"
Nov 03 20:22:37.970873 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:37.970839    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a833c968-f463-4a3a-9451-57c0ee19a768" path="/var/lib/kubelet/pods/a833c968-f463-4a3a-9451-57c0ee19a768/volumes"
Nov 03 20:22:38.262288 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:38.262189    5570 patch_prober.go:29] interesting pod/e2e-migrate-9798ad8e-gw-openshift-default-575b7f8f9c-xg9nd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body=
Nov 03 20:22:38.262288 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:38.262262    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-9798ad8e-gw-openshift-default-575b7f8f9c-xg9nd" podUID="b9bd3e2e-35ec-482d-8dc2-1a18f255c220" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused"
Nov 03 20:22:39.266799 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:39.264200    5570 patch_prober.go:29] interesting pod/e2e-migrate-9798ad8e-gw-openshift-default-575b7f8f9c-xg9nd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body=
Nov 03 20:22:39.266799 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:39.264271    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-9798ad8e-gw-openshift-default-575b7f8f9c-xg9nd" podUID="b9bd3e2e-35ec-482d-8dc2-1a18f255c220" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused"
Nov 03 20:22:39.497321 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:39.497057    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-9798ad8e-gw-openshift-default-575b7f8f9c-xg9nd"]
Nov 03 20:22:40.030551 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:40.030508    5570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-9798ad8e-gw-openshift-default-575b7f8f9c-xg9nd" podUID="b9bd3e2e-35ec-482d-8dc2-1a18f255c220" containerName="istio-proxy" containerID="cri-o://b9803342d64166210f8684fc1ff7ac901d162d929fb81e72f946ffd6c7fffb2c" gracePeriod=30
Nov 03 20:22:43.402286 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:43.402247    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-11a7d765-58f7bcf694-bwl46"]
Nov 03 20:22:45.917294 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:45.917257    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-04bf5fc9-7f4947f948-qv8lm"]
Nov 03 20:22:45.920976 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:45.920952    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-04bf5fc9-7f4947f948-qv8lm"
Nov 03 20:22:45.938474 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:45.938441    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-04bf5fc9-7f4947f948-qv8lm"]
Nov 03 20:22:45.946753 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:45.946712    5570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-04bf5fc9\""
Nov 03 20:22:45.951364 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:45.951329    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-04bf5fc9-dockercfg-dqdqd\""
Nov 03 20:22:45.987968 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:45.987931    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-04bf5fc9-c7bd84cd7-p8qbk"]
Nov 03 20:22:45.992711 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:45.992686    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-04bf5fc9-c7bd84cd7-p8qbk"
Nov 03 20:22:46.006092 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:46.006066    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-04bf5fc9-c7bd84cd7-p8qbk"]
Nov 03 20:22:46.208854 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:46.145538    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61d0dabb-f778-4b31-a2c1-99b7d06de1b8-plugins-config-volume\") pod \"payload-processing-e2e-ait-04bf5fc9-7f4947f948-qv8lm\" (UID: \"61d0dabb-f778-4b31-a2c1-99b7d06de1b8\") " pod="openshift-ingress/payload-processing-e2e-ait-04bf5fc9-7f4947f948-qv8lm"
Nov 03 20:22:46.216177 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:46.216126    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htgsg\" (UniqueName: \"kubernetes.io/projected/61d0dabb-f778-4b31-a2c1-99b7d06de1b8-kube-api-access-htgsg\") pod \"payload-processing-e2e-ait-04bf5fc9-7f4947f948-qv8lm\" (UID: \"61d0dabb-f778-4b31-a2c1-99b7d06de1b8\") " pod="openshift-ingress/payload-processing-e2e-ait-04bf5fc9-7f4947f948-qv8lm"
Nov 03 20:22:46.216837 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:46.216756    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlg5f\" (UniqueName: \"kubernetes.io/projected/8feccf11-f12a-4d1c-a5d0-b210351dca24-kube-api-access-qlg5f\") pod \"payload-pre-processing-e2e-ait-04bf5fc9-c7bd84cd7-p8qbk\" (UID: \"8feccf11-f12a-4d1c-a5d0-b210351dca24\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-04bf5fc9-c7bd84cd7-p8qbk"
Nov 03 20:22:46.217103 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:46.217085    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8feccf11-f12a-4d1c-a5d0-b210351dca24-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-04bf5fc9-c7bd84cd7-p8qbk\" (UID: \"8feccf11-f12a-4d1c-a5d0-b210351dca24\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-04bf5fc9-c7bd84cd7-p8qbk"
Nov 03 20:22:46.335797 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:46.335415    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htgsg\" (UniqueName: \"kubernetes.io/projected/61d0dabb-f778-4b31-a2c1-99b7d06de1b8-kube-api-access-htgsg\") pod \"payload-processing-e2e-ait-04bf5fc9-7f4947f948-qv8lm\" (UID: \"61d0dabb-f778-4b31-a2c1-99b7d06de1b8\") " pod="openshift-ingress/payload-processing-e2e-ait-04bf5fc9-7f4947f948-qv8lm"
Nov 03 20:22:46.335797 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:46.335631    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlg5f\" (UniqueName: \"kubernetes.io/projected/8feccf11-f12a-4d1c-a5d0-b210351dca24-kube-api-access-qlg5f\") pod \"payload-pre-processing-e2e-ait-04bf5fc9-c7bd84cd7-p8qbk\" (UID: \"8feccf11-f12a-4d1c-a5d0-b210351dca24\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-04bf5fc9-c7bd84cd7-p8qbk"
Nov 03 20:22:46.335797 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:46.335714    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8feccf11-f12a-4d1c-a5d0-b210351dca24-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-04bf5fc9-c7bd84cd7-p8qbk\" (UID: \"8feccf11-f12a-4d1c-a5d0-b210351dca24\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-04bf5fc9-c7bd84cd7-p8qbk"
Nov 03 20:22:46.336134 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:46.335925    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61d0dabb-f778-4b31-a2c1-99b7d06de1b8-plugins-config-volume\") pod \"payload-processing-e2e-ait-04bf5fc9-7f4947f948-qv8lm\" (UID: \"61d0dabb-f778-4b31-a2c1-99b7d06de1b8\") " pod="openshift-ingress/payload-processing-e2e-ait-04bf5fc9-7f4947f948-qv8lm"
Nov 03 20:22:46.355671 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:46.337017    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61d0dabb-f778-4b31-a2c1-99b7d06de1b8-plugins-config-volume\") pod \"payload-processing-e2e-ait-04bf5fc9-7f4947f948-qv8lm\" (UID: \"61d0dabb-f778-4b31-a2c1-99b7d06de1b8\") " pod="openshift-ingress/payload-processing-e2e-ait-04bf5fc9-7f4947f948-qv8lm"
Nov 03 20:22:46.375796 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:46.365959    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8feccf11-f12a-4d1c-a5d0-b210351dca24-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-04bf5fc9-c7bd84cd7-p8qbk\" (UID: \"8feccf11-f12a-4d1c-a5d0-b210351dca24\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-04bf5fc9-c7bd84cd7-p8qbk"
Nov 03 20:22:46.459692 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:46.453065    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-htgsg\" (UniqueName: \"kubernetes.io/projected/61d0dabb-f778-4b31-a2c1-99b7d06de1b8-kube-api-access-htgsg\") pod \"payload-processing-e2e-ait-04bf5fc9-7f4947f948-qv8lm\" (UID: \"61d0dabb-f778-4b31-a2c1-99b7d06de1b8\") " pod="openshift-ingress/payload-processing-e2e-ait-04bf5fc9-7f4947f948-qv8lm"
Nov 03 20:22:46.465889 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:46.462419    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlg5f\" (UniqueName: \"kubernetes.io/projected/8feccf11-f12a-4d1c-a5d0-b210351dca24-kube-api-access-qlg5f\") pod \"payload-pre-processing-e2e-ait-04bf5fc9-c7bd84cd7-p8qbk\" (UID: \"8feccf11-f12a-4d1c-a5d0-b210351dca24\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-04bf5fc9-c7bd84cd7-p8qbk"
Nov 03 20:22:46.537676 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:46.537592    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-04bf5fc9-7f4947f948-qv8lm"
Nov 03 20:22:46.608558 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:46.608530    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-04bf5fc9-c7bd84cd7-p8qbk"
Nov 03 20:22:47.724322 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:47.724289    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-04bf5fc9-c7bd84cd7-p8qbk"]
Nov 03 20:22:47.733912 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:47.733475    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-04bf5fc9-7f4947f948-qv8lm"]
Nov 03 20:22:47.743025 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:22:47.742987    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61d0dabb_f778_4b31_a2c1_99b7d06de1b8.slice/crio-eeac0e0765fb57d0cef27da93c77deca15e897031f09a7aa956d22e13b2f926a WatchSource:0}: Error finding container eeac0e0765fb57d0cef27da93c77deca15e897031f09a7aa956d22e13b2f926a: Status 404 returned error can't find the container with id eeac0e0765fb57d0cef27da93c77deca15e897031f09a7aa956d22e13b2f926a
Nov 03 20:22:47.768847 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:22:47.768689    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8feccf11_f12a_4d1c_a5d0_b210351dca24.slice/crio-274f65a436631237f2d0fcfe5a258f51449275dd7ad87bc7626f6ff18f1812d0 WatchSource:0}: Error finding container 274f65a436631237f2d0fcfe5a258f51449275dd7ad87bc7626f6ff18f1812d0: Status 404 returned error can't find the container with id 274f65a436631237f2d0fcfe5a258f51449275dd7ad87bc7626f6ff18f1812d0
Nov 03 20:22:47.946743 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:47.946713    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-9798ad8e-gw-openshift-default-575b7f8f9c-xg9nd"
Nov 03 20:22:48.072658 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:48.072567    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9bd3e2e-35ec-482d-8dc2-1a18f255c220-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9bd3e2e-35ec-482d-8dc2-1a18f255c220-workload-certs\") pod \"b9bd3e2e-35ec-482d-8dc2-1a18f255c220\" (UID: \"b9bd3e2e-35ec-482d-8dc2-1a18f255c220\") "
Nov 03 20:22:48.072871 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:48.072719    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b9bd3e2e-35ec-482d-8dc2-1a18f255c220-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9bd3e2e-35ec-482d-8dc2-1a18f255c220-istio-podinfo\") pod \"b9bd3e2e-35ec-482d-8dc2-1a18f255c220\" (UID: \"b9bd3e2e-35ec-482d-8dc2-1a18f255c220\") "
Nov 03 20:22:48.072871 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:48.072829    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9bd3e2e-35ec-482d-8dc2-1a18f255c220-kube-api-access-v67wt\" (UniqueName: \"kubernetes.io/projected/b9bd3e2e-35ec-482d-8dc2-1a18f255c220-kube-api-access-v67wt\") pod \"b9bd3e2e-35ec-482d-8dc2-1a18f255c220\" (UID: \"b9bd3e2e-35ec-482d-8dc2-1a18f255c220\") "
Nov 03 20:22:48.072871 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:48.072860    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9bd3e2e-35ec-482d-8dc2-1a18f255c220-workload-certs" pod "b9bd3e2e-35ec-482d-8dc2-1a18f255c220" (UID: "b9bd3e2e-35ec-482d-8dc2-1a18f255c220"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:22:48.073039 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:48.072881    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9bd3e2e-35ec-482d-8dc2-1a18f255c220-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9bd3e2e-35ec-482d-8dc2-1a18f255c220-credential-socket\") pod \"b9bd3e2e-35ec-482d-8dc2-1a18f255c220\" (UID: \"b9bd3e2e-35ec-482d-8dc2-1a18f255c220\") "
Nov 03 20:22:48.073039 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:48.072996    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9bd3e2e-35ec-482d-8dc2-1a18f255c220-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9bd3e2e-35ec-482d-8dc2-1a18f255c220-workload-socket\") pod \"b9bd3e2e-35ec-482d-8dc2-1a18f255c220\" (UID: \"b9bd3e2e-35ec-482d-8dc2-1a18f255c220\") "
Nov 03 20:22:48.073151 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:48.073053    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9bd3e2e-35ec-482d-8dc2-1a18f255c220-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9bd3e2e-35ec-482d-8dc2-1a18f255c220-istiod-ca-cert\") pod \"b9bd3e2e-35ec-482d-8dc2-1a18f255c220\" (UID: \"b9bd3e2e-35ec-482d-8dc2-1a18f255c220\") "
Nov 03 20:22:48.073200 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:48.073168    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9bd3e2e-35ec-482d-8dc2-1a18f255c220-credential-socket" pod "b9bd3e2e-35ec-482d-8dc2-1a18f255c220" (UID: "b9bd3e2e-35ec-482d-8dc2-1a18f255c220"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:22:48.073251 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:48.073200    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9bd3e2e-35ec-482d-8dc2-1a18f255c220-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9bd3e2e-35ec-482d-8dc2-1a18f255c220-istio-envoy\") pod \"b9bd3e2e-35ec-482d-8dc2-1a18f255c220\" (UID: \"b9bd3e2e-35ec-482d-8dc2-1a18f255c220\") "
Nov 03 20:22:48.073300 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:48.073281    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9bd3e2e-35ec-482d-8dc2-1a18f255c220-workload-socket" pod "b9bd3e2e-35ec-482d-8dc2-1a18f255c220" (UID: "b9bd3e2e-35ec-482d-8dc2-1a18f255c220"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:22:48.073358 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:48.073296    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9bd3e2e-35ec-482d-8dc2-1a18f255c220-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9bd3e2e-35ec-482d-8dc2-1a18f255c220-istio-data\") pod \"b9bd3e2e-35ec-482d-8dc2-1a18f255c220\" (UID: \"b9bd3e2e-35ec-482d-8dc2-1a18f255c220\") "
Nov 03 20:22:48.073358 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:48.073344    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9bd3e2e-35ec-482d-8dc2-1a18f255c220-istio-token\" (UniqueName: \"kubernetes.io/projected/b9bd3e2e-35ec-482d-8dc2-1a18f255c220-istio-token\") pod \"b9bd3e2e-35ec-482d-8dc2-1a18f255c220\" (UID: \"b9bd3e2e-35ec-482d-8dc2-1a18f255c220\") "
Nov 03 20:22:48.073526 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:48.073505    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9bd3e2e-35ec-482d-8dc2-1a18f255c220-istio-data" pod "b9bd3e2e-35ec-482d-8dc2-1a18f255c220" (UID: "b9bd3e2e-35ec-482d-8dc2-1a18f255c220"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:22:48.073602 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:48.073580    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9bd3e2e-35ec-482d-8dc2-1a18f255c220-istiod-ca-cert" pod "b9bd3e2e-35ec-482d-8dc2-1a18f255c220" (UID: "b9bd3e2e-35ec-482d-8dc2-1a18f255c220"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:22:48.074091 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:48.074072    5570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9bd3e2e-35ec-482d-8dc2-1a18f255c220-istio-data\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:22:48.074199 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:48.074094    5570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9bd3e2e-35ec-482d-8dc2-1a18f255c220-workload-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:22:48.074199 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:48.074107    5570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9bd3e2e-35ec-482d-8dc2-1a18f255c220-credential-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:22:48.074199 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:48.074120    5570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9bd3e2e-35ec-482d-8dc2-1a18f255c220-workload-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:22:48.074199 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:48.074132    5570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9bd3e2e-35ec-482d-8dc2-1a18f255c220-istiod-ca-cert\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:22:48.075064 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:48.075037    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b9bd3e2e-35ec-482d-8dc2-1a18f255c220-istio-podinfo" pod "b9bd3e2e-35ec-482d-8dc2-1a18f255c220" (UID: "b9bd3e2e-35ec-482d-8dc2-1a18f255c220"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:22:48.075384 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:48.075360    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9bd3e2e-35ec-482d-8dc2-1a18f255c220-kube-api-access-v67wt" pod "b9bd3e2e-35ec-482d-8dc2-1a18f255c220" (UID: "b9bd3e2e-35ec-482d-8dc2-1a18f255c220"). InnerVolumeSpecName "kube-api-access-v67wt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:22:48.075460 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:48.075397    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9bd3e2e-35ec-482d-8dc2-1a18f255c220-istio-token" pod "b9bd3e2e-35ec-482d-8dc2-1a18f255c220" (UID: "b9bd3e2e-35ec-482d-8dc2-1a18f255c220"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:22:48.075460 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:48.075425    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9bd3e2e-35ec-482d-8dc2-1a18f255c220-istio-envoy" pod "b9bd3e2e-35ec-482d-8dc2-1a18f255c220" (UID: "b9bd3e2e-35ec-482d-8dc2-1a18f255c220"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:22:48.175316 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:48.175277    5570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9bd3e2e-35ec-482d-8dc2-1a18f255c220-istio-envoy\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:22:48.175316 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:48.175306    5570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9bd3e2e-35ec-482d-8dc2-1a18f255c220-istio-token\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:22:48.175316 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:48.175318    5570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9bd3e2e-35ec-482d-8dc2-1a18f255c220-istio-podinfo\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:22:48.175548 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:48.175331    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v67wt\" (UniqueName: \"kubernetes.io/projected/b9bd3e2e-35ec-482d-8dc2-1a18f255c220-kube-api-access-v67wt\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:22:48.410856 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:48.408721    5570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-11a7d765-58f7bcf694-bwl46" podUID="30787f8a-2875-48eb-b4d8-f4b2fa97d29d" containerName="payload-processing" containerID="cri-o://d63387c45ba1613f69aef56cb0d7dda6b838c84ebc1fff7aea0a35fe56967590" gracePeriod=25
Nov 03 20:22:48.764714 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:48.764679    5570 generic.go:356] "Generic (PLEG): container finished" podID="30787f8a-2875-48eb-b4d8-f4b2fa97d29d" containerID="d63387c45ba1613f69aef56cb0d7dda6b838c84ebc1fff7aea0a35fe56967590" exitCode=0
Nov 03 20:22:48.765230 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:48.765211    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-11a7d765-58f7bcf694-bwl46" event={"ID":"30787f8a-2875-48eb-b4d8-f4b2fa97d29d","Type":"ContainerDied","Data":"d63387c45ba1613f69aef56cb0d7dda6b838c84ebc1fff7aea0a35fe56967590"}
Nov 03 20:22:48.774795 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:48.770816    5570 generic.go:356] "Generic (PLEG): container finished" podID="b9bd3e2e-35ec-482d-8dc2-1a18f255c220" containerID="b9803342d64166210f8684fc1ff7ac901d162d929fb81e72f946ffd6c7fffb2c" exitCode=0
Nov 03 20:22:48.774795 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:48.771006    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-9798ad8e-gw-openshift-default-575b7f8f9c-xg9nd"
Nov 03 20:22:48.774795 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:48.771151    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-9798ad8e-gw-openshift-default-575b7f8f9c-xg9nd" event={"ID":"b9bd3e2e-35ec-482d-8dc2-1a18f255c220","Type":"ContainerDied","Data":"b9803342d64166210f8684fc1ff7ac901d162d929fb81e72f946ffd6c7fffb2c"}
Nov 03 20:22:48.774795 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:48.771193    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-9798ad8e-gw-openshift-default-575b7f8f9c-xg9nd" event={"ID":"b9bd3e2e-35ec-482d-8dc2-1a18f255c220","Type":"ContainerDied","Data":"7439411a98f60fc0301865ae8060d5ac52dd2f1a5b1624a27702e41e9194cf2b"}
Nov 03 20:22:48.774795 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:48.771214    5570 scope.go:122] "RemoveContainer" containerID="b9803342d64166210f8684fc1ff7ac901d162d929fb81e72f946ffd6c7fffb2c"
Nov 03 20:22:48.778445 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:48.778379    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-04bf5fc9-c7bd84cd7-p8qbk" event={"ID":"8feccf11-f12a-4d1c-a5d0-b210351dca24","Type":"ContainerStarted","Data":"6a97c77e38d7534c04e946915e07b20e05124dcf60bcb203042b000f68308135"}
Nov 03 20:22:48.778604 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:48.778491    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-04bf5fc9-c7bd84cd7-p8qbk" event={"ID":"8feccf11-f12a-4d1c-a5d0-b210351dca24","Type":"ContainerStarted","Data":"274f65a436631237f2d0fcfe5a258f51449275dd7ad87bc7626f6ff18f1812d0"}
Nov 03 20:22:48.778894 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:48.778870    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-04bf5fc9-c7bd84cd7-p8qbk"
Nov 03 20:22:48.786967 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:48.786937    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-04bf5fc9-7f4947f948-qv8lm" event={"ID":"61d0dabb-f778-4b31-a2c1-99b7d06de1b8","Type":"ContainerStarted","Data":"89eb1f9fe3ebba1fc9162ab74a792d20ed26b277961eda2903b11584958447e0"}
Nov 03 20:22:48.787141 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:48.787089    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-04bf5fc9-7f4947f948-qv8lm" event={"ID":"61d0dabb-f778-4b31-a2c1-99b7d06de1b8","Type":"ContainerStarted","Data":"eeac0e0765fb57d0cef27da93c77deca15e897031f09a7aa956d22e13b2f926a"}
Nov 03 20:22:48.788677 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:48.788656    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-04bf5fc9-7f4947f948-qv8lm"
Nov 03 20:22:48.807498 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:48.807479    5570 scope.go:122] "RemoveContainer" containerID="e3bed704031bf8816c6564d3549a7c19e2f619f6ff9c7ec362d1fda9e7309df2"
Nov 03 20:22:48.836526 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:48.836501    5570 scope.go:122] "RemoveContainer" containerID="b9803342d64166210f8684fc1ff7ac901d162d929fb81e72f946ffd6c7fffb2c"
Nov 03 20:22:48.836685 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:48.836646    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-04bf5fc9-7f4947f948-qv8lm" podStartSLOduration=3.8366341029999997 podStartE2EDuration="3.836634103s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:22:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:22:48.835086606 +0000 UTC m=+1102.987748504" watchObservedRunningTime="2026-11-03 20:22:48.836634103 +0000 UTC m=+1102.989295990"
Nov 03 20:22:48.837094 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:22:48.837059    5570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9803342d64166210f8684fc1ff7ac901d162d929fb81e72f946ffd6c7fffb2c\": container with ID starting with b9803342d64166210f8684fc1ff7ac901d162d929fb81e72f946ffd6c7fffb2c not found: ID does not exist" containerID="b9803342d64166210f8684fc1ff7ac901d162d929fb81e72f946ffd6c7fffb2c"
Nov 03 20:22:48.837218 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:48.837108    5570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9803342d64166210f8684fc1ff7ac901d162d929fb81e72f946ffd6c7fffb2c"} err="failed to get container status \"b9803342d64166210f8684fc1ff7ac901d162d929fb81e72f946ffd6c7fffb2c\": rpc error: code = NotFound desc = could not find container \"b9803342d64166210f8684fc1ff7ac901d162d929fb81e72f946ffd6c7fffb2c\": container with ID starting with b9803342d64166210f8684fc1ff7ac901d162d929fb81e72f946ffd6c7fffb2c not found: ID does not exist"
Nov 03 20:22:48.837218 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:48.837144    5570 scope.go:122] "RemoveContainer" containerID="e3bed704031bf8816c6564d3549a7c19e2f619f6ff9c7ec362d1fda9e7309df2"
Nov 03 20:22:48.837975 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:22:48.837949    5570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3bed704031bf8816c6564d3549a7c19e2f619f6ff9c7ec362d1fda9e7309df2\": container with ID starting with e3bed704031bf8816c6564d3549a7c19e2f619f6ff9c7ec362d1fda9e7309df2 not found: ID does not exist" containerID="e3bed704031bf8816c6564d3549a7c19e2f619f6ff9c7ec362d1fda9e7309df2"
Nov 03 20:22:48.838043 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:48.837988    5570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3bed704031bf8816c6564d3549a7c19e2f619f6ff9c7ec362d1fda9e7309df2"} err="failed to get container status \"e3bed704031bf8816c6564d3549a7c19e2f619f6ff9c7ec362d1fda9e7309df2\": rpc error: code = NotFound desc = could not find container \"e3bed704031bf8816c6564d3549a7c19e2f619f6ff9c7ec362d1fda9e7309df2\": container with ID starting with e3bed704031bf8816c6564d3549a7c19e2f619f6ff9c7ec362d1fda9e7309df2 not found: ID does not exist"
Nov 03 20:22:48.880547 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:48.880482    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-04bf5fc9-c7bd84cd7-p8qbk" podStartSLOduration=3.8804672460000003 podStartE2EDuration="3.880467246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:22:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:22:48.874962913 +0000 UTC m=+1103.027624802" watchObservedRunningTime="2026-11-03 20:22:48.880467246 +0000 UTC m=+1103.033129134"
Nov 03 20:22:48.917948 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:48.917918    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-9798ad8e-gw-openshift-default-575b7f8f9c-xg9nd"]
Nov 03 20:22:48.922145 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:48.922123    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-11a7d765-58f7bcf694-bwl46"
Nov 03 20:22:48.922366 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:48.922347    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-9798ad8e-gw-openshift-default-575b7f8f9c-xg9nd"]
Nov 03 20:22:49.038207 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:49.037328    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30787f8a-2875-48eb-b4d8-f4b2fa97d29d-kube-api-access-2tq75\" (UniqueName: \"kubernetes.io/projected/30787f8a-2875-48eb-b4d8-f4b2fa97d29d-kube-api-access-2tq75\") pod \"30787f8a-2875-48eb-b4d8-f4b2fa97d29d\" (UID: \"30787f8a-2875-48eb-b4d8-f4b2fa97d29d\") "
Nov 03 20:22:49.066585 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:49.049306    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30787f8a-2875-48eb-b4d8-f4b2fa97d29d-kube-api-access-2tq75" pod "30787f8a-2875-48eb-b4d8-f4b2fa97d29d" (UID: "30787f8a-2875-48eb-b4d8-f4b2fa97d29d"). InnerVolumeSpecName "kube-api-access-2tq75". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:22:49.066585 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:49.052741    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71203a3a-hx4zg"]
Nov 03 20:22:49.066585 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:49.058046    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30787f8a-2875-48eb-b4d8-f4b2fa97d29d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30787f8a-2875-48eb-b4d8-f4b2fa97d29d-plugins-config-volume\") pod \"30787f8a-2875-48eb-b4d8-f4b2fa97d29d\" (UID: \"30787f8a-2875-48eb-b4d8-f4b2fa97d29d\") "
Nov 03 20:22:49.066585 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:49.058999    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2tq75\" (UniqueName: \"kubernetes.io/projected/30787f8a-2875-48eb-b4d8-f4b2fa97d29d-kube-api-access-2tq75\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:22:49.066585 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:49.061428    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30787f8a-2875-48eb-b4d8-f4b2fa97d29d-plugins-config-volume" pod "30787f8a-2875-48eb-b4d8-f4b2fa97d29d" (UID: "30787f8a-2875-48eb-b4d8-f4b2fa97d29d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:22:49.066585 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:49.064583    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71203a3a-hx4zg"]
Nov 03 20:22:49.161341 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:49.161286    5570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30787f8a-2875-48eb-b4d8-f4b2fa97d29d-plugins-config-volume\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:22:49.248070 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:49.248040    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-a619f336-openshift-default-66dcdd6969-h9lgf"]
Nov 03 20:22:49.248826 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:49.248809    5570 cpu_manager.go:414] "containerMap: removing container" podUID="b9bd3e2e-35ec-482d-8dc2-1a18f255c220" containerName="istio-proxy"
Nov 03 20:22:49.248889 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:49.248827    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9bd3e2e-35ec-482d-8dc2-1a18f255c220" containerName="istio-proxy"
Nov 03 20:22:49.248889 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:49.248844    5570 cpu_manager.go:414] "containerMap: removing container" podUID="b9bd3e2e-35ec-482d-8dc2-1a18f255c220" containerName="istio-proxy"
Nov 03 20:22:49.248889 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:49.248849    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9bd3e2e-35ec-482d-8dc2-1a18f255c220" containerName="istio-proxy"
Nov 03 20:22:49.248889 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:49.248874    5570 cpu_manager.go:414] "containerMap: removing container" podUID="30787f8a-2875-48eb-b4d8-f4b2fa97d29d" containerName="payload-processing"
Nov 03 20:22:49.248889 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:49.248879    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30787f8a-2875-48eb-b4d8-f4b2fa97d29d" containerName="payload-processing"
Nov 03 20:22:49.249074 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:49.249018    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="30787f8a-2875-48eb-b4d8-f4b2fa97d29d" containerName="payload-processing"
Nov 03 20:22:49.249074 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:49.249041    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9bd3e2e-35ec-482d-8dc2-1a18f255c220" containerName="istio-proxy"
Nov 03 20:22:49.249074 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:49.249053    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9bd3e2e-35ec-482d-8dc2-1a18f255c220" containerName="istio-proxy"
Nov 03 20:22:49.252163 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:49.252141    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-a619f336-openshift-default-66dcdd6969-h9lgf"
Nov 03 20:22:49.270383 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:49.268625    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-a619f336-openshift-default-dockercfg-q8qqd\""
Nov 03 20:22:49.278180 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:49.278158    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-a619f336-openshift-default-66dcdd6969-h9lgf"]
Nov 03 20:22:49.381854 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:49.371959    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f3e4187-d5b7-4ef6-8301-2b838530c525-istio-token\") pod \"e2e-derive-a619f336-openshift-default-66dcdd6969-h9lgf\" (UID: \"3f3e4187-d5b7-4ef6-8301-2b838530c525\") " pod="openshift-ingress/e2e-derive-a619f336-openshift-default-66dcdd6969-h9lgf"
Nov 03 20:22:49.381854 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:49.372012    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f3e4187-d5b7-4ef6-8301-2b838530c525-istio-podinfo\") pod \"e2e-derive-a619f336-openshift-default-66dcdd6969-h9lgf\" (UID: \"3f3e4187-d5b7-4ef6-8301-2b838530c525\") " pod="openshift-ingress/e2e-derive-a619f336-openshift-default-66dcdd6969-h9lgf"
Nov 03 20:22:49.381854 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:49.372146    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f3e4187-d5b7-4ef6-8301-2b838530c525-workload-socket\") pod \"e2e-derive-a619f336-openshift-default-66dcdd6969-h9lgf\" (UID: \"3f3e4187-d5b7-4ef6-8301-2b838530c525\") " pod="openshift-ingress/e2e-derive-a619f336-openshift-default-66dcdd6969-h9lgf"
Nov 03 20:22:49.381854 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:49.372202    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f3e4187-d5b7-4ef6-8301-2b838530c525-istiod-ca-cert\") pod \"e2e-derive-a619f336-openshift-default-66dcdd6969-h9lgf\" (UID: \"3f3e4187-d5b7-4ef6-8301-2b838530c525\") " pod="openshift-ingress/e2e-derive-a619f336-openshift-default-66dcdd6969-h9lgf"
Nov 03 20:22:49.381854 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:49.372378    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f3e4187-d5b7-4ef6-8301-2b838530c525-workload-certs\") pod \"e2e-derive-a619f336-openshift-default-66dcdd6969-h9lgf\" (UID: \"3f3e4187-d5b7-4ef6-8301-2b838530c525\") " pod="openshift-ingress/e2e-derive-a619f336-openshift-default-66dcdd6969-h9lgf"
Nov 03 20:22:49.381854 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:49.372506    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f3e4187-d5b7-4ef6-8301-2b838530c525-credential-socket\") pod \"e2e-derive-a619f336-openshift-default-66dcdd6969-h9lgf\" (UID: \"3f3e4187-d5b7-4ef6-8301-2b838530c525\") " pod="openshift-ingress/e2e-derive-a619f336-openshift-default-66dcdd6969-h9lgf"
Nov 03 20:22:49.381854 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:49.372578    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqp49\" (UniqueName: \"kubernetes.io/projected/3f3e4187-d5b7-4ef6-8301-2b838530c525-kube-api-access-pqp49\") pod \"e2e-derive-a619f336-openshift-default-66dcdd6969-h9lgf\" (UID: \"3f3e4187-d5b7-4ef6-8301-2b838530c525\") " pod="openshift-ingress/e2e-derive-a619f336-openshift-default-66dcdd6969-h9lgf"
Nov 03 20:22:49.381854 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:49.372616    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f3e4187-d5b7-4ef6-8301-2b838530c525-istio-data\") pod \"e2e-derive-a619f336-openshift-default-66dcdd6969-h9lgf\" (UID: \"3f3e4187-d5b7-4ef6-8301-2b838530c525\") " pod="openshift-ingress/e2e-derive-a619f336-openshift-default-66dcdd6969-h9lgf"
Nov 03 20:22:49.381854 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:49.372722    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f3e4187-d5b7-4ef6-8301-2b838530c525-istio-envoy\") pod \"e2e-derive-a619f336-openshift-default-66dcdd6969-h9lgf\" (UID: \"3f3e4187-d5b7-4ef6-8301-2b838530c525\") " pod="openshift-ingress/e2e-derive-a619f336-openshift-default-66dcdd6969-h9lgf"
Nov 03 20:22:49.678860 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:49.677212    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqp49\" (UniqueName: \"kubernetes.io/projected/3f3e4187-d5b7-4ef6-8301-2b838530c525-kube-api-access-pqp49\") pod \"e2e-derive-a619f336-openshift-default-66dcdd6969-h9lgf\" (UID: \"3f3e4187-d5b7-4ef6-8301-2b838530c525\") " pod="openshift-ingress/e2e-derive-a619f336-openshift-default-66dcdd6969-h9lgf"
Nov 03 20:22:49.678860 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:49.677589    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f3e4187-d5b7-4ef6-8301-2b838530c525-istio-data\") pod \"e2e-derive-a619f336-openshift-default-66dcdd6969-h9lgf\" (UID: \"3f3e4187-d5b7-4ef6-8301-2b838530c525\") " pod="openshift-ingress/e2e-derive-a619f336-openshift-default-66dcdd6969-h9lgf"
Nov 03 20:22:49.678860 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:49.677807    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f3e4187-d5b7-4ef6-8301-2b838530c525-istio-envoy\") pod \"e2e-derive-a619f336-openshift-default-66dcdd6969-h9lgf\" (UID: \"3f3e4187-d5b7-4ef6-8301-2b838530c525\") " pod="openshift-ingress/e2e-derive-a619f336-openshift-default-66dcdd6969-h9lgf"
Nov 03 20:22:49.678860 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:49.678219    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f3e4187-d5b7-4ef6-8301-2b838530c525-istio-token\") pod \"e2e-derive-a619f336-openshift-default-66dcdd6969-h9lgf\" (UID: \"3f3e4187-d5b7-4ef6-8301-2b838530c525\") " pod="openshift-ingress/e2e-derive-a619f336-openshift-default-66dcdd6969-h9lgf"
Nov 03 20:22:49.678860 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:49.678587    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f3e4187-d5b7-4ef6-8301-2b838530c525-istio-podinfo\") pod \"e2e-derive-a619f336-openshift-default-66dcdd6969-h9lgf\" (UID: \"3f3e4187-d5b7-4ef6-8301-2b838530c525\") " pod="openshift-ingress/e2e-derive-a619f336-openshift-default-66dcdd6969-h9lgf"
Nov 03 20:22:49.678860 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:49.678707    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f3e4187-d5b7-4ef6-8301-2b838530c525-workload-socket\") pod \"e2e-derive-a619f336-openshift-default-66dcdd6969-h9lgf\" (UID: \"3f3e4187-d5b7-4ef6-8301-2b838530c525\") " pod="openshift-ingress/e2e-derive-a619f336-openshift-default-66dcdd6969-h9lgf"
Nov 03 20:22:49.679308 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:49.679069    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f3e4187-d5b7-4ef6-8301-2b838530c525-istiod-ca-cert\") pod \"e2e-derive-a619f336-openshift-default-66dcdd6969-h9lgf\" (UID: \"3f3e4187-d5b7-4ef6-8301-2b838530c525\") " pod="openshift-ingress/e2e-derive-a619f336-openshift-default-66dcdd6969-h9lgf"
Nov 03 20:22:49.686829 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:49.679294    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f3e4187-d5b7-4ef6-8301-2b838530c525-workload-certs\") pod \"e2e-derive-a619f336-openshift-default-66dcdd6969-h9lgf\" (UID: \"3f3e4187-d5b7-4ef6-8301-2b838530c525\") " pod="openshift-ingress/e2e-derive-a619f336-openshift-default-66dcdd6969-h9lgf"
Nov 03 20:22:49.686829 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:49.679615    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f3e4187-d5b7-4ef6-8301-2b838530c525-credential-socket\") pod \"e2e-derive-a619f336-openshift-default-66dcdd6969-h9lgf\" (UID: \"3f3e4187-d5b7-4ef6-8301-2b838530c525\") " pod="openshift-ingress/e2e-derive-a619f336-openshift-default-66dcdd6969-h9lgf"
Nov 03 20:22:49.686829 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:49.682853    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f3e4187-d5b7-4ef6-8301-2b838530c525-credential-socket\") pod \"e2e-derive-a619f336-openshift-default-66dcdd6969-h9lgf\" (UID: \"3f3e4187-d5b7-4ef6-8301-2b838530c525\") " pod="openshift-ingress/e2e-derive-a619f336-openshift-default-66dcdd6969-h9lgf"
Nov 03 20:22:49.686829 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:49.683945    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f3e4187-d5b7-4ef6-8301-2b838530c525-istio-data\") pod \"e2e-derive-a619f336-openshift-default-66dcdd6969-h9lgf\" (UID: \"3f3e4187-d5b7-4ef6-8301-2b838530c525\") " pod="openshift-ingress/e2e-derive-a619f336-openshift-default-66dcdd6969-h9lgf"
Nov 03 20:22:49.686829 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:49.685176    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f3e4187-d5b7-4ef6-8301-2b838530c525-workload-socket\") pod \"e2e-derive-a619f336-openshift-default-66dcdd6969-h9lgf\" (UID: \"3f3e4187-d5b7-4ef6-8301-2b838530c525\") " pod="openshift-ingress/e2e-derive-a619f336-openshift-default-66dcdd6969-h9lgf"
Nov 03 20:22:49.699296 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:49.699262    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f3e4187-d5b7-4ef6-8301-2b838530c525-istiod-ca-cert\") pod \"e2e-derive-a619f336-openshift-default-66dcdd6969-h9lgf\" (UID: \"3f3e4187-d5b7-4ef6-8301-2b838530c525\") " pod="openshift-ingress/e2e-derive-a619f336-openshift-default-66dcdd6969-h9lgf"
Nov 03 20:22:49.704494 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:49.704267    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f3e4187-d5b7-4ef6-8301-2b838530c525-workload-certs\") pod \"e2e-derive-a619f336-openshift-default-66dcdd6969-h9lgf\" (UID: \"3f3e4187-d5b7-4ef6-8301-2b838530c525\") " pod="openshift-ingress/e2e-derive-a619f336-openshift-default-66dcdd6969-h9lgf"
Nov 03 20:22:49.707275 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:49.707247    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f3e4187-d5b7-4ef6-8301-2b838530c525-istio-podinfo\") pod \"e2e-derive-a619f336-openshift-default-66dcdd6969-h9lgf\" (UID: \"3f3e4187-d5b7-4ef6-8301-2b838530c525\") " pod="openshift-ingress/e2e-derive-a619f336-openshift-default-66dcdd6969-h9lgf"
Nov 03 20:22:49.711360 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:49.709088    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f3e4187-d5b7-4ef6-8301-2b838530c525-istio-token\") pod \"e2e-derive-a619f336-openshift-default-66dcdd6969-h9lgf\" (UID: \"3f3e4187-d5b7-4ef6-8301-2b838530c525\") " pod="openshift-ingress/e2e-derive-a619f336-openshift-default-66dcdd6969-h9lgf"
Nov 03 20:22:49.713131 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:49.711466    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqp49\" (UniqueName: \"kubernetes.io/projected/3f3e4187-d5b7-4ef6-8301-2b838530c525-kube-api-access-pqp49\") pod \"e2e-derive-a619f336-openshift-default-66dcdd6969-h9lgf\" (UID: \"3f3e4187-d5b7-4ef6-8301-2b838530c525\") " pod="openshift-ingress/e2e-derive-a619f336-openshift-default-66dcdd6969-h9lgf"
Nov 03 20:22:49.773797 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:49.770376    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f3e4187-d5b7-4ef6-8301-2b838530c525-istio-envoy\") pod \"e2e-derive-a619f336-openshift-default-66dcdd6969-h9lgf\" (UID: \"3f3e4187-d5b7-4ef6-8301-2b838530c525\") " pod="openshift-ingress/e2e-derive-a619f336-openshift-default-66dcdd6969-h9lgf"
Nov 03 20:22:49.856950 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:49.856919    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-11a7d765-58f7bcf694-bwl46"
Nov 03 20:22:49.857556 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:49.857524    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-11a7d765-58f7bcf694-bwl46" event={"ID":"30787f8a-2875-48eb-b4d8-f4b2fa97d29d","Type":"ContainerDied","Data":"a986609b60a4d20bf45299a99ee756331909d33d759410078ccc8507c045fe16"}
Nov 03 20:22:49.857672 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:49.857618    5570 scope.go:122] "RemoveContainer" containerID="d63387c45ba1613f69aef56cb0d7dda6b838c84ebc1fff7aea0a35fe56967590"
Nov 03 20:22:49.874274 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:49.874233    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-a619f336-openshift-default-66dcdd6969-h9lgf"
Nov 03 20:22:50.019618 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:50.019265    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9bd3e2e-35ec-482d-8dc2-1a18f255c220" path="/var/lib/kubelet/pods/b9bd3e2e-35ec-482d-8dc2-1a18f255c220/volumes"
Nov 03 20:22:50.020418 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:50.020390    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7f12100-03b3-44fe-ac21-e8504a730164" path="/var/lib/kubelet/pods/c7f12100-03b3-44fe-ac21-e8504a730164/volumes"
Nov 03 20:22:50.020922 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:50.020901    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-11a7d765-58f7bcf694-bwl46"]
Nov 03 20:22:50.021020 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:50.020946    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-11a7d765-58f7bcf694-bwl46"]
Nov 03 20:22:50.342187 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:22:50.342152    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f3e4187_d5b7_4ef6_8301_2b838530c525.slice/crio-58e37911baead6c9b9294bb90da463ea39e45ca68c8524cffc9f351c341f6ccc WatchSource:0}: Error finding container 58e37911baead6c9b9294bb90da463ea39e45ca68c8524cffc9f351c341f6ccc: Status 404 returned error can't find the container with id 58e37911baead6c9b9294bb90da463ea39e45ca68c8524cffc9f351c341f6ccc
Nov 03 20:22:50.342597 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:50.342572    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-a619f336-openshift-default-66dcdd6969-h9lgf"]
Nov 03 20:22:50.345430 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:50.345399    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:22:50.345506 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:50.345464    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:22:50.345506 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:50.345493    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:22:50.871732 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:50.871694    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-a619f336-openshift-default-66dcdd6969-h9lgf" event={"ID":"3f3e4187-d5b7-4ef6-8301-2b838530c525","Type":"ContainerStarted","Data":"138e37aa1ed4c1bafd9fc822ef0a425ae8d031b7f3f6c8492e5e0d36f8b8e8c5"}
Nov 03 20:22:50.872284 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:50.871758    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-a619f336-openshift-default-66dcdd6969-h9lgf" event={"ID":"3f3e4187-d5b7-4ef6-8301-2b838530c525","Type":"ContainerStarted","Data":"58e37911baead6c9b9294bb90da463ea39e45ca68c8524cffc9f351c341f6ccc"}
Nov 03 20:22:50.875327 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:50.875305    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-a619f336-openshift-default-66dcdd6969-h9lgf"
Nov 03 20:22:50.919310 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:50.919255    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-a619f336-openshift-default-66dcdd6969-h9lgf" podStartSLOduration=1.919238921 podStartE2EDuration="1.919238921s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:22:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:22:50.913666339 +0000 UTC m=+1105.066328225" watchObservedRunningTime="2026-11-03 20:22:50.919238921 +0000 UTC m=+1105.071900808"
Nov 03 20:22:51.881808 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:51.881742    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-a619f336-openshift-default-66dcdd6969-h9lgf"
Nov 03 20:22:51.883267 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:51.883235    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-a619f336-openshift-default-66dcdd6969-h9lgf"
Nov 03 20:22:51.883976 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:51.883959    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-a619f336-openshift-default-66dcdd6969-h9lgf"
Nov 03 20:22:51.967020 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:51.966987    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30787f8a-2875-48eb-b4d8-f4b2fa97d29d" path="/var/lib/kubelet/pods/30787f8a-2875-48eb-b4d8-f4b2fa97d29d/volumes"
Nov 03 20:22:53.820800 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:53.819661    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g69f8"]
Nov 03 20:22:57.493800 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:22:57.486579    5570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8bwhz" containerName="extractor"
Nov 03 20:22:58.122830 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:58.119408    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/premium-w4-e80203-kserve-5f9f445456-bms9d" podUID="6ad76efc-20c2-4951-b22a-c5a39cd39ed4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:22:59.324965 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:59.320730    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1b67df53-86998f8f74-df6c6"]
Nov 03 20:22:59.870309 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:22:59.870277    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-04bf5fc9-7f4947f948-qv8lm"
Nov 03 20:23:00.894018 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:00.893971    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-04bf5fc9-c7bd84cd7-p8qbk"
Nov 03 20:23:01.031237 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:01.031202    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca60b04a-rnj6h"]
Nov 03 20:23:01.035498 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:01.035473    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca60b04a-rnj6h"]
Nov 03 20:23:01.970811 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:01.970757    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10096163-2b34-48f3-95fa-cdd9798a9fdf" path="/var/lib/kubelet/pods/10096163-2b34-48f3-95fa-cdd9798a9fdf/volumes"
Nov 03 20:23:04.322614 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:04.322491    5570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-1b67df53-86998f8f74-df6c6" podUID="848f4854-6df4-4ab9-9baf-b0710fd158d2" containerName="payload-processing" containerID="cri-o://d32c0d456877777ed0f51aa4e5276f82ce45c53e91a7e19d5c916e8ebd319c09" gracePeriod=25
Nov 03 20:23:04.620657 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:04.606897    5570 generic.go:356] "Generic (PLEG): container finished" podID="848f4854-6df4-4ab9-9baf-b0710fd158d2" containerID="d32c0d456877777ed0f51aa4e5276f82ce45c53e91a7e19d5c916e8ebd319c09" exitCode=0
Nov 03 20:23:04.620657 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:04.607181    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-1b67df53-86998f8f74-df6c6" event={"ID":"848f4854-6df4-4ab9-9baf-b0710fd158d2","Type":"ContainerDied","Data":"d32c0d456877777ed0f51aa4e5276f82ce45c53e91a7e19d5c916e8ebd319c09"}
Nov 03 20:23:05.114460 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:05.114417    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-149c62d4-x6xrj"]
Nov 03 20:23:05.119791 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:05.119728    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-149c62d4-x6xrj"]
Nov 03 20:23:05.292117 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:05.292088    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-1b67df53-86998f8f74-df6c6"
Nov 03 20:23:05.418522 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:05.418486    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/848f4854-6df4-4ab9-9baf-b0710fd158d2-kube-api-access-vwwd5\" (UniqueName: \"kubernetes.io/projected/848f4854-6df4-4ab9-9baf-b0710fd158d2-kube-api-access-vwwd5\") pod \"848f4854-6df4-4ab9-9baf-b0710fd158d2\" (UID: \"848f4854-6df4-4ab9-9baf-b0710fd158d2\") "
Nov 03 20:23:05.419029 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:05.418561    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/848f4854-6df4-4ab9-9baf-b0710fd158d2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/848f4854-6df4-4ab9-9baf-b0710fd158d2-plugins-config-volume\") pod \"848f4854-6df4-4ab9-9baf-b0710fd158d2\" (UID: \"848f4854-6df4-4ab9-9baf-b0710fd158d2\") "
Nov 03 20:23:05.419092 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:05.419022    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/848f4854-6df4-4ab9-9baf-b0710fd158d2-plugins-config-volume" pod "848f4854-6df4-4ab9-9baf-b0710fd158d2" (UID: "848f4854-6df4-4ab9-9baf-b0710fd158d2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:23:05.419634 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:05.419612    5570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/848f4854-6df4-4ab9-9baf-b0710fd158d2-plugins-config-volume\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:23:05.420880 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:05.420852    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/848f4854-6df4-4ab9-9baf-b0710fd158d2-kube-api-access-vwwd5" pod "848f4854-6df4-4ab9-9baf-b0710fd158d2" (UID: "848f4854-6df4-4ab9-9baf-b0710fd158d2"). InnerVolumeSpecName "kube-api-access-vwwd5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:23:05.521839 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:05.521729    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwwd5\" (UniqueName: \"kubernetes.io/projected/848f4854-6df4-4ab9-9baf-b0710fd158d2-kube-api-access-vwwd5\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:23:05.617057 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:05.617031    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-1b67df53-86998f8f74-df6c6"
Nov 03 20:23:05.617057 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:05.617047    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-1b67df53-86998f8f74-df6c6" event={"ID":"848f4854-6df4-4ab9-9baf-b0710fd158d2","Type":"ContainerDied","Data":"5e7f857b71665638c9d9b15d407fcad7e0942c9d08bc859f8d1a54e63f0136c1"}
Nov 03 20:23:05.617299 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:05.617105    5570 scope.go:122] "RemoveContainer" containerID="d32c0d456877777ed0f51aa4e5276f82ce45c53e91a7e19d5c916e8ebd319c09"
Nov 03 20:23:05.673290 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:05.673259    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1b67df53-86998f8f74-df6c6"]
Nov 03 20:23:05.677345 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:05.677322    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1b67df53-86998f8f74-df6c6"]
Nov 03 20:23:05.968288 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:05.968253    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08701155-f481-4997-b146-ebc997e838c5" path="/var/lib/kubelet/pods/08701155-f481-4997-b146-ebc997e838c5/volumes"
Nov 03 20:23:05.968825 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:05.968810    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="848f4854-6df4-4ab9-9baf-b0710fd158d2" path="/var/lib/kubelet/pods/848f4854-6df4-4ab9-9baf-b0710fd158d2/volumes"
Nov 03 20:23:08.256037 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:08.255994    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xhr9f"]
Nov 03 20:23:08.257876 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:08.257843    5570 cpu_manager.go:414] "containerMap: removing container" podUID="848f4854-6df4-4ab9-9baf-b0710fd158d2" containerName="payload-processing"
Nov 03 20:23:08.258093 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:08.258074    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="848f4854-6df4-4ab9-9baf-b0710fd158d2" containerName="payload-processing"
Nov 03 20:23:08.346745 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:08.346703    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="848f4854-6df4-4ab9-9baf-b0710fd158d2" containerName="payload-processing"
Nov 03 20:23:08.353163 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:08.353116    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xhr9f"]
Nov 03 20:23:08.353568 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:08.353547    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xhr9f"
Nov 03 20:23:08.512203 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:08.512112    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzx8m\" (UniqueName: \"kubernetes.io/projected/5668dee7-8a03-4a0c-ab66-303fa052ea77-kube-api-access-xzx8m\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xhr9f\" (UID: \"5668dee7-8a03-4a0c-ab66-303fa052ea77\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xhr9f"
Nov 03 20:23:08.512203 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:08.512158    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5668dee7-8a03-4a0c-ab66-303fa052ea77-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xhr9f\" (UID: \"5668dee7-8a03-4a0c-ab66-303fa052ea77\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xhr9f"
Nov 03 20:23:08.512399 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:08.512335    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5668dee7-8a03-4a0c-ab66-303fa052ea77-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xhr9f\" (UID: \"5668dee7-8a03-4a0c-ab66-303fa052ea77\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xhr9f"
Nov 03 20:23:08.512455 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:08.512439    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5668dee7-8a03-4a0c-ab66-303fa052ea77-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xhr9f\" (UID: \"5668dee7-8a03-4a0c-ab66-303fa052ea77\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xhr9f"
Nov 03 20:23:08.512626 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:08.512613    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5668dee7-8a03-4a0c-ab66-303fa052ea77-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xhr9f\" (UID: \"5668dee7-8a03-4a0c-ab66-303fa052ea77\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xhr9f"
Nov 03 20:23:08.512664 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:08.512638    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5668dee7-8a03-4a0c-ab66-303fa052ea77-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xhr9f\" (UID: \"5668dee7-8a03-4a0c-ab66-303fa052ea77\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xhr9f"
Nov 03 20:23:08.512717 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:08.512679    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5668dee7-8a03-4a0c-ab66-303fa052ea77-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xhr9f\" (UID: \"5668dee7-8a03-4a0c-ab66-303fa052ea77\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xhr9f"
Nov 03 20:23:08.614829 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:08.614789    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5668dee7-8a03-4a0c-ab66-303fa052ea77-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xhr9f\" (UID: \"5668dee7-8a03-4a0c-ab66-303fa052ea77\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xhr9f"
Nov 03 20:23:08.614829 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:08.614827    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5668dee7-8a03-4a0c-ab66-303fa052ea77-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xhr9f\" (UID: \"5668dee7-8a03-4a0c-ab66-303fa052ea77\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xhr9f"
Nov 03 20:23:08.615109 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:08.614885    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5668dee7-8a03-4a0c-ab66-303fa052ea77-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xhr9f\" (UID: \"5668dee7-8a03-4a0c-ab66-303fa052ea77\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xhr9f"
Nov 03 20:23:08.615109 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:08.614955    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzx8m\" (UniqueName: \"kubernetes.io/projected/5668dee7-8a03-4a0c-ab66-303fa052ea77-kube-api-access-xzx8m\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xhr9f\" (UID: \"5668dee7-8a03-4a0c-ab66-303fa052ea77\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xhr9f"
Nov 03 20:23:08.615109 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:08.614977    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5668dee7-8a03-4a0c-ab66-303fa052ea77-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xhr9f\" (UID: \"5668dee7-8a03-4a0c-ab66-303fa052ea77\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xhr9f"
Nov 03 20:23:08.615109 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:08.615052    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5668dee7-8a03-4a0c-ab66-303fa052ea77-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xhr9f\" (UID: \"5668dee7-8a03-4a0c-ab66-303fa052ea77\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xhr9f"
Nov 03 20:23:08.615328 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:08.615140    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5668dee7-8a03-4a0c-ab66-303fa052ea77-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xhr9f\" (UID: \"5668dee7-8a03-4a0c-ab66-303fa052ea77\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xhr9f"
Nov 03 20:23:08.616023 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:08.615992    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5668dee7-8a03-4a0c-ab66-303fa052ea77-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xhr9f\" (UID: \"5668dee7-8a03-4a0c-ab66-303fa052ea77\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xhr9f"
Nov 03 20:23:08.617482 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:08.617458    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5668dee7-8a03-4a0c-ab66-303fa052ea77-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xhr9f\" (UID: \"5668dee7-8a03-4a0c-ab66-303fa052ea77\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xhr9f"
Nov 03 20:23:08.617581 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:08.617518    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5668dee7-8a03-4a0c-ab66-303fa052ea77-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xhr9f\" (UID: \"5668dee7-8a03-4a0c-ab66-303fa052ea77\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xhr9f"
Nov 03 20:23:08.617668 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:08.617647    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5668dee7-8a03-4a0c-ab66-303fa052ea77-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xhr9f\" (UID: \"5668dee7-8a03-4a0c-ab66-303fa052ea77\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xhr9f"
Nov 03 20:23:08.617726 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:08.617671    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5668dee7-8a03-4a0c-ab66-303fa052ea77-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xhr9f\" (UID: \"5668dee7-8a03-4a0c-ab66-303fa052ea77\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xhr9f"
Nov 03 20:23:08.635195 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:08.635167    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5668dee7-8a03-4a0c-ab66-303fa052ea77-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xhr9f\" (UID: \"5668dee7-8a03-4a0c-ab66-303fa052ea77\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xhr9f"
Nov 03 20:23:08.635509 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:08.635481    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzx8m\" (UniqueName: \"kubernetes.io/projected/5668dee7-8a03-4a0c-ab66-303fa052ea77-kube-api-access-xzx8m\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xhr9f\" (UID: \"5668dee7-8a03-4a0c-ab66-303fa052ea77\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xhr9f"
Nov 03 20:23:08.690797 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:08.690579    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xhr9f"
Nov 03 20:23:09.387383 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:09.382321    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xhr9f"]
Nov 03 20:23:09.388626 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:09.388577    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:23:09.388764 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:09.388707    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:23:09.920801 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:09.918653    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xhr9f" event={"ID":"5668dee7-8a03-4a0c-ab66-303fa052ea77","Type":"ContainerStarted","Data":"1f19f2144b102cc1956626a8bb852b8709eee1ead55e6bac6ab748e23cc1fe79"}
Nov 03 20:23:09.920801 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:09.918911    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xhr9f" event={"ID":"5668dee7-8a03-4a0c-ab66-303fa052ea77","Type":"ContainerStarted","Data":"418742f11838292768a3113bd777cd3ca270581e010416f91c64e86e9fa8a6ba"}
Nov 03 20:23:09.920801 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:09.919146    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xhr9f"
Nov 03 20:23:10.026795 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:10.023137    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xhr9f" podStartSLOduration=2.023115564 podStartE2EDuration="2.023115564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:23:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:23:10.007988055 +0000 UTC m=+1124.160649944" watchObservedRunningTime="2026-11-03 20:23:10.023115564 +0000 UTC m=+1124.175777452"
Nov 03 20:23:10.984097 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:10.983711    5570 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-xhr9f container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 20:23:10.984097 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:10.983767    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xhr9f" podUID="5668dee7-8a03-4a0c-ab66-303fa052ea77" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 20:23:13.982383 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:13.982354    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xhr9f"
Nov 03 20:23:16.028008 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:16.027972    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2e12a2f-4h7wg"]
Nov 03 20:23:16.040333 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:16.040305    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2e12a2f-4h7wg"]
Nov 03 20:23:17.966712 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:17.966672    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2513f556-dddd-489a-b544-08dc7884aec3" path="/var/lib/kubelet/pods/2513f556-dddd-489a-b544-08dc7884aec3/volumes"
Nov 03 20:23:18.030191 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:18.030163    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5dd10855-b4dqz"]
Nov 03 20:23:18.043416 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:18.043384    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5dd10855-b4dqz"]
Nov 03 20:23:20.026797 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:20.025117    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3137dfc5-1780-4326-a6d5-469e1e780699" path="/var/lib/kubelet/pods/3137dfc5-1780-4326-a6d5-469e1e780699/volumes"
Nov 03 20:23:27.231063 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:23:27.231021    5570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8bwhz" containerName="extractor"
Nov 03 20:23:31.055229 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:31.055144    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-527bbe8c-9ktsh"]
Nov 03 20:23:31.064879 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:31.064825    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-527bbe8c-9ktsh"]
Nov 03 20:23:32.081406 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:32.081364    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22541c4e-43ae-4075-ab06-365e275ba881" path="/var/lib/kubelet/pods/22541c4e-43ae-4075-ab06-365e275ba881/volumes"
Nov 03 20:23:35.569366 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:35.569330    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-04bf5fc9-7f4947f948-qv8lm"]
Nov 03 20:23:35.626911 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:35.626877    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-04bf5fc9-c7bd84cd7-p8qbk"]
Nov 03 20:23:40.569995 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:40.569930    5570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-04bf5fc9-7f4947f948-qv8lm" podUID="61d0dabb-f778-4b31-a2c1-99b7d06de1b8" containerName="payload-processing" containerID="cri-o://89eb1f9fe3ebba1fc9162ab74a792d20ed26b277961eda2903b11584958447e0" gracePeriod=25
Nov 03 20:23:40.628017 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:40.627933    5570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-04bf5fc9-c7bd84cd7-p8qbk" podUID="8feccf11-f12a-4d1c-a5d0-b210351dca24" containerName="payload-pre-processing" containerID="cri-o://6a97c77e38d7534c04e946915e07b20e05124dcf60bcb203042b000f68308135" gracePeriod=25
Nov 03 20:23:40.766664 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:40.766356    5570 generic.go:356] "Generic (PLEG): container finished" podID="8feccf11-f12a-4d1c-a5d0-b210351dca24" containerID="6a97c77e38d7534c04e946915e07b20e05124dcf60bcb203042b000f68308135" exitCode=0
Nov 03 20:23:40.767067 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:40.767011    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-04bf5fc9-c7bd84cd7-p8qbk" event={"ID":"8feccf11-f12a-4d1c-a5d0-b210351dca24","Type":"ContainerDied","Data":"6a97c77e38d7534c04e946915e07b20e05124dcf60bcb203042b000f68308135"}
Nov 03 20:23:40.769547 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:40.769522    5570 generic.go:356] "Generic (PLEG): container finished" podID="61d0dabb-f778-4b31-a2c1-99b7d06de1b8" containerID="89eb1f9fe3ebba1fc9162ab74a792d20ed26b277961eda2903b11584958447e0" exitCode=0
Nov 03 20:23:40.769678 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:40.769603    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-04bf5fc9-7f4947f948-qv8lm" event={"ID":"61d0dabb-f778-4b31-a2c1-99b7d06de1b8","Type":"ContainerDied","Data":"89eb1f9fe3ebba1fc9162ab74a792d20ed26b277961eda2903b11584958447e0"}
Nov 03 20:23:40.873888 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:23:40.873750    5570 logging.go:55] [core] [Channel #356 SubChannel #357]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.104:50052", ServerName: "10.134.0.104:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.104:50052: connect: connection refused"
Nov 03 20:23:41.387800 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:41.385735    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-04bf5fc9-7f4947f948-qv8lm"
Nov 03 20:23:41.450722 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:41.450693    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-04bf5fc9-c7bd84cd7-p8qbk"
Nov 03 20:23:41.510940 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:41.510907    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61d0dabb-f778-4b31-a2c1-99b7d06de1b8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61d0dabb-f778-4b31-a2c1-99b7d06de1b8-plugins-config-volume\") pod \"61d0dabb-f778-4b31-a2c1-99b7d06de1b8\" (UID: \"61d0dabb-f778-4b31-a2c1-99b7d06de1b8\") "
Nov 03 20:23:41.511129 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:41.511114    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8feccf11-f12a-4d1c-a5d0-b210351dca24-kube-api-access-qlg5f\" (UniqueName: \"kubernetes.io/projected/8feccf11-f12a-4d1c-a5d0-b210351dca24-kube-api-access-qlg5f\") pod \"8feccf11-f12a-4d1c-a5d0-b210351dca24\" (UID: \"8feccf11-f12a-4d1c-a5d0-b210351dca24\") "
Nov 03 20:23:41.511254 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:41.511239    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8feccf11-f12a-4d1c-a5d0-b210351dca24-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8feccf11-f12a-4d1c-a5d0-b210351dca24-plugins-config-volume\") pod \"8feccf11-f12a-4d1c-a5d0-b210351dca24\" (UID: \"8feccf11-f12a-4d1c-a5d0-b210351dca24\") "
Nov 03 20:23:41.511340 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:41.511325    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61d0dabb-f778-4b31-a2c1-99b7d06de1b8-kube-api-access-htgsg\" (UniqueName: \"kubernetes.io/projected/61d0dabb-f778-4b31-a2c1-99b7d06de1b8-kube-api-access-htgsg\") pod \"61d0dabb-f778-4b31-a2c1-99b7d06de1b8\" (UID: \"61d0dabb-f778-4b31-a2c1-99b7d06de1b8\") "
Nov 03 20:23:41.511395 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:41.511367    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61d0dabb-f778-4b31-a2c1-99b7d06de1b8-plugins-config-volume" pod "61d0dabb-f778-4b31-a2c1-99b7d06de1b8" (UID: "61d0dabb-f778-4b31-a2c1-99b7d06de1b8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:23:41.511708 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:41.511677    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8feccf11-f12a-4d1c-a5d0-b210351dca24-plugins-config-volume" pod "8feccf11-f12a-4d1c-a5d0-b210351dca24" (UID: "8feccf11-f12a-4d1c-a5d0-b210351dca24"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:23:41.512340 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:41.512317    5570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8feccf11-f12a-4d1c-a5d0-b210351dca24-plugins-config-volume\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:23:41.512413 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:41.512349    5570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61d0dabb-f778-4b31-a2c1-99b7d06de1b8-plugins-config-volume\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:23:41.513415 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:41.513392    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8feccf11-f12a-4d1c-a5d0-b210351dca24-kube-api-access-qlg5f" pod "8feccf11-f12a-4d1c-a5d0-b210351dca24" (UID: "8feccf11-f12a-4d1c-a5d0-b210351dca24"). InnerVolumeSpecName "kube-api-access-qlg5f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:23:41.513764 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:41.513745    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61d0dabb-f778-4b31-a2c1-99b7d06de1b8-kube-api-access-htgsg" pod "61d0dabb-f778-4b31-a2c1-99b7d06de1b8" (UID: "61d0dabb-f778-4b31-a2c1-99b7d06de1b8"). InnerVolumeSpecName "kube-api-access-htgsg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:23:41.614461 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:41.614426    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qlg5f\" (UniqueName: \"kubernetes.io/projected/8feccf11-f12a-4d1c-a5d0-b210351dca24-kube-api-access-qlg5f\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:23:41.614461 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:41.614458    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-htgsg\" (UniqueName: \"kubernetes.io/projected/61d0dabb-f778-4b31-a2c1-99b7d06de1b8-kube-api-access-htgsg\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:23:41.794300 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:41.794240    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-04bf5fc9-c7bd84cd7-p8qbk" event={"ID":"8feccf11-f12a-4d1c-a5d0-b210351dca24","Type":"ContainerDied","Data":"274f65a436631237f2d0fcfe5a258f51449275dd7ad87bc7626f6ff18f1812d0"}
Nov 03 20:23:41.794574 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:41.794555    5570 scope.go:122] "RemoveContainer" containerID="6a97c77e38d7534c04e946915e07b20e05124dcf60bcb203042b000f68308135"
Nov 03 20:23:41.794930 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:41.794898    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-04bf5fc9-c7bd84cd7-p8qbk"
Nov 03 20:23:41.819861 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:41.807756    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-04bf5fc9-7f4947f948-qv8lm" event={"ID":"61d0dabb-f778-4b31-a2c1-99b7d06de1b8","Type":"ContainerDied","Data":"eeac0e0765fb57d0cef27da93c77deca15e897031f09a7aa956d22e13b2f926a"}
Nov 03 20:23:41.819861 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:41.807956    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-04bf5fc9-7f4947f948-qv8lm"
Nov 03 20:23:41.847197 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:41.847175    5570 scope.go:122] "RemoveContainer" containerID="89eb1f9fe3ebba1fc9162ab74a792d20ed26b277961eda2903b11584958447e0"
Nov 03 20:23:41.873983 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:41.873949    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-ait-04bf5fc9-c7bd84cd7-p8qbk" podUID="8feccf11-f12a-4d1c-a5d0-b210351dca24" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.104:50052\" within 1s: context deadline exceeded"
Nov 03 20:23:41.880100 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:41.880071    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-04bf5fc9-c7bd84cd7-p8qbk"]
Nov 03 20:23:41.884933 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:41.884908    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-04bf5fc9-c7bd84cd7-p8qbk"]
Nov 03 20:23:41.918928 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:41.918897    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-04bf5fc9-7f4947f948-qv8lm"]
Nov 03 20:23:41.925968 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:41.925938    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-04bf5fc9-7f4947f948-qv8lm"]
Nov 03 20:23:41.967844 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:41.967810    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61d0dabb-f778-4b31-a2c1-99b7d06de1b8" path="/var/lib/kubelet/pods/61d0dabb-f778-4b31-a2c1-99b7d06de1b8/volumes"
Nov 03 20:23:41.968318 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:41.968303    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8feccf11-f12a-4d1c-a5d0-b210351dca24" path="/var/lib/kubelet/pods/8feccf11-f12a-4d1c-a5d0-b210351dca24/volumes"
Nov 03 20:23:46.586949 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:46.508454    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-de82c8-78dc548fc4-24g87"]
Nov 03 20:23:46.586949 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:46.509635    5570 cpu_manager.go:414] "containerMap: removing container" podUID="8feccf11-f12a-4d1c-a5d0-b210351dca24" containerName="payload-pre-processing"
Nov 03 20:23:46.586949 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:46.509660    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8feccf11-f12a-4d1c-a5d0-b210351dca24" containerName="payload-pre-processing"
Nov 03 20:23:46.586949 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:46.509694    5570 cpu_manager.go:414] "containerMap: removing container" podUID="61d0dabb-f778-4b31-a2c1-99b7d06de1b8" containerName="payload-processing"
Nov 03 20:23:46.586949 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:46.509702    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61d0dabb-f778-4b31-a2c1-99b7d06de1b8" containerName="payload-processing"
Nov 03 20:23:46.586949 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:46.509879    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="8feccf11-f12a-4d1c-a5d0-b210351dca24" containerName="payload-pre-processing"
Nov 03 20:23:46.586949 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:46.509898    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="61d0dabb-f778-4b31-a2c1-99b7d06de1b8" containerName="payload-processing"
Nov 03 20:23:46.830864 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:46.755073    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-de82c8-78dc548fc4-24g87"
Nov 03 20:23:46.905866 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:46.900391    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w5-de82c8-dockercfg-qxt7k\""
Nov 03 20:23:46.905866 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:46.901402    5570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w5-de82c8\""
Nov 03 20:23:47.107854 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:47.043927    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-de82c8-78dc548fc4-24g87"]
Nov 03 20:23:47.107854 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:47.080330    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhp6s\" (UniqueName: \"kubernetes.io/projected/ec1dc0b9-d778-48b7-986a-0d460bb6a52f-kube-api-access-hhp6s\") pod \"payload-processing-e2e-shared-a-w5-de82c8-78dc548fc4-24g87\" (UID: \"ec1dc0b9-d778-48b7-986a-0d460bb6a52f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-de82c8-78dc548fc4-24g87"
Nov 03 20:23:47.107854 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:47.080835    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec1dc0b9-d778-48b7-986a-0d460bb6a52f-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-de82c8-78dc548fc4-24g87\" (UID: \"ec1dc0b9-d778-48b7-986a-0d460bb6a52f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-de82c8-78dc548fc4-24g87"
Nov 03 20:23:47.266799 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:47.266015    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhp6s\" (UniqueName: \"kubernetes.io/projected/ec1dc0b9-d778-48b7-986a-0d460bb6a52f-kube-api-access-hhp6s\") pod \"payload-processing-e2e-shared-a-w5-de82c8-78dc548fc4-24g87\" (UID: \"ec1dc0b9-d778-48b7-986a-0d460bb6a52f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-de82c8-78dc548fc4-24g87"
Nov 03 20:23:47.266799 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:47.266297    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec1dc0b9-d778-48b7-986a-0d460bb6a52f-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-de82c8-78dc548fc4-24g87\" (UID: \"ec1dc0b9-d778-48b7-986a-0d460bb6a52f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-de82c8-78dc548fc4-24g87"
Nov 03 20:23:47.295798 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:47.290963    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec1dc0b9-d778-48b7-986a-0d460bb6a52f-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-de82c8-78dc548fc4-24g87\" (UID: \"ec1dc0b9-d778-48b7-986a-0d460bb6a52f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-de82c8-78dc548fc4-24g87"
Nov 03 20:23:47.415797 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:47.415052    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/distinct-2-w4-e80203-kserve-756f9b9bc6-69gxj" podUID="ccf5dc2f-f116-4917-bb1a-b136d8ed17f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:23:47.434799 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:47.418637    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/simulator-w0-e40816-kserve-688f79796f-2tvw5" podUID="c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Nov 03 20:23:47.460802 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:47.436862    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/premium-w3-482722-kserve-689dc7bd5d-x8rjp" podUID="369045c3-4d6a-49a7-9bc6-f2939883bee8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Nov 03 20:23:47.623871 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:47.615988    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/premium-w2-9973d9-kserve-5c6dd4669c-5gv8x" podUID="cefc89ad-7ae1-433b-979d-6601678a6d98" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:23:47.635343 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:47.633810    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-2-w6-9e1bcb-kserve-7d96d9f6f6-4f7fl" podUID="56353b0f-08ee-4ada-9dc0-ac5afd5a59c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:23:47.677830 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:47.670976    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh" podUID="2d3647f2-89e9-4908-a8ee-d84e527abeb0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:23:47.701305 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:47.678211    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/distinct-w3-482722-kserve-d876967c-kd488" podUID="9a97f9cc-2261-471d-9053-51ee965850bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:23:47.701305 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:47.696379    5570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-9e1bcb/premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh" podUID="2d3647f2-89e9-4908-a8ee-d84e527abeb0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": context deadline exceeded"
Nov 03 20:23:47.783798 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:47.769466    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhp6s\" (UniqueName: \"kubernetes.io/projected/ec1dc0b9-d778-48b7-986a-0d460bb6a52f-kube-api-access-hhp6s\") pod \"payload-processing-e2e-shared-a-w5-de82c8-78dc548fc4-24g87\" (UID: \"ec1dc0b9-d778-48b7-986a-0d460bb6a52f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-de82c8-78dc548fc4-24g87"
Nov 03 20:23:47.856804 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:47.850654    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/distinct-2-w0-e40816-kserve-79fb46654d-s242x" podUID="4ad9b048-925c-4d01-a839-a5f25d4dd02d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:23:47.902877 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:47.901397    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz" podUID="3a715ef1-38e9-4386-9eec-e10e8df11a10" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": context deadline exceeded"
Nov 03 20:23:47.933587 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:47.933549    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-de82c8-78dc548fc4-24g87"
Nov 03 20:23:48.133798 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:48.132122    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/premium-w1-6fc748-kserve-7474db9cd7-nf2xk" podUID="830e062e-83fc-4de7-91a7-5534dac1d46b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:23:48.192576 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:48.187508    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-a619f336-openshift-default-66dcdd6969-h9lgf"]
Nov 03 20:23:48.249015 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:48.242760    5570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-a619f336-openshift-default-66dcdd6969-h9lgf" podUID="3f3e4187-d5b7-4ef6-8301-2b838530c525" containerName="istio-proxy" containerID="cri-o://138e37aa1ed4c1bafd9fc822ef0a425ae8d031b7f3f6c8492e5e0d36f8b8e8c5" gracePeriod=30
Nov 03 20:23:48.626900 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:48.622363    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/premium-w2-9973d9-kserve-5c6dd4669c-5gv8x" podUID="cefc89ad-7ae1-433b-979d-6601678a6d98" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:23:48.651282 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:48.635547    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9973d9/premium-w2-9973d9-kserve-5c6dd4669c-5gv8x"
Nov 03 20:23:48.722893 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:48.684902    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh" podUID="2d3647f2-89e9-4908-a8ee-d84e527abeb0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:23:48.722893 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:48.685348    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-9e1bcb/premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh"
Nov 03 20:23:48.722893 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:48.686679    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/distinct-w3-482722-kserve-d876967c-kd488" podUID="9a97f9cc-2261-471d-9053-51ee965850bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:23:48.813470 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:48.792998    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-482722/distinct-w3-482722-kserve-d876967c-kd488"
Nov 03 20:23:48.851749 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:48.851683    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/distinct-2-w0-e40816-kserve-79fb46654d-s242x" podUID="4ad9b048-925c-4d01-a839-a5f25d4dd02d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:23:48.853861 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:48.852813    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e40816/distinct-2-w0-e40816-kserve-79fb46654d-s242x"
Nov 03 20:23:48.922218 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:48.920299    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz" podUID="3a715ef1-38e9-4386-9eec-e10e8df11a10" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:23:48.922218 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:48.920733    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz"
Nov 03 20:23:48.936075 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:48.933346    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-9e1bcb/premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh"
Nov 03 20:23:48.941789 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:48.941747    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9973d9/premium-w2-9973d9-kserve-5c6dd4669c-5gv8x"
Nov 03 20:23:48.973725 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:48.956307    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-482722/distinct-w3-482722-kserve-d876967c-kd488"
Nov 03 20:23:48.986561 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:48.986518    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e40816/distinct-2-w0-e40816-kserve-79fb46654d-s242x"
Nov 03 20:23:49.001812 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:49.001655    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz"
Nov 03 20:23:49.184856 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:49.184827    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-de82c8-78dc548fc4-24g87"]
Nov 03 20:23:49.740005 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:49.734662    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-de82c8-78dc548fc4-24g87" event={"ID":"ec1dc0b9-d778-48b7-986a-0d460bb6a52f","Type":"ContainerStarted","Data":"9f9f5f310addd3b32215994257334bf8ab8e4a389edd3bb6eac7c9cbddde890f"}
Nov 03 20:23:50.780672 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:50.780631    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-de82c8-78dc548fc4-24g87" event={"ID":"ec1dc0b9-d778-48b7-986a-0d460bb6a52f","Type":"ContainerStarted","Data":"993c17641348655709085554057bbcd967eca31e3684298b29a1ef1929137826"}
Nov 03 20:23:50.781221 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:50.780721    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-de82c8-78dc548fc4-24g87"
Nov 03 20:23:50.823223 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:50.823165    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-de82c8-78dc548fc4-24g87" podStartSLOduration=4.823150377 podStartE2EDuration="4.823150377s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:23:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:23:50.822325784 +0000 UTC m=+1164.974987698" watchObservedRunningTime="2026-11-03 20:23:50.823150377 +0000 UTC m=+1164.975812454"
Nov 03 20:23:51.917807 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:51.908654    5570 patch_prober.go:29] interesting pod/e2e-derive-a619f336-openshift-default-66dcdd6969-h9lgf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 20:23:51.917807 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:51.908764    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-a619f336-openshift-default-66dcdd6969-h9lgf" podUID="3f3e4187-d5b7-4ef6-8301-2b838530c525" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 20:23:57.231906 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:23:57.231854    5570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8bwhz" containerName="extractor"
Nov 03 20:23:58.590265 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:58.590224    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-a619f336-openshift-default-66dcdd6969-h9lgf"
Nov 03 20:23:58.636226 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:58.636135    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f3e4187-d5b7-4ef6-8301-2b838530c525-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f3e4187-d5b7-4ef6-8301-2b838530c525-workload-certs\") pod \"3f3e4187-d5b7-4ef6-8301-2b838530c525\" (UID: \"3f3e4187-d5b7-4ef6-8301-2b838530c525\") "
Nov 03 20:23:58.636226 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:58.636188    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f3e4187-d5b7-4ef6-8301-2b838530c525-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f3e4187-d5b7-4ef6-8301-2b838530c525-istiod-ca-cert\") pod \"3f3e4187-d5b7-4ef6-8301-2b838530c525\" (UID: \"3f3e4187-d5b7-4ef6-8301-2b838530c525\") "
Nov 03 20:23:58.636226 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:58.636224    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f3e4187-d5b7-4ef6-8301-2b838530c525-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f3e4187-d5b7-4ef6-8301-2b838530c525-credential-socket\") pod \"3f3e4187-d5b7-4ef6-8301-2b838530c525\" (UID: \"3f3e4187-d5b7-4ef6-8301-2b838530c525\") "
Nov 03 20:23:58.636533 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:58.636285    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f3e4187-d5b7-4ef6-8301-2b838530c525-kube-api-access-pqp49\" (UniqueName: \"kubernetes.io/projected/3f3e4187-d5b7-4ef6-8301-2b838530c525-kube-api-access-pqp49\") pod \"3f3e4187-d5b7-4ef6-8301-2b838530c525\" (UID: \"3f3e4187-d5b7-4ef6-8301-2b838530c525\") "
Nov 03 20:23:58.636533 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:58.636352    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f3e4187-d5b7-4ef6-8301-2b838530c525-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f3e4187-d5b7-4ef6-8301-2b838530c525-istio-data\") pod \"3f3e4187-d5b7-4ef6-8301-2b838530c525\" (UID: \"3f3e4187-d5b7-4ef6-8301-2b838530c525\") "
Nov 03 20:23:58.636533 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:58.636415    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f3e4187-d5b7-4ef6-8301-2b838530c525-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f3e4187-d5b7-4ef6-8301-2b838530c525-workload-socket\") pod \"3f3e4187-d5b7-4ef6-8301-2b838530c525\" (UID: \"3f3e4187-d5b7-4ef6-8301-2b838530c525\") "
Nov 03 20:23:58.636533 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:58.636453    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3f3e4187-d5b7-4ef6-8301-2b838530c525-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f3e4187-d5b7-4ef6-8301-2b838530c525-istio-podinfo\") pod \"3f3e4187-d5b7-4ef6-8301-2b838530c525\" (UID: \"3f3e4187-d5b7-4ef6-8301-2b838530c525\") "
Nov 03 20:23:58.636753 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:58.636515    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f3e4187-d5b7-4ef6-8301-2b838530c525-workload-certs" pod "3f3e4187-d5b7-4ef6-8301-2b838530c525" (UID: "3f3e4187-d5b7-4ef6-8301-2b838530c525"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:23:58.636753 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:58.636621    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f3e4187-d5b7-4ef6-8301-2b838530c525-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f3e4187-d5b7-4ef6-8301-2b838530c525-istio-envoy\") pod \"3f3e4187-d5b7-4ef6-8301-2b838530c525\" (UID: \"3f3e4187-d5b7-4ef6-8301-2b838530c525\") "
Nov 03 20:23:58.636753 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:58.636692    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f3e4187-d5b7-4ef6-8301-2b838530c525-istiod-ca-cert" pod "3f3e4187-d5b7-4ef6-8301-2b838530c525" (UID: "3f3e4187-d5b7-4ef6-8301-2b838530c525"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:23:58.636936 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:58.636768    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f3e4187-d5b7-4ef6-8301-2b838530c525-istio-token\" (UniqueName: \"kubernetes.io/projected/3f3e4187-d5b7-4ef6-8301-2b838530c525-istio-token\") pod \"3f3e4187-d5b7-4ef6-8301-2b838530c525\" (UID: \"3f3e4187-d5b7-4ef6-8301-2b838530c525\") "
Nov 03 20:23:58.637114 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:58.637077    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f3e4187-d5b7-4ef6-8301-2b838530c525-credential-socket" pod "3f3e4187-d5b7-4ef6-8301-2b838530c525" (UID: "3f3e4187-d5b7-4ef6-8301-2b838530c525"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:23:58.637302 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:58.637277    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f3e4187-d5b7-4ef6-8301-2b838530c525-workload-socket" pod "3f3e4187-d5b7-4ef6-8301-2b838530c525" (UID: "3f3e4187-d5b7-4ef6-8301-2b838530c525"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:23:58.637474 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:58.637453    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f3e4187-d5b7-4ef6-8301-2b838530c525-istio-data" pod "3f3e4187-d5b7-4ef6-8301-2b838530c525" (UID: "3f3e4187-d5b7-4ef6-8301-2b838530c525"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:23:58.638035 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:58.638011    5570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f3e4187-d5b7-4ef6-8301-2b838530c525-istio-data\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:23:58.638138 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:58.638040    5570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f3e4187-d5b7-4ef6-8301-2b838530c525-workload-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:23:58.638138 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:58.638053    5570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f3e4187-d5b7-4ef6-8301-2b838530c525-workload-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:23:58.638138 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:58.638064    5570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f3e4187-d5b7-4ef6-8301-2b838530c525-istiod-ca-cert\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:23:58.638138 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:58.638077    5570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f3e4187-d5b7-4ef6-8301-2b838530c525-credential-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:23:58.639539 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:58.639512    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f3e4187-d5b7-4ef6-8301-2b838530c525-kube-api-access-pqp49" pod "3f3e4187-d5b7-4ef6-8301-2b838530c525" (UID: "3f3e4187-d5b7-4ef6-8301-2b838530c525"). InnerVolumeSpecName "kube-api-access-pqp49". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:23:58.639639 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:58.639610    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f3e4187-d5b7-4ef6-8301-2b838530c525-istio-envoy" pod "3f3e4187-d5b7-4ef6-8301-2b838530c525" (UID: "3f3e4187-d5b7-4ef6-8301-2b838530c525"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:23:58.640387 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:58.640355    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f3e4187-d5b7-4ef6-8301-2b838530c525-istio-token" pod "3f3e4187-d5b7-4ef6-8301-2b838530c525" (UID: "3f3e4187-d5b7-4ef6-8301-2b838530c525"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:23:58.640495 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:58.640417    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3f3e4187-d5b7-4ef6-8301-2b838530c525-istio-podinfo" pod "3f3e4187-d5b7-4ef6-8301-2b838530c525" (UID: "3f3e4187-d5b7-4ef6-8301-2b838530c525"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:23:58.741387 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:58.740829    5570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f3e4187-d5b7-4ef6-8301-2b838530c525-istio-envoy\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:23:58.741387 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:58.740878    5570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f3e4187-d5b7-4ef6-8301-2b838530c525-istio-token\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:23:58.741387 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:58.740897    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqp49\" (UniqueName: \"kubernetes.io/projected/3f3e4187-d5b7-4ef6-8301-2b838530c525-kube-api-access-pqp49\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:23:58.741387 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:58.740914    5570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f3e4187-d5b7-4ef6-8301-2b838530c525-istio-podinfo\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:23:58.978348 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:58.978311    5570 generic.go:356] "Generic (PLEG): container finished" podID="3f3e4187-d5b7-4ef6-8301-2b838530c525" containerID="138e37aa1ed4c1bafd9fc822ef0a425ae8d031b7f3f6c8492e5e0d36f8b8e8c5" exitCode=0
Nov 03 20:23:58.978559 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:58.978386    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-a619f336-openshift-default-66dcdd6969-h9lgf"
Nov 03 20:23:58.978559 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:58.978386    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-a619f336-openshift-default-66dcdd6969-h9lgf" event={"ID":"3f3e4187-d5b7-4ef6-8301-2b838530c525","Type":"ContainerDied","Data":"138e37aa1ed4c1bafd9fc822ef0a425ae8d031b7f3f6c8492e5e0d36f8b8e8c5"}
Nov 03 20:23:58.978559 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:58.978454    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-a619f336-openshift-default-66dcdd6969-h9lgf" event={"ID":"3f3e4187-d5b7-4ef6-8301-2b838530c525","Type":"ContainerDied","Data":"58e37911baead6c9b9294bb90da463ea39e45ca68c8524cffc9f351c341f6ccc"}
Nov 03 20:23:58.978559 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:58.978486    5570 scope.go:122] "RemoveContainer" containerID="138e37aa1ed4c1bafd9fc822ef0a425ae8d031b7f3f6c8492e5e0d36f8b8e8c5"
Nov 03 20:23:58.993994 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:58.993952    5570 scope.go:122] "RemoveContainer" containerID="138e37aa1ed4c1bafd9fc822ef0a425ae8d031b7f3f6c8492e5e0d36f8b8e8c5"
Nov 03 20:23:58.994226 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:23:58.994197    5570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"138e37aa1ed4c1bafd9fc822ef0a425ae8d031b7f3f6c8492e5e0d36f8b8e8c5\": container with ID starting with 138e37aa1ed4c1bafd9fc822ef0a425ae8d031b7f3f6c8492e5e0d36f8b8e8c5 not found: ID does not exist" containerID="138e37aa1ed4c1bafd9fc822ef0a425ae8d031b7f3f6c8492e5e0d36f8b8e8c5"
Nov 03 20:23:58.994348 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:58.994235    5570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"138e37aa1ed4c1bafd9fc822ef0a425ae8d031b7f3f6c8492e5e0d36f8b8e8c5"} err="failed to get container status \"138e37aa1ed4c1bafd9fc822ef0a425ae8d031b7f3f6c8492e5e0d36f8b8e8c5\": rpc error: code = NotFound desc = could not find container \"138e37aa1ed4c1bafd9fc822ef0a425ae8d031b7f3f6c8492e5e0d36f8b8e8c5\": container with ID starting with 138e37aa1ed4c1bafd9fc822ef0a425ae8d031b7f3f6c8492e5e0d36f8b8e8c5 not found: ID does not exist"
Nov 03 20:23:59.036366 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:59.036331    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-a619f336-openshift-default-66dcdd6969-h9lgf"]
Nov 03 20:23:59.041017 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:59.040987    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-a619f336-openshift-default-66dcdd6969-h9lgf"]
Nov 03 20:23:59.968098 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:23:59.968063    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f3e4187-d5b7-4ef6-8301-2b838530c525" path="/var/lib/kubelet/pods/3f3e4187-d5b7-4ef6-8301-2b838530c525/volumes"
Nov 03 20:24:01.842170 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:01.842143    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-de82c8-78dc548fc4-24g87"
Nov 03 20:24:07.712851 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:07.707584    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g69f8"]
Nov 03 20:24:26.743821 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:26.743787    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-jshb8"]
Nov 03 20:24:26.744539 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:26.744522    5570 cpu_manager.go:414] "containerMap: removing container" podUID="3f3e4187-d5b7-4ef6-8301-2b838530c525" containerName="istio-proxy"
Nov 03 20:24:26.744539 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:26.744539    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f3e4187-d5b7-4ef6-8301-2b838530c525" containerName="istio-proxy"
Nov 03 20:24:26.744696 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:26.744673    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f3e4187-d5b7-4ef6-8301-2b838530c525" containerName="istio-proxy"
Nov 03 20:24:26.747626 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:26.747602    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-jshb8"
Nov 03 20:24:26.771694 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:26.771666    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-jshb8"]
Nov 03 20:24:26.778007 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:26.777981    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w5-e3ba87-openshift-default-dockercfg-gblhq\""
Nov 03 20:24:26.911810 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:26.902977    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2137488e-8e6c-46f7-a0ed-b4721d4f2645-workload-certs\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-jshb8\" (UID: \"2137488e-8e6c-46f7-a0ed-b4721d4f2645\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-jshb8"
Nov 03 20:24:26.911810 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:26.903200    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nf4b\" (UniqueName: \"kubernetes.io/projected/2137488e-8e6c-46f7-a0ed-b4721d4f2645-kube-api-access-8nf4b\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-jshb8\" (UID: \"2137488e-8e6c-46f7-a0ed-b4721d4f2645\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-jshb8"
Nov 03 20:24:26.911810 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:26.903251    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2137488e-8e6c-46f7-a0ed-b4721d4f2645-credential-socket\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-jshb8\" (UID: \"2137488e-8e6c-46f7-a0ed-b4721d4f2645\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-jshb8"
Nov 03 20:24:26.911810 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:26.903331    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2137488e-8e6c-46f7-a0ed-b4721d4f2645-istio-data\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-jshb8\" (UID: \"2137488e-8e6c-46f7-a0ed-b4721d4f2645\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-jshb8"
Nov 03 20:24:26.911810 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:26.903489    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2137488e-8e6c-46f7-a0ed-b4721d4f2645-istio-podinfo\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-jshb8\" (UID: \"2137488e-8e6c-46f7-a0ed-b4721d4f2645\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-jshb8"
Nov 03 20:24:26.911810 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:26.903530    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2137488e-8e6c-46f7-a0ed-b4721d4f2645-istio-envoy\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-jshb8\" (UID: \"2137488e-8e6c-46f7-a0ed-b4721d4f2645\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-jshb8"
Nov 03 20:24:26.911810 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:26.903573    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2137488e-8e6c-46f7-a0ed-b4721d4f2645-workload-socket\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-jshb8\" (UID: \"2137488e-8e6c-46f7-a0ed-b4721d4f2645\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-jshb8"
Nov 03 20:24:26.911810 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:26.903763    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2137488e-8e6c-46f7-a0ed-b4721d4f2645-istiod-ca-cert\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-jshb8\" (UID: \"2137488e-8e6c-46f7-a0ed-b4721d4f2645\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-jshb8"
Nov 03 20:24:26.911810 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:26.903839    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2137488e-8e6c-46f7-a0ed-b4721d4f2645-istio-token\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-jshb8\" (UID: \"2137488e-8e6c-46f7-a0ed-b4721d4f2645\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-jshb8"
Nov 03 20:24:27.005572 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:27.005469    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2137488e-8e6c-46f7-a0ed-b4721d4f2645-istiod-ca-cert\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-jshb8\" (UID: \"2137488e-8e6c-46f7-a0ed-b4721d4f2645\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-jshb8"
Nov 03 20:24:27.007180 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:27.007153    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2137488e-8e6c-46f7-a0ed-b4721d4f2645-istio-token\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-jshb8\" (UID: \"2137488e-8e6c-46f7-a0ed-b4721d4f2645\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-jshb8"
Nov 03 20:24:27.007451 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:27.007429    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2137488e-8e6c-46f7-a0ed-b4721d4f2645-workload-certs\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-jshb8\" (UID: \"2137488e-8e6c-46f7-a0ed-b4721d4f2645\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-jshb8"
Nov 03 20:24:27.007701 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:27.007680    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nf4b\" (UniqueName: \"kubernetes.io/projected/2137488e-8e6c-46f7-a0ed-b4721d4f2645-kube-api-access-8nf4b\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-jshb8\" (UID: \"2137488e-8e6c-46f7-a0ed-b4721d4f2645\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-jshb8"
Nov 03 20:24:27.007878 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:27.007848    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2137488e-8e6c-46f7-a0ed-b4721d4f2645-credential-socket\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-jshb8\" (UID: \"2137488e-8e6c-46f7-a0ed-b4721d4f2645\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-jshb8"
Nov 03 20:24:27.008082 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:27.008063    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2137488e-8e6c-46f7-a0ed-b4721d4f2645-istio-data\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-jshb8\" (UID: \"2137488e-8e6c-46f7-a0ed-b4721d4f2645\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-jshb8"
Nov 03 20:24:27.008311 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:27.008293    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2137488e-8e6c-46f7-a0ed-b4721d4f2645-istio-podinfo\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-jshb8\" (UID: \"2137488e-8e6c-46f7-a0ed-b4721d4f2645\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-jshb8"
Nov 03 20:24:27.013801 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:27.008839    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2137488e-8e6c-46f7-a0ed-b4721d4f2645-istio-envoy\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-jshb8\" (UID: \"2137488e-8e6c-46f7-a0ed-b4721d4f2645\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-jshb8"
Nov 03 20:24:27.015583 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:27.015563    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2137488e-8e6c-46f7-a0ed-b4721d4f2645-workload-socket\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-jshb8\" (UID: \"2137488e-8e6c-46f7-a0ed-b4721d4f2645\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-jshb8"
Nov 03 20:24:27.015986 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:27.014758    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2137488e-8e6c-46f7-a0ed-b4721d4f2645-workload-certs\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-jshb8\" (UID: \"2137488e-8e6c-46f7-a0ed-b4721d4f2645\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-jshb8"
Nov 03 20:24:27.028186 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:27.015005    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2137488e-8e6c-46f7-a0ed-b4721d4f2645-istio-data\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-jshb8\" (UID: \"2137488e-8e6c-46f7-a0ed-b4721d4f2645\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-jshb8"
Nov 03 20:24:27.028345 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:27.015185    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2137488e-8e6c-46f7-a0ed-b4721d4f2645-credential-socket\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-jshb8\" (UID: \"2137488e-8e6c-46f7-a0ed-b4721d4f2645\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-jshb8"
Nov 03 20:24:27.028448 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:27.026700    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2137488e-8e6c-46f7-a0ed-b4721d4f2645-workload-socket\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-jshb8\" (UID: \"2137488e-8e6c-46f7-a0ed-b4721d4f2645\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-jshb8"
Nov 03 20:24:27.028540 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:27.007065    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2137488e-8e6c-46f7-a0ed-b4721d4f2645-istiod-ca-cert\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-jshb8\" (UID: \"2137488e-8e6c-46f7-a0ed-b4721d4f2645\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-jshb8"
Nov 03 20:24:27.049336 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:27.043424    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2137488e-8e6c-46f7-a0ed-b4721d4f2645-istio-envoy\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-jshb8\" (UID: \"2137488e-8e6c-46f7-a0ed-b4721d4f2645\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-jshb8"
Nov 03 20:24:27.065797 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:27.055298    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2137488e-8e6c-46f7-a0ed-b4721d4f2645-istio-token\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-jshb8\" (UID: \"2137488e-8e6c-46f7-a0ed-b4721d4f2645\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-jshb8"
Nov 03 20:24:27.084191 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:27.084158    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2137488e-8e6c-46f7-a0ed-b4721d4f2645-istio-podinfo\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-jshb8\" (UID: \"2137488e-8e6c-46f7-a0ed-b4721d4f2645\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-jshb8"
Nov 03 20:24:27.091682 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:27.091647    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nf4b\" (UniqueName: \"kubernetes.io/projected/2137488e-8e6c-46f7-a0ed-b4721d4f2645-kube-api-access-8nf4b\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-jshb8\" (UID: \"2137488e-8e6c-46f7-a0ed-b4721d4f2645\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-jshb8"
Nov 03 20:24:27.317342 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:24:27.317208    5570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8bwhz" containerName="extractor"
Nov 03 20:24:27.371183 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:27.371139    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-jshb8"
Nov 03 20:24:27.477791 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:27.477752    5570 scope.go:122] "RemoveContainer" containerID="4ae68a259a5b7161d3d33ad6695f2104469a7b23847066d880e5d5985a4fc31b"
Nov 03 20:24:27.490024 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:27.489999    5570 scope.go:122] "RemoveContainer" containerID="e118c468a5c7f954ef0bc4918703c3d9a614fd8305322d46265b4ab2c5152692"
Nov 03 20:24:27.501445 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:27.501410    5570 scope.go:122] "RemoveContainer" containerID="dab00e9a83a21bf12d6f4e1e1b9de560477adf7fcd0778cda63edba11fcccdb2"
Nov 03 20:24:27.511830 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:27.511702    5570 scope.go:122] "RemoveContainer" containerID="f8fe2380352955f74cba24b5ca77339070e0316b4fc02555293f3c33c84b0856"
Nov 03 20:24:27.525191 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:27.524845    5570 scope.go:122] "RemoveContainer" containerID="1e965ca4919ebe84922b40a1c631ea5e6cf63909c32fda4a403b7f217fa3185b"
Nov 03 20:24:27.546509 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:27.546482    5570 scope.go:122] "RemoveContainer" containerID="d0ffb9fc1f732a4ecad34a9b58046139943f0e616a1eccf1d8d21a3e9902aac5"
Nov 03 20:24:27.564881 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:27.564852    5570 scope.go:122] "RemoveContainer" containerID="f655aee66fa3761cc6fa99517c64c01f20c20d8a62136f29963855f1572557f5"
Nov 03 20:24:27.786047 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:24:27.786007    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2137488e_8e6c_46f7_a0ed_b4721d4f2645.slice/crio-3e8680d5d08f95692b79cef9b5ca20ce0df9b6c1a4f3ae92411d7dc55892cdff WatchSource:0}: Error finding container 3e8680d5d08f95692b79cef9b5ca20ce0df9b6c1a4f3ae92411d7dc55892cdff: Status 404 returned error can't find the container with id 3e8680d5d08f95692b79cef9b5ca20ce0df9b6c1a4f3ae92411d7dc55892cdff
Nov 03 20:24:27.786372 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:27.786156    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-jshb8"]
Nov 03 20:24:27.788637 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:27.788608    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:24:27.788715 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:27.788668    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:24:27.788715 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:27.788699    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:24:28.660145 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:28.660104    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-jshb8" event={"ID":"2137488e-8e6c-46f7-a0ed-b4721d4f2645","Type":"ContainerStarted","Data":"117061888c8f2f87481828900cffe3e409615fd7d0abfd26850b40bf839788ee"}
Nov 03 20:24:28.660425 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:28.660193    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-jshb8" event={"ID":"2137488e-8e6c-46f7-a0ed-b4721d4f2645","Type":"ContainerStarted","Data":"3e8680d5d08f95692b79cef9b5ca20ce0df9b6c1a4f3ae92411d7dc55892cdff"}
Nov 03 20:24:28.732521 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:28.732463    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-jshb8" podStartSLOduration=2.732449207 podStartE2EDuration="2.732449207s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:24:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:24:28.714850438 +0000 UTC m=+1202.867512325" watchObservedRunningTime="2026-11-03 20:24:28.732449207 +0000 UTC m=+1202.885111095"
Nov 03 20:24:29.376089 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:29.376056    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-jshb8"
Nov 03 20:24:29.420807 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:29.417109    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-jshb8"
Nov 03 20:24:29.717220 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:29.717146    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-jshb8"
Nov 03 20:24:29.721882 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:29.721411    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-jshb8"
Nov 03 20:24:32.978808 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:32.978743    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-e3ba87-7c7fd945bd-dzmfm"]
Nov 03 20:24:32.984135 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:32.984115    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-e3ba87-7c7fd945bd-dzmfm"
Nov 03 20:24:33.001069 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:33.001039    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-e3ba87-7c7fd945bd-dzmfm"]
Nov 03 20:24:33.002755 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:33.002716    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qj9s9\" (UniqueName: \"kubernetes.io/projected/5046089f-f2fa-4ed2-b424-7661fb3ea8c2-kube-api-access-qj9s9\") pod \"payload-processing-e2e-shared-b-w5-e3ba87-7c7fd945bd-dzmfm\" (UID: \"5046089f-f2fa-4ed2-b424-7661fb3ea8c2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-e3ba87-7c7fd945bd-dzmfm"
Nov 03 20:24:33.003015 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:33.002991    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5046089f-f2fa-4ed2-b424-7661fb3ea8c2-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-e3ba87-7c7fd945bd-dzmfm\" (UID: \"5046089f-f2fa-4ed2-b424-7661fb3ea8c2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-e3ba87-7c7fd945bd-dzmfm"
Nov 03 20:24:33.008106 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:33.008083    5570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w5-e3ba87\""
Nov 03 20:24:33.008263 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:33.008211    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w5-e3ba87-dockercfg-d52q6\""
Nov 03 20:24:33.129796 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:33.104528    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5046089f-f2fa-4ed2-b424-7661fb3ea8c2-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-e3ba87-7c7fd945bd-dzmfm\" (UID: \"5046089f-f2fa-4ed2-b424-7661fb3ea8c2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-e3ba87-7c7fd945bd-dzmfm"
Nov 03 20:24:33.130566 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:33.130518    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qj9s9\" (UniqueName: \"kubernetes.io/projected/5046089f-f2fa-4ed2-b424-7661fb3ea8c2-kube-api-access-qj9s9\") pod \"payload-processing-e2e-shared-b-w5-e3ba87-7c7fd945bd-dzmfm\" (UID: \"5046089f-f2fa-4ed2-b424-7661fb3ea8c2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-e3ba87-7c7fd945bd-dzmfm"
Nov 03 20:24:33.232068 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:33.231968    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5046089f-f2fa-4ed2-b424-7661fb3ea8c2-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-e3ba87-7c7fd945bd-dzmfm\" (UID: \"5046089f-f2fa-4ed2-b424-7661fb3ea8c2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-e3ba87-7c7fd945bd-dzmfm"
Nov 03 20:24:33.408799 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:33.385537    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qj9s9\" (UniqueName: \"kubernetes.io/projected/5046089f-f2fa-4ed2-b424-7661fb3ea8c2-kube-api-access-qj9s9\") pod \"payload-processing-e2e-shared-b-w5-e3ba87-7c7fd945bd-dzmfm\" (UID: \"5046089f-f2fa-4ed2-b424-7661fb3ea8c2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-e3ba87-7c7fd945bd-dzmfm"
Nov 03 20:24:33.617847 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:33.605071    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-e3ba87-7c7fd945bd-dzmfm"
Nov 03 20:24:33.835181 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:33.835129    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-a0866c-openshift-default-59c9f599d5-4h8ks"]
Nov 03 20:24:33.842981 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:33.842952    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-a0866c-openshift-default-59c9f599d5-4h8ks"
Nov 03 20:24:33.871190 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:33.871160    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-a0866c-openshift-default-59c9f599d5-4h8ks"]
Nov 03 20:24:33.898686 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:33.898650    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-a0866c-openshift-default-dockercfg-j2qzt\""
Nov 03 20:24:33.921233 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:33.921200    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxpfl\" (UniqueName: \"kubernetes.io/projected/9e1c6800-b892-4846-8686-63dbb9b87690-kube-api-access-cxpfl\") pod \"e2e-shared-b-w4-a0866c-openshift-default-59c9f599d5-4h8ks\" (UID: \"9e1c6800-b892-4846-8686-63dbb9b87690\") " pod="openshift-ingress/e2e-shared-b-w4-a0866c-openshift-default-59c9f599d5-4h8ks"
Nov 03 20:24:33.921389 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:33.921238    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e1c6800-b892-4846-8686-63dbb9b87690-credential-socket\") pod \"e2e-shared-b-w4-a0866c-openshift-default-59c9f599d5-4h8ks\" (UID: \"9e1c6800-b892-4846-8686-63dbb9b87690\") " pod="openshift-ingress/e2e-shared-b-w4-a0866c-openshift-default-59c9f599d5-4h8ks"
Nov 03 20:24:33.921427 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:33.921402    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e1c6800-b892-4846-8686-63dbb9b87690-workload-certs\") pod \"e2e-shared-b-w4-a0866c-openshift-default-59c9f599d5-4h8ks\" (UID: \"9e1c6800-b892-4846-8686-63dbb9b87690\") " pod="openshift-ingress/e2e-shared-b-w4-a0866c-openshift-default-59c9f599d5-4h8ks"
Nov 03 20:24:33.921540 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:33.921525    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e1c6800-b892-4846-8686-63dbb9b87690-istio-token\") pod \"e2e-shared-b-w4-a0866c-openshift-default-59c9f599d5-4h8ks\" (UID: \"9e1c6800-b892-4846-8686-63dbb9b87690\") " pod="openshift-ingress/e2e-shared-b-w4-a0866c-openshift-default-59c9f599d5-4h8ks"
Nov 03 20:24:33.921605 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:33.921592    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e1c6800-b892-4846-8686-63dbb9b87690-istio-podinfo\") pod \"e2e-shared-b-w4-a0866c-openshift-default-59c9f599d5-4h8ks\" (UID: \"9e1c6800-b892-4846-8686-63dbb9b87690\") " pod="openshift-ingress/e2e-shared-b-w4-a0866c-openshift-default-59c9f599d5-4h8ks"
Nov 03 20:24:33.921697 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:33.921682    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e1c6800-b892-4846-8686-63dbb9b87690-istio-envoy\") pod \"e2e-shared-b-w4-a0866c-openshift-default-59c9f599d5-4h8ks\" (UID: \"9e1c6800-b892-4846-8686-63dbb9b87690\") " pod="openshift-ingress/e2e-shared-b-w4-a0866c-openshift-default-59c9f599d5-4h8ks"
Nov 03 20:24:33.921732 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:33.921711    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e1c6800-b892-4846-8686-63dbb9b87690-workload-socket\") pod \"e2e-shared-b-w4-a0866c-openshift-default-59c9f599d5-4h8ks\" (UID: \"9e1c6800-b892-4846-8686-63dbb9b87690\") " pod="openshift-ingress/e2e-shared-b-w4-a0866c-openshift-default-59c9f599d5-4h8ks"
Nov 03 20:24:33.921803 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:33.921791    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e1c6800-b892-4846-8686-63dbb9b87690-istiod-ca-cert\") pod \"e2e-shared-b-w4-a0866c-openshift-default-59c9f599d5-4h8ks\" (UID: \"9e1c6800-b892-4846-8686-63dbb9b87690\") " pod="openshift-ingress/e2e-shared-b-w4-a0866c-openshift-default-59c9f599d5-4h8ks"
Nov 03 20:24:33.921906 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:33.921885    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e1c6800-b892-4846-8686-63dbb9b87690-istio-data\") pod \"e2e-shared-b-w4-a0866c-openshift-default-59c9f599d5-4h8ks\" (UID: \"9e1c6800-b892-4846-8686-63dbb9b87690\") " pod="openshift-ingress/e2e-shared-b-w4-a0866c-openshift-default-59c9f599d5-4h8ks"
Nov 03 20:24:34.046282 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:34.046235    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e1c6800-b892-4846-8686-63dbb9b87690-workload-certs\") pod \"e2e-shared-b-w4-a0866c-openshift-default-59c9f599d5-4h8ks\" (UID: \"9e1c6800-b892-4846-8686-63dbb9b87690\") " pod="openshift-ingress/e2e-shared-b-w4-a0866c-openshift-default-59c9f599d5-4h8ks"
Nov 03 20:24:34.047596 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:34.047563    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e1c6800-b892-4846-8686-63dbb9b87690-istio-token\") pod \"e2e-shared-b-w4-a0866c-openshift-default-59c9f599d5-4h8ks\" (UID: \"9e1c6800-b892-4846-8686-63dbb9b87690\") " pod="openshift-ingress/e2e-shared-b-w4-a0866c-openshift-default-59c9f599d5-4h8ks"
Nov 03 20:24:34.048065 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:34.048044    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e1c6800-b892-4846-8686-63dbb9b87690-istio-podinfo\") pod \"e2e-shared-b-w4-a0866c-openshift-default-59c9f599d5-4h8ks\" (UID: \"9e1c6800-b892-4846-8686-63dbb9b87690\") " pod="openshift-ingress/e2e-shared-b-w4-a0866c-openshift-default-59c9f599d5-4h8ks"
Nov 03 20:24:34.048504 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:34.048480    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e1c6800-b892-4846-8686-63dbb9b87690-istio-envoy\") pod \"e2e-shared-b-w4-a0866c-openshift-default-59c9f599d5-4h8ks\" (UID: \"9e1c6800-b892-4846-8686-63dbb9b87690\") " pod="openshift-ingress/e2e-shared-b-w4-a0866c-openshift-default-59c9f599d5-4h8ks"
Nov 03 20:24:34.068142 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:34.068017    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e1c6800-b892-4846-8686-63dbb9b87690-workload-socket\") pod \"e2e-shared-b-w4-a0866c-openshift-default-59c9f599d5-4h8ks\" (UID: \"9e1c6800-b892-4846-8686-63dbb9b87690\") " pod="openshift-ingress/e2e-shared-b-w4-a0866c-openshift-default-59c9f599d5-4h8ks"
Nov 03 20:24:34.068977 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:34.068953    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e1c6800-b892-4846-8686-63dbb9b87690-istiod-ca-cert\") pod \"e2e-shared-b-w4-a0866c-openshift-default-59c9f599d5-4h8ks\" (UID: \"9e1c6800-b892-4846-8686-63dbb9b87690\") " pod="openshift-ingress/e2e-shared-b-w4-a0866c-openshift-default-59c9f599d5-4h8ks"
Nov 03 20:24:34.069594 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:34.069482    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e1c6800-b892-4846-8686-63dbb9b87690-istio-data\") pod \"e2e-shared-b-w4-a0866c-openshift-default-59c9f599d5-4h8ks\" (UID: \"9e1c6800-b892-4846-8686-63dbb9b87690\") " pod="openshift-ingress/e2e-shared-b-w4-a0866c-openshift-default-59c9f599d5-4h8ks"
Nov 03 20:24:34.071975 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:34.071950    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxpfl\" (UniqueName: \"kubernetes.io/projected/9e1c6800-b892-4846-8686-63dbb9b87690-kube-api-access-cxpfl\") pod \"e2e-shared-b-w4-a0866c-openshift-default-59c9f599d5-4h8ks\" (UID: \"9e1c6800-b892-4846-8686-63dbb9b87690\") " pod="openshift-ingress/e2e-shared-b-w4-a0866c-openshift-default-59c9f599d5-4h8ks"
Nov 03 20:24:34.072159 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:34.072141    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e1c6800-b892-4846-8686-63dbb9b87690-credential-socket\") pod \"e2e-shared-b-w4-a0866c-openshift-default-59c9f599d5-4h8ks\" (UID: \"9e1c6800-b892-4846-8686-63dbb9b87690\") " pod="openshift-ingress/e2e-shared-b-w4-a0866c-openshift-default-59c9f599d5-4h8ks"
Nov 03 20:24:34.088945 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:34.068694    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e1c6800-b892-4846-8686-63dbb9b87690-workload-socket\") pod \"e2e-shared-b-w4-a0866c-openshift-default-59c9f599d5-4h8ks\" (UID: \"9e1c6800-b892-4846-8686-63dbb9b87690\") " pod="openshift-ingress/e2e-shared-b-w4-a0866c-openshift-default-59c9f599d5-4h8ks"
Nov 03 20:24:34.088945 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:34.070747    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e1c6800-b892-4846-8686-63dbb9b87690-istio-data\") pod \"e2e-shared-b-w4-a0866c-openshift-default-59c9f599d5-4h8ks\" (UID: \"9e1c6800-b892-4846-8686-63dbb9b87690\") " pod="openshift-ingress/e2e-shared-b-w4-a0866c-openshift-default-59c9f599d5-4h8ks"
Nov 03 20:24:34.088945 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:34.072594    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e1c6800-b892-4846-8686-63dbb9b87690-credential-socket\") pod \"e2e-shared-b-w4-a0866c-openshift-default-59c9f599d5-4h8ks\" (UID: \"9e1c6800-b892-4846-8686-63dbb9b87690\") " pod="openshift-ingress/e2e-shared-b-w4-a0866c-openshift-default-59c9f599d5-4h8ks"
Nov 03 20:24:34.088945 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:34.070505    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e1c6800-b892-4846-8686-63dbb9b87690-istiod-ca-cert\") pod \"e2e-shared-b-w4-a0866c-openshift-default-59c9f599d5-4h8ks\" (UID: \"9e1c6800-b892-4846-8686-63dbb9b87690\") " pod="openshift-ingress/e2e-shared-b-w4-a0866c-openshift-default-59c9f599d5-4h8ks"
Nov 03 20:24:34.088945 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:34.048248    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e1c6800-b892-4846-8686-63dbb9b87690-workload-certs\") pod \"e2e-shared-b-w4-a0866c-openshift-default-59c9f599d5-4h8ks\" (UID: \"9e1c6800-b892-4846-8686-63dbb9b87690\") " pod="openshift-ingress/e2e-shared-b-w4-a0866c-openshift-default-59c9f599d5-4h8ks"
Nov 03 20:24:34.125135 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:34.121562    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxpfl\" (UniqueName: \"kubernetes.io/projected/9e1c6800-b892-4846-8686-63dbb9b87690-kube-api-access-cxpfl\") pod \"e2e-shared-b-w4-a0866c-openshift-default-59c9f599d5-4h8ks\" (UID: \"9e1c6800-b892-4846-8686-63dbb9b87690\") " pod="openshift-ingress/e2e-shared-b-w4-a0866c-openshift-default-59c9f599d5-4h8ks"
Nov 03 20:24:34.125135 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:34.122008    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e1c6800-b892-4846-8686-63dbb9b87690-istio-token\") pod \"e2e-shared-b-w4-a0866c-openshift-default-59c9f599d5-4h8ks\" (UID: \"9e1c6800-b892-4846-8686-63dbb9b87690\") " pod="openshift-ingress/e2e-shared-b-w4-a0866c-openshift-default-59c9f599d5-4h8ks"
Nov 03 20:24:34.160926 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:34.160855    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e1c6800-b892-4846-8686-63dbb9b87690-istio-podinfo\") pod \"e2e-shared-b-w4-a0866c-openshift-default-59c9f599d5-4h8ks\" (UID: \"9e1c6800-b892-4846-8686-63dbb9b87690\") " pod="openshift-ingress/e2e-shared-b-w4-a0866c-openshift-default-59c9f599d5-4h8ks"
Nov 03 20:24:34.168266 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:34.168232    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e1c6800-b892-4846-8686-63dbb9b87690-istio-envoy\") pod \"e2e-shared-b-w4-a0866c-openshift-default-59c9f599d5-4h8ks\" (UID: \"9e1c6800-b892-4846-8686-63dbb9b87690\") " pod="openshift-ingress/e2e-shared-b-w4-a0866c-openshift-default-59c9f599d5-4h8ks"
Nov 03 20:24:34.182368 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:34.182338    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-a0866c-openshift-default-59c9f599d5-4h8ks"
Nov 03 20:24:34.373334 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:24:34.368958    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5046089f_f2fa_4ed2_b424_7661fb3ea8c2.slice/crio-adf23f87b70594cd11222dd65534aac6826201d0db6fddeb642a2a8380af7075 WatchSource:0}: Error finding container adf23f87b70594cd11222dd65534aac6826201d0db6fddeb642a2a8380af7075: Status 404 returned error can't find the container with id adf23f87b70594cd11222dd65534aac6826201d0db6fddeb642a2a8380af7075
Nov 03 20:24:34.443792 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:34.436919    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-e3ba87-7c7fd945bd-dzmfm"]
Nov 03 20:24:34.759575 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:34.759541    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-e3ba87-7c7fd945bd-dzmfm" event={"ID":"5046089f-f2fa-4ed2-b424-7661fb3ea8c2","Type":"ContainerStarted","Data":"28ec97b6355ca6177efbbf8aa5e614a1eaa02acdfec2e882ee20ad66e948eaad"}
Nov 03 20:24:34.759768 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:34.759601    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-e3ba87-7c7fd945bd-dzmfm" event={"ID":"5046089f-f2fa-4ed2-b424-7661fb3ea8c2","Type":"ContainerStarted","Data":"adf23f87b70594cd11222dd65534aac6826201d0db6fddeb642a2a8380af7075"}
Nov 03 20:24:34.759768 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:34.759662    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-e3ba87-7c7fd945bd-dzmfm"
Nov 03 20:24:34.844797 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:34.834791    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-e3ba87-7c7fd945bd-dzmfm" podStartSLOduration=2.8347569950000002 podStartE2EDuration="2.834756995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:24:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:24:34.833961296 +0000 UTC m=+1208.986623183" watchObservedRunningTime="2026-11-03 20:24:34.834756995 +0000 UTC m=+1208.987418881"
Nov 03 20:24:34.911077 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:34.910984    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-a0866c-openshift-default-59c9f599d5-4h8ks"]
Nov 03 20:24:34.945348 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:34.945269    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:24:34.945560 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:34.945540    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:24:34.945683 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:34.945668    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:24:35.768832 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:35.768784    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-a0866c-openshift-default-59c9f599d5-4h8ks" event={"ID":"9e1c6800-b892-4846-8686-63dbb9b87690","Type":"ContainerStarted","Data":"a37981af87d8ac58fb4bc1501a5fb409c54c5812026347d5aadec77740acfcac"}
Nov 03 20:24:35.769322 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:35.768846    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-a0866c-openshift-default-59c9f599d5-4h8ks" event={"ID":"9e1c6800-b892-4846-8686-63dbb9b87690","Type":"ContainerStarted","Data":"d604e752150e2ecdcfbaf7da193696e6c8ead5baeccd6c390a732bb91cf931e9"}
Nov 03 20:24:35.828968 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:35.828912    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-a0866c-openshift-default-59c9f599d5-4h8ks" podStartSLOduration=2.828898773 podStartE2EDuration="2.828898773s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:24:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:24:35.824872862 +0000 UTC m=+1209.977534750" watchObservedRunningTime="2026-11-03 20:24:35.828898773 +0000 UTC m=+1209.981560659"
Nov 03 20:24:36.183356 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:36.183270    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-a0866c-openshift-default-59c9f599d5-4h8ks"
Nov 03 20:24:36.189537 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:36.189516    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-a0866c-openshift-default-59c9f599d5-4h8ks"
Nov 03 20:24:36.776717 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:36.776685    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-a0866c-openshift-default-59c9f599d5-4h8ks"
Nov 03 20:24:36.777925 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:36.777903    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-a0866c-openshift-default-59c9f599d5-4h8ks"
Nov 03 20:24:37.278459 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:37.278426    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-nsjcd"]
Nov 03 20:24:37.283341 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:37.283318    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-nsjcd"
Nov 03 20:24:37.308999 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:37.308058    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-nsjcd"]
Nov 03 20:24:37.312457 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:37.312433    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-eb87f429-openshift-default-dockercfg-fwjlv\""
Nov 03 20:24:37.349465 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:37.349427    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b57a22a-cf28-4099-8883-c04c61807dfa-credential-socket\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-nsjcd\" (UID: \"7b57a22a-cf28-4099-8883-c04c61807dfa\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-nsjcd"
Nov 03 20:24:37.349667 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:37.349559    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b57a22a-cf28-4099-8883-c04c61807dfa-istio-data\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-nsjcd\" (UID: \"7b57a22a-cf28-4099-8883-c04c61807dfa\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-nsjcd"
Nov 03 20:24:37.349667 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:37.349646    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b57a22a-cf28-4099-8883-c04c61807dfa-istio-podinfo\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-nsjcd\" (UID: \"7b57a22a-cf28-4099-8883-c04c61807dfa\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-nsjcd"
Nov 03 20:24:37.349809 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:37.349676    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b57a22a-cf28-4099-8883-c04c61807dfa-istio-token\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-nsjcd\" (UID: \"7b57a22a-cf28-4099-8883-c04c61807dfa\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-nsjcd"
Nov 03 20:24:37.349878 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:37.349808    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5vqm\" (UniqueName: \"kubernetes.io/projected/7b57a22a-cf28-4099-8883-c04c61807dfa-kube-api-access-f5vqm\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-nsjcd\" (UID: \"7b57a22a-cf28-4099-8883-c04c61807dfa\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-nsjcd"
Nov 03 20:24:37.349945 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:37.349875    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b57a22a-cf28-4099-8883-c04c61807dfa-workload-certs\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-nsjcd\" (UID: \"7b57a22a-cf28-4099-8883-c04c61807dfa\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-nsjcd"
Nov 03 20:24:37.350016 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:37.349998    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b57a22a-cf28-4099-8883-c04c61807dfa-workload-socket\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-nsjcd\" (UID: \"7b57a22a-cf28-4099-8883-c04c61807dfa\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-nsjcd"
Nov 03 20:24:37.350125 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:37.350109    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b57a22a-cf28-4099-8883-c04c61807dfa-istio-envoy\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-nsjcd\" (UID: \"7b57a22a-cf28-4099-8883-c04c61807dfa\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-nsjcd"
Nov 03 20:24:37.350502 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:37.350468    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b57a22a-cf28-4099-8883-c04c61807dfa-istiod-ca-cert\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-nsjcd\" (UID: \"7b57a22a-cf28-4099-8883-c04c61807dfa\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-nsjcd"
Nov 03 20:24:37.463816 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:37.453008    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b57a22a-cf28-4099-8883-c04c61807dfa-istio-podinfo\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-nsjcd\" (UID: \"7b57a22a-cf28-4099-8883-c04c61807dfa\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-nsjcd"
Nov 03 20:24:37.463816 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:37.453059    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b57a22a-cf28-4099-8883-c04c61807dfa-istio-token\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-nsjcd\" (UID: \"7b57a22a-cf28-4099-8883-c04c61807dfa\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-nsjcd"
Nov 03 20:24:37.463816 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:37.453171    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5vqm\" (UniqueName: \"kubernetes.io/projected/7b57a22a-cf28-4099-8883-c04c61807dfa-kube-api-access-f5vqm\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-nsjcd\" (UID: \"7b57a22a-cf28-4099-8883-c04c61807dfa\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-nsjcd"
Nov 03 20:24:37.463816 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:37.453236    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b57a22a-cf28-4099-8883-c04c61807dfa-workload-certs\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-nsjcd\" (UID: \"7b57a22a-cf28-4099-8883-c04c61807dfa\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-nsjcd"
Nov 03 20:24:37.463816 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:37.453351    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b57a22a-cf28-4099-8883-c04c61807dfa-workload-socket\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-nsjcd\" (UID: \"7b57a22a-cf28-4099-8883-c04c61807dfa\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-nsjcd"
Nov 03 20:24:37.463816 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:37.453445    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b57a22a-cf28-4099-8883-c04c61807dfa-istio-envoy\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-nsjcd\" (UID: \"7b57a22a-cf28-4099-8883-c04c61807dfa\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-nsjcd"
Nov 03 20:24:37.463816 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:37.453511    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b57a22a-cf28-4099-8883-c04c61807dfa-istiod-ca-cert\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-nsjcd\" (UID: \"7b57a22a-cf28-4099-8883-c04c61807dfa\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-nsjcd"
Nov 03 20:24:37.492116 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:37.466665    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b57a22a-cf28-4099-8883-c04c61807dfa-istio-podinfo\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-nsjcd\" (UID: \"7b57a22a-cf28-4099-8883-c04c61807dfa\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-nsjcd"
Nov 03 20:24:37.492116 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:37.472986    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b57a22a-cf28-4099-8883-c04c61807dfa-credential-socket\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-nsjcd\" (UID: \"7b57a22a-cf28-4099-8883-c04c61807dfa\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-nsjcd"
Nov 03 20:24:37.492116 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:37.473081    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b57a22a-cf28-4099-8883-c04c61807dfa-istio-data\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-nsjcd\" (UID: \"7b57a22a-cf28-4099-8883-c04c61807dfa\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-nsjcd"
Nov 03 20:24:37.492116 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:37.473642    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b57a22a-cf28-4099-8883-c04c61807dfa-istio-data\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-nsjcd\" (UID: \"7b57a22a-cf28-4099-8883-c04c61807dfa\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-nsjcd"
Nov 03 20:24:37.492116 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:37.474586    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b57a22a-cf28-4099-8883-c04c61807dfa-istiod-ca-cert\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-nsjcd\" (UID: \"7b57a22a-cf28-4099-8883-c04c61807dfa\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-nsjcd"
Nov 03 20:24:37.492116 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:37.474848    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b57a22a-cf28-4099-8883-c04c61807dfa-workload-certs\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-nsjcd\" (UID: \"7b57a22a-cf28-4099-8883-c04c61807dfa\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-nsjcd"
Nov 03 20:24:37.492116 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:37.475069    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b57a22a-cf28-4099-8883-c04c61807dfa-workload-socket\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-nsjcd\" (UID: \"7b57a22a-cf28-4099-8883-c04c61807dfa\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-nsjcd"
Nov 03 20:24:37.492116 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:37.484225    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b57a22a-cf28-4099-8883-c04c61807dfa-istio-envoy\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-nsjcd\" (UID: \"7b57a22a-cf28-4099-8883-c04c61807dfa\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-nsjcd"
Nov 03 20:24:37.492116 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:37.484470    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b57a22a-cf28-4099-8883-c04c61807dfa-credential-socket\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-nsjcd\" (UID: \"7b57a22a-cf28-4099-8883-c04c61807dfa\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-nsjcd"
Nov 03 20:24:37.509797 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:37.508976    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5vqm\" (UniqueName: \"kubernetes.io/projected/7b57a22a-cf28-4099-8883-c04c61807dfa-kube-api-access-f5vqm\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-nsjcd\" (UID: \"7b57a22a-cf28-4099-8883-c04c61807dfa\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-nsjcd"
Nov 03 20:24:37.532846 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:37.518950    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b57a22a-cf28-4099-8883-c04c61807dfa-istio-token\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-nsjcd\" (UID: \"7b57a22a-cf28-4099-8883-c04c61807dfa\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-nsjcd"
Nov 03 20:24:37.622889 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:37.614435    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-nsjcd"
Nov 03 20:24:38.278413 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:38.278385    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-nsjcd"]
Nov 03 20:24:38.278733 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:24:38.278447    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b57a22a_cf28_4099_8883_c04c61807dfa.slice/crio-2295b4345521a6856cdd923c5ce86f588a760b8eea207412781577b5435570a3 WatchSource:0}: Error finding container 2295b4345521a6856cdd923c5ce86f588a760b8eea207412781577b5435570a3: Status 404 returned error can't find the container with id 2295b4345521a6856cdd923c5ce86f588a760b8eea207412781577b5435570a3
Nov 03 20:24:38.281186 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:38.281151    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:24:38.281299 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:38.281233    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:24:38.281299 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:38.281279    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:24:38.894800 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:38.886848    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-nsjcd" event={"ID":"7b57a22a-cf28-4099-8883-c04c61807dfa","Type":"ContainerStarted","Data":"e0359761c7f58587a7e9e0c308a42f2df94efc0661527121e1458f5db9608599"}
Nov 03 20:24:38.894800 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:38.886914    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-nsjcd" event={"ID":"7b57a22a-cf28-4099-8883-c04c61807dfa","Type":"ContainerStarted","Data":"2295b4345521a6856cdd923c5ce86f588a760b8eea207412781577b5435570a3"}
Nov 03 20:24:38.982146 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:38.982089    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-nsjcd" podStartSLOduration=1.982064971 podStartE2EDuration="1.982064971s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:24:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:24:38.967699667 +0000 UTC m=+1213.120361580" watchObservedRunningTime="2026-11-03 20:24:38.982064971 +0000 UTC m=+1213.134726858"
Nov 03 20:24:39.616049 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:39.616013    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-nsjcd"
Nov 03 20:24:39.622994 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:39.622968    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-nsjcd"
Nov 03 20:24:39.894760 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:39.894666    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-nsjcd"
Nov 03 20:24:39.895906 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:39.895881    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-nsjcd"
Nov 03 20:24:43.179420 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:43.179383    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-eb87f429-6b49f7957f-k2q4j"]
Nov 03 20:24:43.188842 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:43.188805    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-eb87f429-6b49f7957f-k2q4j"
Nov 03 20:24:43.201384 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:43.201355    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-eb87f429-6b49f7957f-k2q4j"]
Nov 03 20:24:43.208695 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:43.208661    5570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-eb87f429\""
Nov 03 20:24:43.208695 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:43.208672    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-eb87f429-dockercfg-fxjz4\""
Nov 03 20:24:43.332635 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:43.313484    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9zwv\" (UniqueName: \"kubernetes.io/projected/4711b06e-dba0-4d7a-a217-defa6ab8f008-kube-api-access-g9zwv\") pod \"payload-pre-processing-e2e-mt-eb87f429-6b49f7957f-k2q4j\" (UID: \"4711b06e-dba0-4d7a-a217-defa6ab8f008\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-eb87f429-6b49f7957f-k2q4j"
Nov 03 20:24:43.390944 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:43.390905    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4711b06e-dba0-4d7a-a217-defa6ab8f008-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-eb87f429-6b49f7957f-k2q4j\" (UID: \"4711b06e-dba0-4d7a-a217-defa6ab8f008\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-eb87f429-6b49f7957f-k2q4j"
Nov 03 20:24:43.547799 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:43.528114    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4711b06e-dba0-4d7a-a217-defa6ab8f008-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-eb87f429-6b49f7957f-k2q4j\" (UID: \"4711b06e-dba0-4d7a-a217-defa6ab8f008\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-eb87f429-6b49f7957f-k2q4j"
Nov 03 20:24:43.548704 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:43.548661    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9zwv\" (UniqueName: \"kubernetes.io/projected/4711b06e-dba0-4d7a-a217-defa6ab8f008-kube-api-access-g9zwv\") pod \"payload-pre-processing-e2e-mt-eb87f429-6b49f7957f-k2q4j\" (UID: \"4711b06e-dba0-4d7a-a217-defa6ab8f008\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-eb87f429-6b49f7957f-k2q4j"
Nov 03 20:24:43.620111 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:43.614536    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4711b06e-dba0-4d7a-a217-defa6ab8f008-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-eb87f429-6b49f7957f-k2q4j\" (UID: \"4711b06e-dba0-4d7a-a217-defa6ab8f008\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-eb87f429-6b49f7957f-k2q4j"
Nov 03 20:24:43.691063 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:43.691016    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9zwv\" (UniqueName: \"kubernetes.io/projected/4711b06e-dba0-4d7a-a217-defa6ab8f008-kube-api-access-g9zwv\") pod \"payload-pre-processing-e2e-mt-eb87f429-6b49f7957f-k2q4j\" (UID: \"4711b06e-dba0-4d7a-a217-defa6ab8f008\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-eb87f429-6b49f7957f-k2q4j"
Nov 03 20:24:43.811811 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:43.811700    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-eb87f429-6b49f7957f-k2q4j"
Nov 03 20:24:44.121415 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:44.121386    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-eb87f429-6b49f7957f-k2q4j"]
Nov 03 20:24:45.051838 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:45.045070    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-eb87f429-6b49f7957f-k2q4j" event={"ID":"4711b06e-dba0-4d7a-a217-defa6ab8f008","Type":"ContainerStarted","Data":"8352b9b1531ef87160ee04df65255c2b018105116c1b1b760f1becb449967e32"}
Nov 03 20:24:45.051838 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:45.045144    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-eb87f429-6b49f7957f-k2q4j" event={"ID":"4711b06e-dba0-4d7a-a217-defa6ab8f008","Type":"ContainerStarted","Data":"4ba5bba4799cf4c87494df665c2b48615e428452485794c642eb8881cfd62403"}
Nov 03 20:24:45.051838 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:45.046387    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-eb87f429-6b49f7957f-k2q4j"
Nov 03 20:24:45.095009 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:45.094934    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-eb87f429-6b49f7957f-k2q4j" podStartSLOduration=2.094912507 podStartE2EDuration="2.094912507s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:24:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:24:45.091054897 +0000 UTC m=+1219.243716784" watchObservedRunningTime="2026-11-03 20:24:45.094912507 +0000 UTC m=+1219.247574395"
Nov 03 20:24:45.772486 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:45.772453    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-e3ba87-7c7fd945bd-dzmfm"
Nov 03 20:24:57.089847 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:57.082950    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-eb87f429-6b49f7957f-k2q4j"
Nov 03 20:24:57.555851 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:24:57.537922    5570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8bwhz" containerName="extractor"
Nov 03 20:24:58.948165 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:58.948113    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/premium-w2-9973d9-kserve-5c6dd4669c-5gv8x" podUID="cefc89ad-7ae1-433b-979d-6601678a6d98" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Nov 03 20:24:58.949135 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:58.949107    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh" podUID="2d3647f2-89e9-4908-a8ee-d84e527abeb0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Nov 03 20:24:58.985004 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:58.984955    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/distinct-w3-482722-kserve-d876967c-kd488" podUID="9a97f9cc-2261-471d-9053-51ee965850bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:24:59.066520 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:58.996933    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/distinct-2-w0-e40816-kserve-79fb46654d-s242x" podUID="4ad9b048-925c-4d01-a839-a5f25d4dd02d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:24:59.067432 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:59.008303    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz" podUID="3a715ef1-38e9-4386-9eec-e10e8df11a10" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:24:59.104621 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:59.104579    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/premium-w4-e80203-kserve-5f9f445456-bms9d" podUID="6ad76efc-20c2-4951-b22a-c5a39cd39ed4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:24:59.125764 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:59.125707    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/premium-w1-6fc748-kserve-7474db9cd7-nf2xk" podUID="830e062e-83fc-4de7-91a7-5534dac1d46b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:24:59.372861 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:59.361391    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/distinct-2-w4-e80203-kserve-756f9b9bc6-69gxj" podUID="ccf5dc2f-f116-4917-bb1a-b136d8ed17f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:24:59.393232 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:59.393188    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/premium-w3-482722-kserve-689dc7bd5d-x8rjp" podUID="369045c3-4d6a-49a7-9bc6-f2939883bee8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:24:59.411962 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:59.411919    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/simulator-w0-e40816-kserve-688f79796f-2tvw5" podUID="c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:24:59.515805 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:59.512461    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/distinct-2-w3-482722-kserve-74bf58cf7d-v9cjw" podUID="9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:24:59.659855 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:24:59.659225    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-2-w6-9e1bcb-kserve-7d96d9f6f6-4f7fl" podUID="56353b0f-08ee-4ada-9dc0-ac5afd5a59c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:25:00.138803 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:00.138500    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/premium-w1-6fc748-kserve-7474db9cd7-nf2xk" podUID="830e062e-83fc-4de7-91a7-5534dac1d46b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:25:00.139326 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:00.139146    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz" podUID="3a715ef1-38e9-4386-9eec-e10e8df11a10" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:25:00.148793 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:00.139790    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6fc748/premium-w1-6fc748-kserve-7474db9cd7-nf2xk"
Nov 03 20:25:00.170799 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:00.151398    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/premium-w2-9973d9-kserve-5c6dd4669c-5gv8x" podUID="cefc89ad-7ae1-433b-979d-6601678a6d98" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Nov 03 20:25:00.340593 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:00.171161    5570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-e80203/distinct-2-w4-e80203-kserve-756f9b9bc6-69gxj" podUID="ccf5dc2f-f116-4917-bb1a-b136d8ed17f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:25:00.340593 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:00.158935    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/distinct-2-w0-e40816-kserve-79fb46654d-s242x" podUID="4ad9b048-925c-4d01-a839-a5f25d4dd02d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:25:00.340593 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:00.157854    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh" podUID="2d3647f2-89e9-4908-a8ee-d84e527abeb0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Nov 03 20:25:00.409852 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:00.380908    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/distinct-2-w4-e80203-kserve-756f9b9bc6-69gxj" podUID="ccf5dc2f-f116-4917-bb1a-b136d8ed17f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded"
Nov 03 20:25:00.410523 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:00.408897    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/premium-w3-482722-kserve-689dc7bd5d-x8rjp" podUID="369045c3-4d6a-49a7-9bc6-f2939883bee8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:25:00.442844 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:00.432867    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/simulator-w0-e40816-kserve-688f79796f-2tvw5" podUID="c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:25:00.553387 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:00.553334    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/distinct-2-w3-482722-kserve-74bf58cf7d-v9cjw" podUID="9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:25:00.590969 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:00.590926    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz"
Nov 03 20:25:00.591263 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:00.591247    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9973d9/premium-w2-9973d9-kserve-5c6dd4669c-5gv8x"
Nov 03 20:25:00.591442 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:00.591429    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e40816/distinct-2-w0-e40816-kserve-79fb46654d-s242x"
Nov 03 20:25:00.591614 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:00.591600    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-9e1bcb/premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh"
Nov 03 20:25:00.591796 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:00.591753    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-e80203/distinct-2-w4-e80203-kserve-756f9b9bc6-69gxj"
Nov 03 20:25:00.591954 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:00.591941    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-482722/premium-w3-482722-kserve-689dc7bd5d-x8rjp"
Nov 03 20:25:00.592195 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:00.592180    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e40816/simulator-w0-e40816-kserve-688f79796f-2tvw5"
Nov 03 20:25:00.592343 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:00.592331    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6fc748/premium-w1-6fc748-kserve-7474db9cd7-nf2xk"
Nov 03 20:25:00.592469 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:00.592457    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-e80203/distinct-2-w4-e80203-kserve-756f9b9bc6-69gxj"
Nov 03 20:25:00.592583 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:00.592572    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9973d9/premium-w2-9973d9-kserve-5c6dd4669c-5gv8x"
Nov 03 20:25:00.592707 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:00.592696    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-482722/distinct-2-w3-482722-kserve-74bf58cf7d-v9cjw"
Nov 03 20:25:00.592870 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:00.592850    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz"
Nov 03 20:25:00.593052 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:00.593036    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e40816/simulator-w0-e40816-kserve-688f79796f-2tvw5"
Nov 03 20:25:00.597689 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:00.597651    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-9e1bcb/premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh"
Nov 03 20:25:00.871859 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:00.865298    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-482722/premium-w3-482722-kserve-689dc7bd5d-x8rjp"
Nov 03 20:25:00.930792 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:00.924971    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-482722/distinct-2-w3-482722-kserve-74bf58cf7d-v9cjw"
Nov 03 20:25:00.930792 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:00.925406    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e40816/distinct-2-w0-e40816-kserve-79fb46654d-s242x"
Nov 03 20:25:04.063128 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:04.063093    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-eb87f429-59cb4f857f-d25tg"]
Nov 03 20:25:04.067946 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:04.067922    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-eb87f429-59cb4f857f-d25tg"
Nov 03 20:25:04.092145 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:04.092110    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-eb87f429-59cb4f857f-d25tg"]
Nov 03 20:25:04.100185 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:04.100143    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-eb87f429-serving-cert\""
Nov 03 20:25:04.188993 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:04.188957    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/257fbf5f-f3b8-4ac2-a7f2-8032a68526db-metrics-certs\") pod \"maas-api-e2e-mt-eb87f429-59cb4f857f-d25tg\" (UID: \"257fbf5f-f3b8-4ac2-a7f2-8032a68526db\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-eb87f429-59cb4f857f-d25tg"
Nov 03 20:25:04.189191 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:04.189167    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/257fbf5f-f3b8-4ac2-a7f2-8032a68526db-openshift-service-ca\") pod \"maas-api-e2e-mt-eb87f429-59cb4f857f-d25tg\" (UID: \"257fbf5f-f3b8-4ac2-a7f2-8032a68526db\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-eb87f429-59cb4f857f-d25tg"
Nov 03 20:25:04.189279 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:04.189268    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmxn4\" (UniqueName: \"kubernetes.io/projected/257fbf5f-f3b8-4ac2-a7f2-8032a68526db-kube-api-access-tmxn4\") pod \"maas-api-e2e-mt-eb87f429-59cb4f857f-d25tg\" (UID: \"257fbf5f-f3b8-4ac2-a7f2-8032a68526db\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-eb87f429-59cb4f857f-d25tg"
Nov 03 20:25:04.189331 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:04.189320    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/257fbf5f-f3b8-4ac2-a7f2-8032a68526db-maas-api-tls\") pod \"maas-api-e2e-mt-eb87f429-59cb4f857f-d25tg\" (UID: \"257fbf5f-f3b8-4ac2-a7f2-8032a68526db\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-eb87f429-59cb4f857f-d25tg"
Nov 03 20:25:04.290963 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:04.290928    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/257fbf5f-f3b8-4ac2-a7f2-8032a68526db-metrics-certs\") pod \"maas-api-e2e-mt-eb87f429-59cb4f857f-d25tg\" (UID: \"257fbf5f-f3b8-4ac2-a7f2-8032a68526db\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-eb87f429-59cb4f857f-d25tg"
Nov 03 20:25:04.291170 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:04.291144    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/257fbf5f-f3b8-4ac2-a7f2-8032a68526db-openshift-service-ca\") pod \"maas-api-e2e-mt-eb87f429-59cb4f857f-d25tg\" (UID: \"257fbf5f-f3b8-4ac2-a7f2-8032a68526db\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-eb87f429-59cb4f857f-d25tg"
Nov 03 20:25:04.291259 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:04.291240    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmxn4\" (UniqueName: \"kubernetes.io/projected/257fbf5f-f3b8-4ac2-a7f2-8032a68526db-kube-api-access-tmxn4\") pod \"maas-api-e2e-mt-eb87f429-59cb4f857f-d25tg\" (UID: \"257fbf5f-f3b8-4ac2-a7f2-8032a68526db\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-eb87f429-59cb4f857f-d25tg"
Nov 03 20:25:04.291357 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:04.291339    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/257fbf5f-f3b8-4ac2-a7f2-8032a68526db-maas-api-tls\") pod \"maas-api-e2e-mt-eb87f429-59cb4f857f-d25tg\" (UID: \"257fbf5f-f3b8-4ac2-a7f2-8032a68526db\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-eb87f429-59cb4f857f-d25tg"
Nov 03 20:25:04.292042 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:04.292005    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/257fbf5f-f3b8-4ac2-a7f2-8032a68526db-openshift-service-ca\") pod \"maas-api-e2e-mt-eb87f429-59cb4f857f-d25tg\" (UID: \"257fbf5f-f3b8-4ac2-a7f2-8032a68526db\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-eb87f429-59cb4f857f-d25tg"
Nov 03 20:25:04.294523 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:04.294253    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/257fbf5f-f3b8-4ac2-a7f2-8032a68526db-maas-api-tls\") pod \"maas-api-e2e-mt-eb87f429-59cb4f857f-d25tg\" (UID: \"257fbf5f-f3b8-4ac2-a7f2-8032a68526db\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-eb87f429-59cb4f857f-d25tg"
Nov 03 20:25:04.294523 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:04.294400    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/257fbf5f-f3b8-4ac2-a7f2-8032a68526db-metrics-certs\") pod \"maas-api-e2e-mt-eb87f429-59cb4f857f-d25tg\" (UID: \"257fbf5f-f3b8-4ac2-a7f2-8032a68526db\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-eb87f429-59cb4f857f-d25tg"
Nov 03 20:25:04.342171 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:04.336534    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmxn4\" (UniqueName: \"kubernetes.io/projected/257fbf5f-f3b8-4ac2-a7f2-8032a68526db-kube-api-access-tmxn4\") pod \"maas-api-e2e-mt-eb87f429-59cb4f857f-d25tg\" (UID: \"257fbf5f-f3b8-4ac2-a7f2-8032a68526db\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-eb87f429-59cb4f857f-d25tg"
Nov 03 20:25:04.437854 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:04.437819    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-eb87f429-59cb4f857f-d25tg"
Nov 03 20:25:04.929558 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:25:04.927529    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod257fbf5f_f3b8_4ac2_a7f2_8032a68526db.slice/crio-f717ed30d3f0812c7cc52ce26f5f375eb8a2dd9457f328b89832ec7fe169639f WatchSource:0}: Error finding container f717ed30d3f0812c7cc52ce26f5f375eb8a2dd9457f328b89832ec7fe169639f: Status 404 returned error can't find the container with id f717ed30d3f0812c7cc52ce26f5f375eb8a2dd9457f328b89832ec7fe169639f
Nov 03 20:25:04.929558 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:04.929014    5570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 20:25:04.955857 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:04.945095    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-eb87f429-59cb4f857f-d25tg"]
Nov 03 20:25:05.037792 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:05.035045    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50725cca-29cpj"]
Nov 03 20:25:05.040283 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:05.040258    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50725cca-29cpj"]
Nov 03 20:25:05.738017 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:05.737914    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-eb87f429-59cb4f857f-d25tg" event={"ID":"257fbf5f-f3b8-4ac2-a7f2-8032a68526db","Type":"ContainerStarted","Data":"adc2adfa52bf7342d17e4b9f29b8b24917b5d3a4f45729f13c94f35aa82ea52e"}
Nov 03 20:25:05.738491 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:05.738019    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-eb87f429-59cb4f857f-d25tg" event={"ID":"257fbf5f-f3b8-4ac2-a7f2-8032a68526db","Type":"ContainerStarted","Data":"f717ed30d3f0812c7cc52ce26f5f375eb8a2dd9457f328b89832ec7fe169639f"}
Nov 03 20:25:05.738491 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:05.738163    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-eb87f429-59cb4f857f-d25tg"
Nov 03 20:25:05.783935 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:05.781156    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-eb87f429-59cb4f857f-d25tg" podStartSLOduration=1.215991591 podStartE2EDuration="1.781136823s" totalImagesPullingTime="565.145232ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:25:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:25:05.780619804 +0000 UTC m=+1239.933281692" watchObservedRunningTime="2026-11-03 20:25:05.781136823 +0000 UTC m=+1239.933798711"
Nov 03 20:25:05.973863 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:05.973824    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c5d552d-f9a3-40f2-8742-95e484f55ef1" path="/var/lib/kubelet/pods/4c5d552d-f9a3-40f2-8742-95e484f55ef1/volumes"
Nov 03 20:25:11.756687 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:11.756653    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-eb87f429-59cb4f857f-d25tg"
Nov 03 20:25:23.098435 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:23.098382    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0afe7ff-hlp9w"]
Nov 03 20:25:23.134769 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:23.134535    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0afe7ff-hlp9w"]
Nov 03 20:25:23.966440 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:23.966407    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a935f9f-1413-4063-9622-1147b8b80dbf" path="/var/lib/kubelet/pods/0a935f9f-1413-4063-9622-1147b8b80dbf/volumes"
Nov 03 20:25:27.231452 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:25:27.231411    5570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8bwhz" containerName="extractor"
Nov 03 20:25:31.903718 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:31.903681    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g69f8"]
Nov 03 20:25:34.928644 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:34.928597    5570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-482722/distinct-w3-482722-kserve-d876967c-kd488" podUID="9a97f9cc-2261-471d-9053-51ee965850bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/health\": context deadline exceeded"
Nov 03 20:25:34.929941 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:34.929909    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/premium-w3-482722-kserve-689dc7bd5d-x8rjp" podUID="369045c3-4d6a-49a7-9bc6-f2939883bee8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:25:35.024973 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:35.024922    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/distinct-w3-482722-kserve-d876967c-kd488" podUID="9a97f9cc-2261-471d-9053-51ee965850bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:25:35.134925 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:35.134872    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/premium-w4-e80203-kserve-5f9f445456-bms9d" podUID="6ad76efc-20c2-4951-b22a-c5a39cd39ed4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Nov 03 20:25:35.432676 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:35.432632    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/simulator-w0-e40816-kserve-688f79796f-2tvw5" podUID="c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Nov 03 20:25:35.442816 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:35.440498    5570 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz" podUID="3a715ef1-38e9-4386-9eec-e10e8df11a10" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/health\": context deadline exceeded"
Nov 03 20:25:35.561468 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:35.561425    5570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-482722/premium-w3-482722-kserve-689dc7bd5d-x8rjp" podUID="369045c3-4d6a-49a7-9bc6-f2939883bee8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:25:35.569983 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:35.569937    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/premium-w1-6fc748-kserve-7474db9cd7-nf2xk" podUID="830e062e-83fc-4de7-91a7-5534dac1d46b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:25:35.587536 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:35.587475    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-2-w6-9e1bcb-kserve-7d96d9f6f6-4f7fl" podUID="56353b0f-08ee-4ada-9dc0-ac5afd5a59c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded"
Nov 03 20:25:35.631947 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:35.631881    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh" podUID="2d3647f2-89e9-4908-a8ee-d84e527abeb0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:25:35.950799 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:35.949883    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/premium-w3-482722-kserve-689dc7bd5d-x8rjp" podUID="369045c3-4d6a-49a7-9bc6-f2939883bee8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:25:35.950799 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:35.950060    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/premium-w2-9973d9-kserve-5c6dd4669c-5gv8x" podUID="cefc89ad-7ae1-433b-979d-6601678a6d98" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:25:35.950799 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:35.950437    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/distinct-2-w0-e40816-kserve-79fb46654d-s242x" podUID="4ad9b048-925c-4d01-a839-a5f25d4dd02d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded"
Nov 03 20:25:35.950799 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:35.950727    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/distinct-2-w3-482722-kserve-74bf58cf7d-v9cjw" podUID="9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:25:35.955810 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:35.951582    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-482722/premium-w3-482722-kserve-689dc7bd5d-x8rjp"
Nov 03 20:25:36.004020 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:36.003975    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz" podUID="3a715ef1-38e9-4386-9eec-e10e8df11a10" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:25:36.025573 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:36.025529    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/distinct-w3-482722-kserve-d876967c-kd488" podUID="9a97f9cc-2261-471d-9053-51ee965850bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:25:36.168291 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:36.168240    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/premium-w4-e80203-kserve-5f9f445456-bms9d" podUID="6ad76efc-20c2-4951-b22a-c5a39cd39ed4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:25:36.232494 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:36.232397    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-482722/distinct-w3-482722-kserve-d876967c-kd488"
Nov 03 20:25:36.284844 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:36.259379    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-e80203/premium-w4-e80203-kserve-5f9f445456-bms9d"
Nov 03 20:25:36.856056 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:36.570454    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/premium-w1-6fc748-kserve-7474db9cd7-nf2xk" podUID="830e062e-83fc-4de7-91a7-5534dac1d46b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:25:36.856056 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:36.611106    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6fc748/premium-w1-6fc748-kserve-7474db9cd7-nf2xk"
Nov 03 20:25:36.856056 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:36.643054    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/distinct-2-w4-e80203-kserve-756f9b9bc6-69gxj" podUID="ccf5dc2f-f116-4917-bb1a-b136d8ed17f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded"
Nov 03 20:25:36.856056 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:36.662924    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/simulator-w0-e40816-kserve-688f79796f-2tvw5" podUID="c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:25:36.856056 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:36.663272    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e40816/simulator-w0-e40816-kserve-688f79796f-2tvw5"
Nov 03 20:25:36.856056 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:36.717618    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-2-w6-9e1bcb-kserve-7d96d9f6f6-4f7fl" podUID="56353b0f-08ee-4ada-9dc0-ac5afd5a59c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded"
Nov 03 20:25:36.856056 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:36.727925    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-2-w6-9e1bcb-kserve-7d96d9f6f6-4f7fl"
Nov 03 20:25:36.856056 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:36.764179    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh" podUID="2d3647f2-89e9-4908-a8ee-d84e527abeb0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:25:36.856056 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:36.764738    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-9e1bcb/premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh"
Nov 03 20:25:36.912481 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:36.912125    5570 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-2x92x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:25:36.912481 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:36.912190    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-2x92x" podUID="2ac2fc9c-0a3e-4326-a4f8-8856b1a30e36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:25:36.968800 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:36.956882    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/distinct-2-w3-482722-kserve-74bf58cf7d-v9cjw" podUID="9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:25:36.968800 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:36.959962    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-482722/distinct-2-w3-482722-kserve-74bf58cf7d-v9cjw"
Nov 03 20:25:37.003194 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:36.997881    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/distinct-2-w0-e40816-kserve-79fb46654d-s242x" podUID="4ad9b048-925c-4d01-a839-a5f25d4dd02d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:25:37.040948 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:37.003484    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/premium-w3-482722-kserve-689dc7bd5d-x8rjp" podUID="369045c3-4d6a-49a7-9bc6-f2939883bee8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Nov 03 20:25:37.040948 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:37.017132    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/premium-w2-9973d9-kserve-5c6dd4669c-5gv8x" podUID="cefc89ad-7ae1-433b-979d-6601678a6d98" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Nov 03 20:25:37.040948 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:37.020037    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e40816/distinct-2-w0-e40816-kserve-79fb46654d-s242x"
Nov 03 20:25:37.040948 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:37.020127    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9973d9/premium-w2-9973d9-kserve-5c6dd4669c-5gv8x"
Nov 03 20:25:37.101042 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:37.100799    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/distinct-w3-482722-kserve-d876967c-kd488" podUID="9a97f9cc-2261-471d-9053-51ee965850bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:25:37.102641 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:37.102584    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz" podUID="3a715ef1-38e9-4386-9eec-e10e8df11a10" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:25:37.286133 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:37.103082    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz"
Nov 03 20:25:37.286133 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:37.150185    5570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-2-w6-9e1bcb-kserve-7d96d9f6f6-4f7fl" podUID="56353b0f-08ee-4ada-9dc0-ac5afd5a59c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:25:37.286133 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:37.261703    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/premium-w4-e80203-kserve-5f9f445456-bms9d" podUID="6ad76efc-20c2-4951-b22a-c5a39cd39ed4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Nov 03 20:25:37.584903 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:37.574874    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/premium-w1-6fc748-kserve-7474db9cd7-nf2xk" podUID="830e062e-83fc-4de7-91a7-5534dac1d46b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:25:37.709187 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:37.669935    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/simulator-w0-e40816-kserve-688f79796f-2tvw5" podUID="c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:25:37.718211 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:37.718166    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/distinct-2-w4-e80203-kserve-756f9b9bc6-69gxj" podUID="ccf5dc2f-f116-4917-bb1a-b136d8ed17f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:25:37.722798 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:37.719038    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-e80203/distinct-2-w4-e80203-kserve-756f9b9bc6-69gxj"
Nov 03 20:25:37.746810 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:37.742926    5570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-9e1bcb/premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh" podUID="2d3647f2-89e9-4908-a8ee-d84e527abeb0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:25:37.749945 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:37.749881    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-2-w6-9e1bcb-kserve-7d96d9f6f6-4f7fl" podUID="56353b0f-08ee-4ada-9dc0-ac5afd5a59c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:25:37.764407 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:37.764355    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh" podUID="2d3647f2-89e9-4908-a8ee-d84e527abeb0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:25:38.032801 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:37.981356    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/distinct-2-w3-482722-kserve-74bf58cf7d-v9cjw" podUID="9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded"
Nov 03 20:25:38.208966 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:38.208916    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz" podUID="3a715ef1-38e9-4386-9eec-e10e8df11a10" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": context deadline exceeded"
Nov 03 20:25:38.726822 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:38.725229    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/simulator-w0-e40816-kserve-688f79796f-2tvw5" podUID="c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Nov 03 20:25:38.726822 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:38.449953    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-cpfzl" podUID="30b21f09-fd9e-4836-9fc3-c19f818c9022" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:25:38.772842 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:38.772795    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/premium-w1-6fc748-kserve-7474db9cd7-nf2xk" podUID="830e062e-83fc-4de7-91a7-5534dac1d46b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Nov 03 20:25:38.776654 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:38.724880    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/distinct-2-w4-e80203-kserve-756f9b9bc6-69gxj" podUID="ccf5dc2f-f116-4917-bb1a-b136d8ed17f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:25:38.806845 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:38.683308    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/distinct-2-w0-e40816-kserve-79fb46654d-s242x" podUID="4ad9b048-925c-4d01-a839-a5f25d4dd02d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded"
Nov 03 20:25:38.806845 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:38.683655    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/premium-w2-9973d9-kserve-5c6dd4669c-5gv8x" podUID="cefc89ad-7ae1-433b-979d-6601678a6d98" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Nov 03 20:25:38.876798 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:38.861149    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/premium-w3-482722-kserve-689dc7bd5d-x8rjp" podUID="369045c3-4d6a-49a7-9bc6-f2939883bee8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:25:39.119022 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:38.877196    5570 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-657vf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:25:39.119670 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:39.119631    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-657vf" podUID="f089dd20-d6a6-4c63-88a5-3fbe9a2ff315" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:25:39.119913 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:39.033279    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/premium-w4-e80203-kserve-5f9f445456-bms9d" podUID="6ad76efc-20c2-4951-b22a-c5a39cd39ed4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Nov 03 20:25:39.130168 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:39.031217    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/distinct-w3-482722-kserve-d876967c-kd488" podUID="9a97f9cc-2261-471d-9053-51ee965850bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded"
Nov 03 20:25:39.235871 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:39.110034    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/distinct-2-w3-482722-kserve-74bf58cf7d-v9cjw" podUID="9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded"
Nov 03 20:25:39.242876 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:39.237914    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh" podUID="2d3647f2-89e9-4908-a8ee-d84e527abeb0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Nov 03 20:25:39.243104 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:39.110456    5570 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-vdrfs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:25:39.243292 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:39.243259    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-vdrfs" podUID="1b70a87b-76a6-40e6-9ced-037c3ceee5cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:25:39.259726 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:39.259674    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-2-w6-9e1bcb-kserve-7d96d9f6f6-4f7fl" podUID="56353b0f-08ee-4ada-9dc0-ac5afd5a59c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:25:39.276448 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:39.036330    5570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-e40816/simulator-w0-e40816-kserve-688f79796f-2tvw5" podUID="c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": context deadline exceeded"
Nov 03 20:25:39.378307 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:25:39.378209    5570 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.395s"
Nov 03 20:25:39.592377 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:39.592340    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-76b894-openshift-default-6669d4d94f-vrq8z"]
Nov 03 20:25:39.726031 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:39.725964    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz" podUID="3a715ef1-38e9-4386-9eec-e10e8df11a10" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": context deadline exceeded"
Nov 03 20:25:39.727037 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:39.726992    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/simulator-w0-e40816-kserve-688f79796f-2tvw5" podUID="c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:25:39.780061 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:39.779873    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/distinct-2-w4-e80203-kserve-756f9b9bc6-69gxj" podUID="ccf5dc2f-f116-4917-bb1a-b136d8ed17f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:25:39.780293 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:39.780252    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/premium-w1-6fc748-kserve-7474db9cd7-nf2xk" podUID="830e062e-83fc-4de7-91a7-5534dac1d46b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Nov 03 20:25:39.808799 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:39.805920    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/distinct-2-w0-e40816-kserve-79fb46654d-s242x" podUID="4ad9b048-925c-4d01-a839-a5f25d4dd02d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:25:39.808799 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:39.806546    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/premium-w2-9973d9-kserve-5c6dd4669c-5gv8x" podUID="cefc89ad-7ae1-433b-979d-6601678a6d98" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:25:39.864421 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:39.864385    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-76b894-openshift-default-6669d4d94f-vrq8z"]
Nov 03 20:25:39.864810 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:39.864763    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-76b894-openshift-default-6669d4d94f-vrq8z"
Nov 03 20:25:40.135801 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:40.112601    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3db8ac79-76fc-4d46-ac4f-6536fe192b8f-credential-socket\") pod \"e2e-shared-a-w3-76b894-openshift-default-6669d4d94f-vrq8z\" (UID: \"3db8ac79-76fc-4d46-ac4f-6536fe192b8f\") " pod="openshift-ingress/e2e-shared-a-w3-76b894-openshift-default-6669d4d94f-vrq8z"
Nov 03 20:25:40.135801 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:40.112673    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3db8ac79-76fc-4d46-ac4f-6536fe192b8f-istio-envoy\") pod \"e2e-shared-a-w3-76b894-openshift-default-6669d4d94f-vrq8z\" (UID: \"3db8ac79-76fc-4d46-ac4f-6536fe192b8f\") " pod="openshift-ingress/e2e-shared-a-w3-76b894-openshift-default-6669d4d94f-vrq8z"
Nov 03 20:25:40.135801 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:40.112707    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7nl2\" (UniqueName: \"kubernetes.io/projected/3db8ac79-76fc-4d46-ac4f-6536fe192b8f-kube-api-access-v7nl2\") pod \"e2e-shared-a-w3-76b894-openshift-default-6669d4d94f-vrq8z\" (UID: \"3db8ac79-76fc-4d46-ac4f-6536fe192b8f\") " pod="openshift-ingress/e2e-shared-a-w3-76b894-openshift-default-6669d4d94f-vrq8z"
Nov 03 20:25:40.135801 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:40.112806    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3db8ac79-76fc-4d46-ac4f-6536fe192b8f-workload-socket\") pod \"e2e-shared-a-w3-76b894-openshift-default-6669d4d94f-vrq8z\" (UID: \"3db8ac79-76fc-4d46-ac4f-6536fe192b8f\") " pod="openshift-ingress/e2e-shared-a-w3-76b894-openshift-default-6669d4d94f-vrq8z"
Nov 03 20:25:40.135801 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:40.112860    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3db8ac79-76fc-4d46-ac4f-6536fe192b8f-istio-podinfo\") pod \"e2e-shared-a-w3-76b894-openshift-default-6669d4d94f-vrq8z\" (UID: \"3db8ac79-76fc-4d46-ac4f-6536fe192b8f\") " pod="openshift-ingress/e2e-shared-a-w3-76b894-openshift-default-6669d4d94f-vrq8z"
Nov 03 20:25:40.135801 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:40.112897    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3db8ac79-76fc-4d46-ac4f-6536fe192b8f-istio-data\") pod \"e2e-shared-a-w3-76b894-openshift-default-6669d4d94f-vrq8z\" (UID: \"3db8ac79-76fc-4d46-ac4f-6536fe192b8f\") " pod="openshift-ingress/e2e-shared-a-w3-76b894-openshift-default-6669d4d94f-vrq8z"
Nov 03 20:25:40.135801 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:40.113190    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3db8ac79-76fc-4d46-ac4f-6536fe192b8f-istiod-ca-cert\") pod \"e2e-shared-a-w3-76b894-openshift-default-6669d4d94f-vrq8z\" (UID: \"3db8ac79-76fc-4d46-ac4f-6536fe192b8f\") " pod="openshift-ingress/e2e-shared-a-w3-76b894-openshift-default-6669d4d94f-vrq8z"
Nov 03 20:25:40.135801 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:40.113228    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3db8ac79-76fc-4d46-ac4f-6536fe192b8f-istio-token\") pod \"e2e-shared-a-w3-76b894-openshift-default-6669d4d94f-vrq8z\" (UID: \"3db8ac79-76fc-4d46-ac4f-6536fe192b8f\") " pod="openshift-ingress/e2e-shared-a-w3-76b894-openshift-default-6669d4d94f-vrq8z"
Nov 03 20:25:40.135801 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:40.113323    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3db8ac79-76fc-4d46-ac4f-6536fe192b8f-workload-certs\") pod \"e2e-shared-a-w3-76b894-openshift-default-6669d4d94f-vrq8z\" (UID: \"3db8ac79-76fc-4d46-ac4f-6536fe192b8f\") " pod="openshift-ingress/e2e-shared-a-w3-76b894-openshift-default-6669d4d94f-vrq8z"
Nov 03 20:25:40.135801 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:40.114038    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-76b894-openshift-default-dockercfg-djk5r\""
Nov 03 20:25:40.143420 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:40.137910    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/premium-w4-e80203-kserve-5f9f445456-bms9d" podUID="6ad76efc-20c2-4951-b22a-c5a39cd39ed4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:25:40.143420 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:40.138345    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/premium-w3-482722-kserve-689dc7bd5d-x8rjp" podUID="369045c3-4d6a-49a7-9bc6-f2939883bee8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Nov 03 20:25:40.243799 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:40.234458    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3db8ac79-76fc-4d46-ac4f-6536fe192b8f-istiod-ca-cert\") pod \"e2e-shared-a-w3-76b894-openshift-default-6669d4d94f-vrq8z\" (UID: \"3db8ac79-76fc-4d46-ac4f-6536fe192b8f\") " pod="openshift-ingress/e2e-shared-a-w3-76b894-openshift-default-6669d4d94f-vrq8z"
Nov 03 20:25:40.244209 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:40.244164    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3db8ac79-76fc-4d46-ac4f-6536fe192b8f-istio-token\") pod \"e2e-shared-a-w3-76b894-openshift-default-6669d4d94f-vrq8z\" (UID: \"3db8ac79-76fc-4d46-ac4f-6536fe192b8f\") " pod="openshift-ingress/e2e-shared-a-w3-76b894-openshift-default-6669d4d94f-vrq8z"
Nov 03 20:25:40.244953 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:40.244914    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3db8ac79-76fc-4d46-ac4f-6536fe192b8f-workload-certs\") pod \"e2e-shared-a-w3-76b894-openshift-default-6669d4d94f-vrq8z\" (UID: \"3db8ac79-76fc-4d46-ac4f-6536fe192b8f\") " pod="openshift-ingress/e2e-shared-a-w3-76b894-openshift-default-6669d4d94f-vrq8z"
Nov 03 20:25:40.248972 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:40.239750    5570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-e80203/distinct-2-w4-e80203-kserve-756f9b9bc6-69gxj" podUID="ccf5dc2f-f116-4917-bb1a-b136d8ed17f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:25:40.249365 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:40.239833    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/distinct-2-w3-482722-kserve-74bf58cf7d-v9cjw" podUID="9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:25:40.261450 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:40.239625    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3db8ac79-76fc-4d46-ac4f-6536fe192b8f-istiod-ca-cert\") pod \"e2e-shared-a-w3-76b894-openshift-default-6669d4d94f-vrq8z\" (UID: \"3db8ac79-76fc-4d46-ac4f-6536fe192b8f\") " pod="openshift-ingress/e2e-shared-a-w3-76b894-openshift-default-6669d4d94f-vrq8z"
Nov 03 20:25:40.261664 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:40.239720    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh" podUID="2d3647f2-89e9-4908-a8ee-d84e527abeb0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:25:40.262150 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:40.247178    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3db8ac79-76fc-4d46-ac4f-6536fe192b8f-workload-certs\") pod \"e2e-shared-a-w3-76b894-openshift-default-6669d4d94f-vrq8z\" (UID: \"3db8ac79-76fc-4d46-ac4f-6536fe192b8f\") " pod="openshift-ingress/e2e-shared-a-w3-76b894-openshift-default-6669d4d94f-vrq8z"
Nov 03 20:25:40.262550 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:40.262521    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-2-w6-9e1bcb-kserve-7d96d9f6f6-4f7fl" podUID="56353b0f-08ee-4ada-9dc0-ac5afd5a59c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:25:40.278717 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:40.278618    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/distinct-w3-482722-kserve-d876967c-kd488" podUID="9a97f9cc-2261-471d-9053-51ee965850bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:25:40.327396 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:40.327343    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3db8ac79-76fc-4d46-ac4f-6536fe192b8f-credential-socket\") pod \"e2e-shared-a-w3-76b894-openshift-default-6669d4d94f-vrq8z\" (UID: \"3db8ac79-76fc-4d46-ac4f-6536fe192b8f\") " pod="openshift-ingress/e2e-shared-a-w3-76b894-openshift-default-6669d4d94f-vrq8z"
Nov 03 20:25:40.327706 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:40.327672    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3db8ac79-76fc-4d46-ac4f-6536fe192b8f-istio-envoy\") pod \"e2e-shared-a-w3-76b894-openshift-default-6669d4d94f-vrq8z\" (UID: \"3db8ac79-76fc-4d46-ac4f-6536fe192b8f\") " pod="openshift-ingress/e2e-shared-a-w3-76b894-openshift-default-6669d4d94f-vrq8z"
Nov 03 20:25:40.327901 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:40.327878    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7nl2\" (UniqueName: \"kubernetes.io/projected/3db8ac79-76fc-4d46-ac4f-6536fe192b8f-kube-api-access-v7nl2\") pod \"e2e-shared-a-w3-76b894-openshift-default-6669d4d94f-vrq8z\" (UID: \"3db8ac79-76fc-4d46-ac4f-6536fe192b8f\") " pod="openshift-ingress/e2e-shared-a-w3-76b894-openshift-default-6669d4d94f-vrq8z"
Nov 03 20:25:40.329214 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:40.328431    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3db8ac79-76fc-4d46-ac4f-6536fe192b8f-credential-socket\") pod \"e2e-shared-a-w3-76b894-openshift-default-6669d4d94f-vrq8z\" (UID: \"3db8ac79-76fc-4d46-ac4f-6536fe192b8f\") " pod="openshift-ingress/e2e-shared-a-w3-76b894-openshift-default-6669d4d94f-vrq8z"
Nov 03 20:25:40.329214 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:40.328859    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3db8ac79-76fc-4d46-ac4f-6536fe192b8f-istio-token\") pod \"e2e-shared-a-w3-76b894-openshift-default-6669d4d94f-vrq8z\" (UID: \"3db8ac79-76fc-4d46-ac4f-6536fe192b8f\") " pod="openshift-ingress/e2e-shared-a-w3-76b894-openshift-default-6669d4d94f-vrq8z"
Nov 03 20:25:40.329814 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:40.329762    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3db8ac79-76fc-4d46-ac4f-6536fe192b8f-workload-socket\") pod \"e2e-shared-a-w3-76b894-openshift-default-6669d4d94f-vrq8z\" (UID: \"3db8ac79-76fc-4d46-ac4f-6536fe192b8f\") " pod="openshift-ingress/e2e-shared-a-w3-76b894-openshift-default-6669d4d94f-vrq8z"
Nov 03 20:25:40.330004 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:40.329985    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3db8ac79-76fc-4d46-ac4f-6536fe192b8f-istio-podinfo\") pod \"e2e-shared-a-w3-76b894-openshift-default-6669d4d94f-vrq8z\" (UID: \"3db8ac79-76fc-4d46-ac4f-6536fe192b8f\") " pod="openshift-ingress/e2e-shared-a-w3-76b894-openshift-default-6669d4d94f-vrq8z"
Nov 03 20:25:40.330448 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:40.330428    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3db8ac79-76fc-4d46-ac4f-6536fe192b8f-istio-data\") pod \"e2e-shared-a-w3-76b894-openshift-default-6669d4d94f-vrq8z\" (UID: \"3db8ac79-76fc-4d46-ac4f-6536fe192b8f\") " pod="openshift-ingress/e2e-shared-a-w3-76b894-openshift-default-6669d4d94f-vrq8z"
Nov 03 20:25:40.330873 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:40.330839    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3db8ac79-76fc-4d46-ac4f-6536fe192b8f-workload-socket\") pod \"e2e-shared-a-w3-76b894-openshift-default-6669d4d94f-vrq8z\" (UID: \"3db8ac79-76fc-4d46-ac4f-6536fe192b8f\") " pod="openshift-ingress/e2e-shared-a-w3-76b894-openshift-default-6669d4d94f-vrq8z"
Nov 03 20:25:40.338739 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:40.338717    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3db8ac79-76fc-4d46-ac4f-6536fe192b8f-istio-data\") pod \"e2e-shared-a-w3-76b894-openshift-default-6669d4d94f-vrq8z\" (UID: \"3db8ac79-76fc-4d46-ac4f-6536fe192b8f\") " pod="openshift-ingress/e2e-shared-a-w3-76b894-openshift-default-6669d4d94f-vrq8z"
Nov 03 20:25:40.489003 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:40.488962    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3db8ac79-76fc-4d46-ac4f-6536fe192b8f-istio-envoy\") pod \"e2e-shared-a-w3-76b894-openshift-default-6669d4d94f-vrq8z\" (UID: \"3db8ac79-76fc-4d46-ac4f-6536fe192b8f\") " pod="openshift-ingress/e2e-shared-a-w3-76b894-openshift-default-6669d4d94f-vrq8z"
Nov 03 20:25:40.521940 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:40.495438    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3db8ac79-76fc-4d46-ac4f-6536fe192b8f-istio-podinfo\") pod \"e2e-shared-a-w3-76b894-openshift-default-6669d4d94f-vrq8z\" (UID: \"3db8ac79-76fc-4d46-ac4f-6536fe192b8f\") " pod="openshift-ingress/e2e-shared-a-w3-76b894-openshift-default-6669d4d94f-vrq8z"
Nov 03 20:25:40.610857 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:40.608514    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7nl2\" (UniqueName: \"kubernetes.io/projected/3db8ac79-76fc-4d46-ac4f-6536fe192b8f-kube-api-access-v7nl2\") pod \"e2e-shared-a-w3-76b894-openshift-default-6669d4d94f-vrq8z\" (UID: \"3db8ac79-76fc-4d46-ac4f-6536fe192b8f\") " pod="openshift-ingress/e2e-shared-a-w3-76b894-openshift-default-6669d4d94f-vrq8z"
Nov 03 20:25:40.662671 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:40.661495    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-76b894-openshift-default-6669d4d94f-vrq8z"
Nov 03 20:25:40.758023 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:40.757973    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/simulator-w0-e40816-kserve-688f79796f-2tvw5" podUID="c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:25:40.801798 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:40.760383    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz" podUID="3a715ef1-38e9-4386-9eec-e10e8df11a10" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": context deadline exceeded"
Nov 03 20:25:40.809118 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:40.802122    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/premium-w1-6fc748-kserve-7474db9cd7-nf2xk" podUID="830e062e-83fc-4de7-91a7-5534dac1d46b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:25:40.815844 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:40.802163    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/distinct-2-w4-e80203-kserve-756f9b9bc6-69gxj" podUID="ccf5dc2f-f116-4917-bb1a-b136d8ed17f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded"
Nov 03 20:25:40.816432 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:40.808575    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/premium-w2-9973d9-kserve-5c6dd4669c-5gv8x" podUID="cefc89ad-7ae1-433b-979d-6601678a6d98" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Nov 03 20:25:40.819801 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:40.808617    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/distinct-2-w0-e40816-kserve-79fb46654d-s242x" podUID="4ad9b048-925c-4d01-a839-a5f25d4dd02d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:25:41.137798 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:41.119155    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e40816/distinct-2-w0-e40816-kserve-79fb46654d-s242x"
Nov 03 20:25:41.140079 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:41.138419    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/premium-w4-e80203-kserve-5f9f445456-bms9d" podUID="6ad76efc-20c2-4951-b22a-c5a39cd39ed4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Nov 03 20:25:41.140079 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:41.138544    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/premium-w3-482722-kserve-689dc7bd5d-x8rjp" podUID="369045c3-4d6a-49a7-9bc6-f2939883bee8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Nov 03 20:25:41.370735 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:41.263927    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-2-w6-9e1bcb-kserve-7d96d9f6f6-4f7fl" podUID="56353b0f-08ee-4ada-9dc0-ac5afd5a59c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:25:41.371320 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:41.371279    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/distinct-2-w3-482722-kserve-74bf58cf7d-v9cjw" podUID="9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:25:41.371644 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:41.371575    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh" podUID="2d3647f2-89e9-4908-a8ee-d84e527abeb0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Nov 03 20:25:41.478954 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:41.457416    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/distinct-w3-482722-kserve-d876967c-kd488" podUID="9a97f9cc-2261-471d-9053-51ee965850bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded"
Nov 03 20:25:41.580106 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:41.580072    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e40816/simulator-w0-e40816-kserve-688f79796f-2tvw5"
Nov 03 20:25:41.609457 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:41.609425    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-482722/premium-w3-482722-kserve-689dc7bd5d-x8rjp"
Nov 03 20:25:41.688561 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:41.687210    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6fc748/premium-w1-6fc748-kserve-7474db9cd7-nf2xk"
Nov 03 20:25:41.700795 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:41.696085    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-482722/distinct-w3-482722-kserve-d876967c-kd488"
Nov 03 20:25:41.711109 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:41.711083    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-9e1bcb/premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh"
Nov 03 20:25:41.728359 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:41.728328    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9973d9/premium-w2-9973d9-kserve-5c6dd4669c-5gv8x"
Nov 03 20:25:41.731281 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:41.731250    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-2-w6-9e1bcb-kserve-7d96d9f6f6-4f7fl"
Nov 03 20:25:41.804862 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:41.750025    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-e80203/premium-w4-e80203-kserve-5f9f445456-bms9d"
Nov 03 20:25:41.888706 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:41.888663    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/distinct-2-w4-e80203-kserve-756f9b9bc6-69gxj" podUID="ccf5dc2f-f116-4917-bb1a-b136d8ed17f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:25:41.935480 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:41.935296    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-482722/distinct-2-w3-482722-kserve-74bf58cf7d-v9cjw"
Nov 03 20:25:41.999862 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:41.958369    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz" podUID="3a715ef1-38e9-4386-9eec-e10e8df11a10" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:25:42.011373 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:42.011316    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:25:42.011732 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:42.011700    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:25:42.012065 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:42.012008    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:25:42.265752 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:42.265661    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-76b894-openshift-default-6669d4d94f-vrq8z"]
Nov 03 20:25:42.279046 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:42.279019    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-e80203/distinct-2-w4-e80203-kserve-756f9b9bc6-69gxj"
Nov 03 20:25:42.964888 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:42.964858    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz"
Nov 03 20:25:43.264722 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:43.264666    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-76b894-openshift-default-6669d4d94f-vrq8z" event={"ID":"3db8ac79-76fc-4d46-ac4f-6536fe192b8f","Type":"ContainerStarted","Data":"f117588d69672e70ba1262cea73f2561329718707205e5e418ca24375cb1f8cd"}
Nov 03 20:25:43.594282 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:43.592874    5570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-e80203/premium-w4-e80203-kserve-5f9f445456-bms9d" podUID="6ad76efc-20c2-4951-b22a-c5a39cd39ed4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:25:43.844613 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:43.844370    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/simulator-w0-e40816-kserve-688f79796f-2tvw5" podUID="c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:25:43.920799 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:43.918930    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/distinct-2-w4-e80203-kserve-756f9b9bc6-69gxj" podUID="ccf5dc2f-f116-4917-bb1a-b136d8ed17f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:25:44.287905 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:44.287861    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-76b894-openshift-default-6669d4d94f-vrq8z" event={"ID":"3db8ac79-76fc-4d46-ac4f-6536fe192b8f","Type":"ContainerStarted","Data":"e71f0ebc0297767f2edd7a26e6152c5a29917591acf45764b21d98a005b7a272"}
Nov 03 20:25:44.366606 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:44.366544    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-76b894-openshift-default-6669d4d94f-vrq8z" podStartSLOduration=8.366527646 podStartE2EDuration="8.366527646s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:25:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:25:44.34935095 +0000 UTC m=+1278.502012841" watchObservedRunningTime="2026-11-03 20:25:44.366527646 +0000 UTC m=+1278.519189534"
Nov 03 20:25:44.670468 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:44.667053    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-76b894-openshift-default-6669d4d94f-vrq8z"
Nov 03 20:25:44.703189 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:44.702078    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-76b894-openshift-default-6669d4d94f-vrq8z"
Nov 03 20:25:45.297750 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:45.297716    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-76b894-openshift-default-6669d4d94f-vrq8z"
Nov 03 20:25:45.298978 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:45.298959    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-76b894-openshift-default-6669d4d94f-vrq8z"
Nov 03 20:25:45.760466 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:45.760413    5570 patch_prober.go:29] interesting pod/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-jshb8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:25:45.761210 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:45.761157    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-jshb8" podUID="2137488e-8e6c-46f7-a0ed-b4721d4f2645" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:25:48.130420 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:48.130345    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-dc7b94-777ff98d5f-4l8x2"]
Nov 03 20:25:48.188805 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:48.162814    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-dc7b94-777ff98d5f-4l8x2"]
Nov 03 20:25:48.219437 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:48.163033    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-dc7b94-777ff98d5f-4l8x2"
Nov 03 20:25:48.350586 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:48.329135    5570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-dc7b94\""
Nov 03 20:25:48.350586 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:48.346356    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-dc7b94-dockercfg-lgtq5\""
Nov 03 20:25:48.350586 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:48.347957    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6t7b\" (UniqueName: \"kubernetes.io/projected/edc8b223-be7f-441c-aaa6-d22974f93443-kube-api-access-x6t7b\") pod \"payload-pre-processing-e2e-api-a-dc7b94-777ff98d5f-4l8x2\" (UID: \"edc8b223-be7f-441c-aaa6-d22974f93443\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-dc7b94-777ff98d5f-4l8x2"
Nov 03 20:25:48.350586 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:48.350078    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/edc8b223-be7f-441c-aaa6-d22974f93443-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-dc7b94-777ff98d5f-4l8x2\" (UID: \"edc8b223-be7f-441c-aaa6-d22974f93443\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-dc7b94-777ff98d5f-4l8x2"
Nov 03 20:25:48.472647 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:48.469142    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/edc8b223-be7f-441c-aaa6-d22974f93443-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-dc7b94-777ff98d5f-4l8x2\" (UID: \"edc8b223-be7f-441c-aaa6-d22974f93443\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-dc7b94-777ff98d5f-4l8x2"
Nov 03 20:25:48.472647 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:48.469594    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6t7b\" (UniqueName: \"kubernetes.io/projected/edc8b223-be7f-441c-aaa6-d22974f93443-kube-api-access-x6t7b\") pod \"payload-pre-processing-e2e-api-a-dc7b94-777ff98d5f-4l8x2\" (UID: \"edc8b223-be7f-441c-aaa6-d22974f93443\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-dc7b94-777ff98d5f-4l8x2"
Nov 03 20:25:48.492483 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:48.492444    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/edc8b223-be7f-441c-aaa6-d22974f93443-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-dc7b94-777ff98d5f-4l8x2\" (UID: \"edc8b223-be7f-441c-aaa6-d22974f93443\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-dc7b94-777ff98d5f-4l8x2"
Nov 03 20:25:48.728682 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:48.726860    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6t7b\" (UniqueName: \"kubernetes.io/projected/edc8b223-be7f-441c-aaa6-d22974f93443-kube-api-access-x6t7b\") pod \"payload-pre-processing-e2e-api-a-dc7b94-777ff98d5f-4l8x2\" (UID: \"edc8b223-be7f-441c-aaa6-d22974f93443\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-dc7b94-777ff98d5f-4l8x2"
Nov 03 20:25:48.744825 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:48.744402    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-dc7b94-777ff98d5f-4l8x2"
Nov 03 20:25:51.748049 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:51.747968    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz" podUID="3a715ef1-38e9-4386-9eec-e10e8df11a10" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:25:53.034765 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:53.034725    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3da0091e-b6g97"]
Nov 03 20:25:53.142475 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:53.048070    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3da0091e-b6g97"]
Nov 03 20:25:53.748533 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:53.748482    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-9e1bcb-c7dccc559-tbldl" podUID="cd3d374d-4721-4ccc-8283-dccefe85be90" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.46:8443/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:25:53.929720 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:53.901907    5570 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-vdrfs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:25:53.929720 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:53.901968    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-vdrfs" podUID="1b70a87b-76a6-40e6-9ced-037c3ceee5cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:25:54.071949 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:53.973372    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72de3f5b-77f3-4631-ad8d-6065ad327287" path="/var/lib/kubelet/pods/72de3f5b-77f3-4631-ad8d-6065ad327287/volumes"
Nov 03 20:25:54.117906 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:54.078571    5570 patch_prober.go:29] interesting pod/thanos-querier-5bd4bfb68d-9449x container/kube-rbac-proxy-web namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.134.0.12:9091/-/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:25:54.117906 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:54.078647    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/thanos-querier-5bd4bfb68d-9449x" podUID="59f6a4d6-afe0-4c96-a54a-d8e6b5c66a38" containerName="kube-rbac-proxy-web" probeResult="failure" output="Get \"https://10.134.0.12:9091/-/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:25:54.946630 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:54.946582    5570 patch_prober.go:29] interesting pod/metrics-server-654b7fcdc8-l6l9f container/metrics-server namespace/openshift-monitoring: Liveness probe status=failure output="Get \"https://10.134.0.13:10250/livez\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:25:55.160415 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:54.946680    5570 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-monitoring/metrics-server-654b7fcdc8-l6l9f" podUID="d5d14caa-6661-4146-b888-59c3518216d4" containerName="metrics-server" probeResult="failure" output="Get \"https://10.134.0.13:10250/livez\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:25:55.369990 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:55.369891    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6fd567655f-mghpw" podUID="f9182571-ed14-46b1-bc81-3f901876b06a" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.19:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:25:55.489947 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:55.489893    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-fbjbc" podUID="6308d31e-2cc7-4e66-9f9f-cf2c878b0043" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.20:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:25:55.936930 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:55.936892    5570 patch_prober.go:29] interesting pod/e2e-mt-eb87f429-openshift-default-64649dfdb5-nsjcd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:25:56.000020 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:55.936958    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-nsjcd" podUID="7b57a22a-cf28-4099-8883-c04c61807dfa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:25:56.958929 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:56.958886    5570 patch_prober.go:29] interesting pod/aws-ebs-csi-driver-node-9f5s5 container/csi-driver namespace/openshift-cluster-csi-drivers: Liveness probe status=failure output="Get \"http://127.0.0.1:10300/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:25:57.005375 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:56.958995    5570 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9f5s5" podUID="65008653-586d-4a76-a53f-7efe1f6d839b" containerName="csi-driver" probeResult="failure" output="Get \"http://127.0.0.1:10300/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:25:57.469323 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:25:57.469286    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87s5mf" podUID="b7306cb9-eeae-447b-b834-870c8d49a5aa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": context deadline exceeded"
Nov 03 20:26:00.705767 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:00.705720    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/distinct-w3-482722-kserve-d876967c-kd488" podUID="9a97f9cc-2261-471d-9053-51ee965850bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded"
Nov 03 20:26:01.125135 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:00.706023    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-6nh8m" podUID="b31639d5-0583-4706-b580-f5752fa502a4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": context deadline exceeded"
Nov 03 20:26:01.125135 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:00.706342    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-cpfzl" podUID="30b21f09-fd9e-4836-9fc3-c19f818c9022" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": context deadline exceeded"
Nov 03 20:26:02.892968 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:02.892919    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/simulator-w0-e40816-kserve-688f79796f-2tvw5" podUID="c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:26:02.893409 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:02.893253    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz" podUID="3a715ef1-38e9-4386-9eec-e10e8df11a10" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:26:03.358805 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:02.937502    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/premium-w3-482722-kserve-689dc7bd5d-x8rjp" podUID="369045c3-4d6a-49a7-9bc6-f2939883bee8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:26:03.358805 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:03.062009    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/premium-w4-e80203-kserve-5f9f445456-bms9d" podUID="6ad76efc-20c2-4951-b22a-c5a39cd39ed4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:26:03.358805 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:03.062328    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87s5mf" podUID="b7306cb9-eeae-447b-b834-870c8d49a5aa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:26:06.320970 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:03.646883    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/distinct-w3-482722-kserve-d876967c-kd488" podUID="9a97f9cc-2261-471d-9053-51ee965850bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:26:06.320970 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:03.729010    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh" podUID="2d3647f2-89e9-4908-a8ee-d84e527abeb0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:26:06.320970 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:04.240614    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-cpfzl" podUID="30b21f09-fd9e-4836-9fc3-c19f818c9022" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:26:06.320970 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:04.243465    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/simulator-w0-e40816-kserve-688f79796f-2tvw5" podUID="c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:26:06.320970 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:04.243527    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz" podUID="3a715ef1-38e9-4386-9eec-e10e8df11a10" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:26:06.320970 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:04.422095    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-6nh8m" podUID="b31639d5-0583-4706-b580-f5752fa502a4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:26:06.320970 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:04.436960    5570 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-5mwhf container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.25:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:26:06.320970 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:04.437010    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5mwhf" podUID="be4487de-c7d5-4a34-9064-a38ecfee6810" containerName="discovery" probeResult="failure" output="Get \"http://10.134.0.25:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:26:06.320970 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:04.437458    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-eb87f429-6b49f7957f-k2q4j" podUID="4711b06e-dba0-4d7a-a217-defa6ab8f008" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.112:50052\" within 1s: context deadline exceeded"
Nov 03 20:26:06.320970 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:26:04.484969    5570 logging.go:55] [core] [Channel #521 SubChannel #522]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.112:50052", ServerName: "10.134.0.112:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.112:50052: operation was canceled"
Nov 03 20:26:06.320970 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:04.551645    5570 patch_prober.go:29] interesting pod/dns-default-7zl2f container/dns namespace/openshift-dns: Readiness probe status=failure output="Get \"http://10.134.0.6:8181/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:26:06.320970 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:04.551691    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-dns/dns-default-7zl2f" podUID="2fd3c650-2f4c-404e-ab06-c50612af72ce" containerName="dns" probeResult="failure" output="Get \"http://10.134.0.6:8181/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:26:06.320970 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:04.561342    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-e3ba87-7c7fd945bd-dzmfm" podUID="5046089f-f2fa-4ed2-b424-7661fb3ea8c2" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.109:50052\" within 1s: context deadline exceeded"
Nov 03 20:26:06.320970 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:26:04.565860    5570 logging.go:55] [core] [Channel #520 SubChannel #531]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.109:50052", ServerName: "10.134.0.109:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.109:50052: operation was canceled"
Nov 03 20:26:06.320970 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:04.565902    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e40816-548465ddf5-mzdj7" podUID="94a03b52-b293-4475-8b23-fbbbae06fb26" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.45:50052\" within 1s: context deadline exceeded"
Nov 03 20:26:06.320970 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:05.175125    5570 patch_prober.go:29] interesting pod/metrics-server-654b7fcdc8-l6l9f container/metrics-server namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.134.0.13:10250/livez\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:26:06.320970 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:05.175195    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/metrics-server-654b7fcdc8-l6l9f" podUID="d5d14caa-6661-4146-b888-59c3518216d4" containerName="metrics-server" probeResult="failure" output="Get \"https://10.134.0.13:10250/livez\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:26:06.320970 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:26:05.175429    5570 logging.go:55] [core] [Channel #523 SubChannel #524]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.45:50052", ServerName: "10.134.0.45:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.134.0.2:56566->10.134.0.45:50052: use of closed network connection"
Nov 03 20:26:06.320970 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:05.211404    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/ai-gateway-controller-bf6bcc597-rwjc5" podUID="77fe3c84-a597-4a9d-a49d-392c266db493" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.36:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:26:06.320970 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:05.286904    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-6fc748-6b8d84dfb-4tvlr" podUID="75db83cf-ba54-414a-9041-094aa630a60a" containerName="payload-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s"
Nov 03 20:26:06.320970 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:05.373892    5570 prober.go:122] "Probe failed" probeType="Liveness" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-fbjbc" podUID="6308d31e-2cc7-4e66-9f9f-cf2c878b0043" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.20:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:26:06.320970 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:05.409338    5570 patch_prober.go:29] interesting pod/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-jshb8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:26:06.320970 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:05.409404    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-jshb8" podUID="2137488e-8e6c-46f7-a0ed-b4721d4f2645" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:26:06.320970 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:05.424279    5570 prober.go:122] "Probe failed" probeType="Liveness" pod="opendatahub/ai-gateway-controller-bf6bcc597-rwjc5" podUID="77fe3c84-a597-4a9d-a49d-392c266db493" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.36:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:26:06.320970 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:05.531301    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-fbjbc" podUID="6308d31e-2cc7-4e66-9f9f-cf2c878b0043" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.20:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:26:06.320970 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:05.556872    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-de82c8-78dc548fc4-24g87" podUID="ec1dc0b9-d778-48b7-986a-0d460bb6a52f" containerName="payload-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s"
Nov 03 20:26:06.320970 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:26:05.630427    5570 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="9.673s"
Nov 03 20:26:06.320970 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:05.630462    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-rbvqw"]
Nov 03 20:26:06.320970 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:06.255247    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87s5mf"
Nov 03 20:26:06.320970 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:06.255317    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-482722/distinct-w3-482722-kserve-d876967c-kd488"
Nov 03 20:26:06.320970 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:06.260961    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-rbvqw"
Nov 03 20:26:08.489369 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:06.364030    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-cpfzl"
Nov 03 20:26:08.489369 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:06.364405    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e40816/simulator-w0-e40816-kserve-688f79796f-2tvw5"
Nov 03 20:26:08.489369 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:06.364449    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz"
Nov 03 20:26:08.489369 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:06.364467    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2067241-ntph9"]
Nov 03 20:26:08.489369 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:06.364493    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2067241-ntph9"]
Nov 03 20:26:08.489369 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:06.581654    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae5a764e-25f7-492e-b4aa-8f2696e22f37-istio-podinfo\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-rbvqw\" (UID: \"ae5a764e-25f7-492e-b4aa-8f2696e22f37\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-rbvqw"
Nov 03 20:26:08.489369 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:06.581801    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae5a764e-25f7-492e-b4aa-8f2696e22f37-istio-envoy\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-rbvqw\" (UID: \"ae5a764e-25f7-492e-b4aa-8f2696e22f37\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-rbvqw"
Nov 03 20:26:08.489369 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:06.581863    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52h2d\" (UniqueName: \"kubernetes.io/projected/ae5a764e-25f7-492e-b4aa-8f2696e22f37-kube-api-access-52h2d\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-rbvqw\" (UID: \"ae5a764e-25f7-492e-b4aa-8f2696e22f37\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-rbvqw"
Nov 03 20:26:08.489369 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:06.582043    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae5a764e-25f7-492e-b4aa-8f2696e22f37-workload-socket\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-rbvqw\" (UID: \"ae5a764e-25f7-492e-b4aa-8f2696e22f37\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-rbvqw"
Nov 03 20:26:08.489369 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:06.582138    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae5a764e-25f7-492e-b4aa-8f2696e22f37-istiod-ca-cert\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-rbvqw\" (UID: \"ae5a764e-25f7-492e-b4aa-8f2696e22f37\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-rbvqw"
Nov 03 20:26:08.489369 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:06.582227    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae5a764e-25f7-492e-b4aa-8f2696e22f37-istio-token\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-rbvqw\" (UID: \"ae5a764e-25f7-492e-b4aa-8f2696e22f37\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-rbvqw"
Nov 03 20:26:08.489369 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:06.582270    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae5a764e-25f7-492e-b4aa-8f2696e22f37-workload-certs\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-rbvqw\" (UID: \"ae5a764e-25f7-492e-b4aa-8f2696e22f37\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-rbvqw"
Nov 03 20:26:08.489369 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:06.582301    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae5a764e-25f7-492e-b4aa-8f2696e22f37-istio-data\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-rbvqw\" (UID: \"ae5a764e-25f7-492e-b4aa-8f2696e22f37\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-rbvqw"
Nov 03 20:26:08.489369 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:06.582568    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae5a764e-25f7-492e-b4aa-8f2696e22f37-credential-socket\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-rbvqw\" (UID: \"ae5a764e-25f7-492e-b4aa-8f2696e22f37\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-rbvqw"
Nov 03 20:26:08.489369 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:06.732434    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-6nh8m"
Nov 03 20:26:08.489369 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:06.732502    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87s5mf"
Nov 03 20:26:08.489369 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:06.732540    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-cpfzl"
Nov 03 20:26:08.489369 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:06.732567    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e40816/simulator-w0-e40816-kserve-688f79796f-2tvw5"
Nov 03 20:26:08.489369 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:06.732585    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-482722/distinct-w3-482722-kserve-d876967c-kd488"
Nov 03 20:26:08.489369 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:06.732633    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-6nh8m"
Nov 03 20:26:08.489369 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:06.732657    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz"
Nov 03 20:26:08.489369 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:06.821952    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-9e1bcb-c7dccc559-tbldl" podUID="cd3d374d-4721-4ccc-8283-dccefe85be90" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.46:8443/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:26:08.489369 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:06.853418    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae5a764e-25f7-492e-b4aa-8f2696e22f37-istio-podinfo\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-rbvqw\" (UID: \"ae5a764e-25f7-492e-b4aa-8f2696e22f37\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-rbvqw"
Nov 03 20:26:08.489369 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:06.853549    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae5a764e-25f7-492e-b4aa-8f2696e22f37-istio-envoy\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-rbvqw\" (UID: \"ae5a764e-25f7-492e-b4aa-8f2696e22f37\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-rbvqw"
Nov 03 20:26:08.489369 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:06.853600    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52h2d\" (UniqueName: \"kubernetes.io/projected/ae5a764e-25f7-492e-b4aa-8f2696e22f37-kube-api-access-52h2d\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-rbvqw\" (UID: \"ae5a764e-25f7-492e-b4aa-8f2696e22f37\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-rbvqw"
Nov 03 20:26:08.489369 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:06.853736    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae5a764e-25f7-492e-b4aa-8f2696e22f37-workload-socket\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-rbvqw\" (UID: \"ae5a764e-25f7-492e-b4aa-8f2696e22f37\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-rbvqw"
Nov 03 20:26:08.489369 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:06.853836    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae5a764e-25f7-492e-b4aa-8f2696e22f37-istiod-ca-cert\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-rbvqw\" (UID: \"ae5a764e-25f7-492e-b4aa-8f2696e22f37\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-rbvqw"
Nov 03 20:26:08.489369 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:06.853911    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae5a764e-25f7-492e-b4aa-8f2696e22f37-istio-token\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-rbvqw\" (UID: \"ae5a764e-25f7-492e-b4aa-8f2696e22f37\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-rbvqw"
Nov 03 20:26:08.489369 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:06.853950    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae5a764e-25f7-492e-b4aa-8f2696e22f37-workload-certs\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-rbvqw\" (UID: \"ae5a764e-25f7-492e-b4aa-8f2696e22f37\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-rbvqw"
Nov 03 20:26:08.489369 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:06.853978    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae5a764e-25f7-492e-b4aa-8f2696e22f37-istio-data\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-rbvqw\" (UID: \"ae5a764e-25f7-492e-b4aa-8f2696e22f37\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-rbvqw"
Nov 03 20:26:08.489369 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:06.854125    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae5a764e-25f7-492e-b4aa-8f2696e22f37-credential-socket\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-rbvqw\" (UID: \"ae5a764e-25f7-492e-b4aa-8f2696e22f37\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-rbvqw"
Nov 03 20:26:08.489369 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:06.919570    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-rbvqw"]
Nov 03 20:26:08.489369 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:07.037265    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9668dd6c-qhvqr"]
Nov 03 20:26:08.489369 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:07.042253    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9668dd6c-qhvqr"]
Nov 03 20:26:08.489369 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:07.369886    5570 patch_prober.go:29] interesting pod/aws-ebs-csi-driver-node-9f5s5 container/csi-node-driver-registrar namespace/openshift-cluster-csi-drivers: Liveness probe status=failure output="Get \"http://127.0.0.1:10309/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:26:08.489369 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:07.369940    5570 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9f5s5" podUID="65008653-586d-4a76-a53f-7efe1f6d839b" containerName="csi-node-driver-registrar" probeResult="failure" output="Get \"http://127.0.0.1:10309/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:26:08.489369 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:07.550072    5570 patch_prober.go:29] interesting pod/aws-ebs-csi-driver-node-9f5s5 container/csi-driver namespace/openshift-cluster-csi-drivers: Liveness probe status=failure output="Get \"http://127.0.0.1:10300/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:26:08.489369 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:07.550138    5570 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9f5s5" podUID="65008653-586d-4a76-a53f-7efe1f6d839b" containerName="csi-driver" probeResult="failure" output="Get \"http://127.0.0.1:10300/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:26:08.489369 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:07.571870    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae5a764e-25f7-492e-b4aa-8f2696e22f37-workload-certs\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-rbvqw\" (UID: \"ae5a764e-25f7-492e-b4aa-8f2696e22f37\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-rbvqw"
Nov 03 20:26:08.489369 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:07.786105    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae5a764e-25f7-492e-b4aa-8f2696e22f37-credential-socket\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-rbvqw\" (UID: \"ae5a764e-25f7-492e-b4aa-8f2696e22f37\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-rbvqw"
Nov 03 20:26:08.489369 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:07.816835    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae5a764e-25f7-492e-b4aa-8f2696e22f37-istiod-ca-cert\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-rbvqw\" (UID: \"ae5a764e-25f7-492e-b4aa-8f2696e22f37\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-rbvqw"
Nov 03 20:26:08.489369 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:07.826833    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae5a764e-25f7-492e-b4aa-8f2696e22f37-workload-socket\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-rbvqw\" (UID: \"ae5a764e-25f7-492e-b4aa-8f2696e22f37\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-rbvqw"
Nov 03 20:26:08.489369 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:07.851036    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae5a764e-25f7-492e-b4aa-8f2696e22f37-istio-data\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-rbvqw\" (UID: \"ae5a764e-25f7-492e-b4aa-8f2696e22f37\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-rbvqw"
Nov 03 20:26:08.489369 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:07.908719    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-gd5bg"]
Nov 03 20:26:08.489369 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:08.062087    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae5a764e-25f7-492e-b4aa-8f2696e22f37-istio-podinfo\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-rbvqw\" (UID: \"ae5a764e-25f7-492e-b4aa-8f2696e22f37\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-rbvqw"
Nov 03 20:26:08.489369 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:08.073091    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae5a764e-25f7-492e-b4aa-8f2696e22f37-istio-token\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-rbvqw\" (UID: \"ae5a764e-25f7-492e-b4aa-8f2696e22f37\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-rbvqw"
Nov 03 20:26:08.489369 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:08.137069    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-gd5bg"
Nov 03 20:26:08.489369 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:08.185558    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae5a764e-25f7-492e-b4aa-8f2696e22f37-istio-envoy\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-rbvqw\" (UID: \"ae5a764e-25f7-492e-b4aa-8f2696e22f37\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-rbvqw"
Nov 03 20:26:08.489369 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:08.268558    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-52h2d\" (UniqueName: \"kubernetes.io/projected/ae5a764e-25f7-492e-b4aa-8f2696e22f37-kube-api-access-52h2d\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-rbvqw\" (UID: \"ae5a764e-25f7-492e-b4aa-8f2696e22f37\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-rbvqw"
Nov 03 20:26:08.489369 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:08.291370    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0037c462-11bc-4f13-9d5a-b4caaec482f5-istio-envoy\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-gd5bg\" (UID: \"0037c462-11bc-4f13-9d5a-b4caaec482f5\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-gd5bg"
Nov 03 20:26:08.489369 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:08.291425    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0037c462-11bc-4f13-9d5a-b4caaec482f5-istio-podinfo\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-gd5bg\" (UID: \"0037c462-11bc-4f13-9d5a-b4caaec482f5\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-gd5bg"
Nov 03 20:26:08.489369 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:08.291506    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5kwp\" (UniqueName: \"kubernetes.io/projected/0037c462-11bc-4f13-9d5a-b4caaec482f5-kube-api-access-q5kwp\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-gd5bg\" (UID: \"0037c462-11bc-4f13-9d5a-b4caaec482f5\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-gd5bg"
Nov 03 20:26:08.489369 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:08.291602    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0037c462-11bc-4f13-9d5a-b4caaec482f5-istio-data\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-gd5bg\" (UID: \"0037c462-11bc-4f13-9d5a-b4caaec482f5\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-gd5bg"
Nov 03 20:26:08.489369 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:08.291643    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0037c462-11bc-4f13-9d5a-b4caaec482f5-credential-socket\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-gd5bg\" (UID: \"0037c462-11bc-4f13-9d5a-b4caaec482f5\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-gd5bg"
Nov 03 20:26:08.489369 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:08.291754    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0037c462-11bc-4f13-9d5a-b4caaec482f5-istiod-ca-cert\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-gd5bg\" (UID: \"0037c462-11bc-4f13-9d5a-b4caaec482f5\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-gd5bg"
Nov 03 20:26:08.489369 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:08.291821    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0037c462-11bc-4f13-9d5a-b4caaec482f5-workload-socket\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-gd5bg\" (UID: \"0037c462-11bc-4f13-9d5a-b4caaec482f5\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-gd5bg"
Nov 03 20:26:08.489369 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:08.291850    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0037c462-11bc-4f13-9d5a-b4caaec482f5-istio-token\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-gd5bg\" (UID: \"0037c462-11bc-4f13-9d5a-b4caaec482f5\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-gd5bg"
Nov 03 20:26:08.489369 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:08.291984    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0037c462-11bc-4f13-9d5a-b4caaec482f5-workload-certs\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-gd5bg\" (UID: \"0037c462-11bc-4f13-9d5a-b4caaec482f5\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-gd5bg"
Nov 03 20:26:08.489369 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:08.324253    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aae4bf32-bc99-4177-aecb-4c7169a348c8" path="/var/lib/kubelet/pods/aae4bf32-bc99-4177-aecb-4c7169a348c8/volumes"
Nov 03 20:26:08.489369 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:08.377053    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-rbvqw"
Nov 03 20:26:08.489369 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:08.393694    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0037c462-11bc-4f13-9d5a-b4caaec482f5-istio-envoy\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-gd5bg\" (UID: \"0037c462-11bc-4f13-9d5a-b4caaec482f5\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-gd5bg"
Nov 03 20:26:08.489369 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:08.393731    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0037c462-11bc-4f13-9d5a-b4caaec482f5-istio-podinfo\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-gd5bg\" (UID: \"0037c462-11bc-4f13-9d5a-b4caaec482f5\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-gd5bg"
Nov 03 20:26:08.489369 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:08.393758    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5kwp\" (UniqueName: \"kubernetes.io/projected/0037c462-11bc-4f13-9d5a-b4caaec482f5-kube-api-access-q5kwp\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-gd5bg\" (UID: \"0037c462-11bc-4f13-9d5a-b4caaec482f5\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-gd5bg"
Nov 03 20:26:08.489369 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:08.393823    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0037c462-11bc-4f13-9d5a-b4caaec482f5-istio-data\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-gd5bg\" (UID: \"0037c462-11bc-4f13-9d5a-b4caaec482f5\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-gd5bg"
Nov 03 20:26:08.489369 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:08.393877    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0037c462-11bc-4f13-9d5a-b4caaec482f5-credential-socket\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-gd5bg\" (UID: \"0037c462-11bc-4f13-9d5a-b4caaec482f5\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-gd5bg"
Nov 03 20:26:08.489369 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:08.393938    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0037c462-11bc-4f13-9d5a-b4caaec482f5-istiod-ca-cert\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-gd5bg\" (UID: \"0037c462-11bc-4f13-9d5a-b4caaec482f5\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-gd5bg"
Nov 03 20:26:08.489369 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:08.393973    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0037c462-11bc-4f13-9d5a-b4caaec482f5-workload-socket\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-gd5bg\" (UID: \"0037c462-11bc-4f13-9d5a-b4caaec482f5\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-gd5bg"
Nov 03 20:26:08.489369 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:08.394001    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0037c462-11bc-4f13-9d5a-b4caaec482f5-istio-token\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-gd5bg\" (UID: \"0037c462-11bc-4f13-9d5a-b4caaec482f5\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-gd5bg"
Nov 03 20:26:08.489369 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:08.394140    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0037c462-11bc-4f13-9d5a-b4caaec482f5-workload-certs\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-gd5bg\" (UID: \"0037c462-11bc-4f13-9d5a-b4caaec482f5\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-gd5bg"
Nov 03 20:26:09.499098 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:08.757230    5570 trace.go:236] Trace[467114633]: "Calculate volume metrics of manifests for pod opendatahub-operator-system/opendatahub-operator-controller-manager-6fd567655f-mghpw" (03-Nov-2026 20:26:06.853) (total time: 1903ms):
Nov 03 20:26:09.499098 ip-10-0-132-4 kubenswrapper[5570]: Trace[467114633]: [1.903661531s] [1.903661531s] END
Nov 03 20:26:09.499098 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:08.840996    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5d89a76-31b0-4631-8756-48d9f887ee21" path="/var/lib/kubelet/pods/c5d89a76-31b0-4631-8756-48d9f887ee21/volumes"
Nov 03 20:26:09.499098 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:08.861022    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0037c462-11bc-4f13-9d5a-b4caaec482f5-istio-data\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-gd5bg\" (UID: \"0037c462-11bc-4f13-9d5a-b4caaec482f5\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-gd5bg"
Nov 03 20:26:09.499098 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:08.881358    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0037c462-11bc-4f13-9d5a-b4caaec482f5-credential-socket\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-gd5bg\" (UID: \"0037c462-11bc-4f13-9d5a-b4caaec482f5\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-gd5bg"
Nov 03 20:26:09.499098 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:08.881480    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zdsqp"]
Nov 03 20:26:09.499098 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:08.881610    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0037c462-11bc-4f13-9d5a-b4caaec482f5-istio-podinfo\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-gd5bg\" (UID: \"0037c462-11bc-4f13-9d5a-b4caaec482f5\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-gd5bg"
Nov 03 20:26:09.499098 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:09.009894    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0037c462-11bc-4f13-9d5a-b4caaec482f5-workload-certs\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-gd5bg\" (UID: \"0037c462-11bc-4f13-9d5a-b4caaec482f5\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-gd5bg"
Nov 03 20:26:09.499098 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:09.115855    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0037c462-11bc-4f13-9d5a-b4caaec482f5-workload-socket\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-gd5bg\" (UID: \"0037c462-11bc-4f13-9d5a-b4caaec482f5\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-gd5bg"
Nov 03 20:26:09.499098 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:09.115867    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0037c462-11bc-4f13-9d5a-b4caaec482f5-istiod-ca-cert\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-gd5bg\" (UID: \"0037c462-11bc-4f13-9d5a-b4caaec482f5\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-gd5bg"
Nov 03 20:26:09.499098 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:09.188200    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-gd5bg"]
Nov 03 20:26:09.499098 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:09.188348    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zdsqp"]
Nov 03 20:26:09.499098 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:09.188352    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zdsqp"
Nov 03 20:26:09.499098 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:09.188584    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5kwp\" (UniqueName: \"kubernetes.io/projected/0037c462-11bc-4f13-9d5a-b4caaec482f5-kube-api-access-q5kwp\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-gd5bg\" (UID: \"0037c462-11bc-4f13-9d5a-b4caaec482f5\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-gd5bg"
Nov 03 20:26:09.499098 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:09.263910    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0037c462-11bc-4f13-9d5a-b4caaec482f5-istio-envoy\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-gd5bg\" (UID: \"0037c462-11bc-4f13-9d5a-b4caaec482f5\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-gd5bg"
Nov 03 20:26:09.499098 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:09.264207    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0037c462-11bc-4f13-9d5a-b4caaec482f5-istio-token\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-gd5bg\" (UID: \"0037c462-11bc-4f13-9d5a-b4caaec482f5\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-gd5bg"
Nov 03 20:26:09.499098 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:09.314249    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6768\" (UniqueName: \"kubernetes.io/projected/f747f534-4e45-446f-948e-c66ce316db5d-kube-api-access-w6768\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-zdsqp\" (UID: \"f747f534-4e45-446f-948e-c66ce316db5d\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zdsqp"
Nov 03 20:26:09.499098 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:09.314296    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f747f534-4e45-446f-948e-c66ce316db5d-istio-token\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-zdsqp\" (UID: \"f747f534-4e45-446f-948e-c66ce316db5d\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zdsqp"
Nov 03 20:26:09.499098 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:09.314633    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f747f534-4e45-446f-948e-c66ce316db5d-istio-data\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-zdsqp\" (UID: \"f747f534-4e45-446f-948e-c66ce316db5d\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zdsqp"
Nov 03 20:26:09.499098 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:09.314720    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f747f534-4e45-446f-948e-c66ce316db5d-workload-certs\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-zdsqp\" (UID: \"f747f534-4e45-446f-948e-c66ce316db5d\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zdsqp"
Nov 03 20:26:09.499098 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:09.314897    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f747f534-4e45-446f-948e-c66ce316db5d-istio-envoy\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-zdsqp\" (UID: \"f747f534-4e45-446f-948e-c66ce316db5d\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zdsqp"
Nov 03 20:26:09.499098 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:09.314924    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f747f534-4e45-446f-948e-c66ce316db5d-workload-socket\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-zdsqp\" (UID: \"f747f534-4e45-446f-948e-c66ce316db5d\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zdsqp"
Nov 03 20:26:09.499098 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:09.314944    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f747f534-4e45-446f-948e-c66ce316db5d-credential-socket\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-zdsqp\" (UID: \"f747f534-4e45-446f-948e-c66ce316db5d\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zdsqp"
Nov 03 20:26:09.499098 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:09.315030    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f747f534-4e45-446f-948e-c66ce316db5d-istiod-ca-cert\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-zdsqp\" (UID: \"f747f534-4e45-446f-948e-c66ce316db5d\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zdsqp"
Nov 03 20:26:09.499098 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:09.315078    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f747f534-4e45-446f-948e-c66ce316db5d-istio-podinfo\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-zdsqp\" (UID: \"f747f534-4e45-446f-948e-c66ce316db5d\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zdsqp"
Nov 03 20:26:09.499098 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:09.417523    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6768\" (UniqueName: \"kubernetes.io/projected/f747f534-4e45-446f-948e-c66ce316db5d-kube-api-access-w6768\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-zdsqp\" (UID: \"f747f534-4e45-446f-948e-c66ce316db5d\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zdsqp"
Nov 03 20:26:09.499098 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:09.417576    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f747f534-4e45-446f-948e-c66ce316db5d-istio-token\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-zdsqp\" (UID: \"f747f534-4e45-446f-948e-c66ce316db5d\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zdsqp"
Nov 03 20:26:09.499098 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:09.417742    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f747f534-4e45-446f-948e-c66ce316db5d-istio-data\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-zdsqp\" (UID: \"f747f534-4e45-446f-948e-c66ce316db5d\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zdsqp"
Nov 03 20:26:09.499098 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:09.417819    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f747f534-4e45-446f-948e-c66ce316db5d-workload-certs\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-zdsqp\" (UID: \"f747f534-4e45-446f-948e-c66ce316db5d\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zdsqp"
Nov 03 20:26:09.499098 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:09.417884    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f747f534-4e45-446f-948e-c66ce316db5d-istio-envoy\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-zdsqp\" (UID: \"f747f534-4e45-446f-948e-c66ce316db5d\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zdsqp"
Nov 03 20:26:09.499098 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:09.417904    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f747f534-4e45-446f-948e-c66ce316db5d-workload-socket\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-zdsqp\" (UID: \"f747f534-4e45-446f-948e-c66ce316db5d\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zdsqp"
Nov 03 20:26:09.499098 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:09.417923    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f747f534-4e45-446f-948e-c66ce316db5d-credential-socket\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-zdsqp\" (UID: \"f747f534-4e45-446f-948e-c66ce316db5d\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zdsqp"
Nov 03 20:26:09.499098 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:09.417956    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f747f534-4e45-446f-948e-c66ce316db5d-istiod-ca-cert\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-zdsqp\" (UID: \"f747f534-4e45-446f-948e-c66ce316db5d\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zdsqp"
Nov 03 20:26:09.499098 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:09.418007    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f747f534-4e45-446f-948e-c66ce316db5d-istio-podinfo\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-zdsqp\" (UID: \"f747f534-4e45-446f-948e-c66ce316db5d\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zdsqp"
Nov 03 20:26:09.499098 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:09.443044    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-gd5bg"
Nov 03 20:26:09.919922 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:26:09.517479    5570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8bwhz" containerName="extractor"
Nov 03 20:26:09.919922 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:09.527977    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f747f534-4e45-446f-948e-c66ce316db5d-credential-socket\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-zdsqp\" (UID: \"f747f534-4e45-446f-948e-c66ce316db5d\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zdsqp"
Nov 03 20:26:09.919922 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:09.536479    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f747f534-4e45-446f-948e-c66ce316db5d-workload-certs\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-zdsqp\" (UID: \"f747f534-4e45-446f-948e-c66ce316db5d\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zdsqp"
Nov 03 20:26:09.919922 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:09.573303    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f747f534-4e45-446f-948e-c66ce316db5d-istio-podinfo\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-zdsqp\" (UID: \"f747f534-4e45-446f-948e-c66ce316db5d\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zdsqp"
Nov 03 20:26:09.919922 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:09.586439    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f747f534-4e45-446f-948e-c66ce316db5d-istiod-ca-cert\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-zdsqp\" (UID: \"f747f534-4e45-446f-948e-c66ce316db5d\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zdsqp"
Nov 03 20:26:09.919922 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:09.621991    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f747f534-4e45-446f-948e-c66ce316db5d-istio-envoy\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-zdsqp\" (UID: \"f747f534-4e45-446f-948e-c66ce316db5d\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zdsqp"
Nov 03 20:26:09.919922 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:09.622667    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-dc7b94-777ff98d5f-4l8x2"]
Nov 03 20:26:09.919922 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:09.633824    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f747f534-4e45-446f-948e-c66ce316db5d-istio-data\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-zdsqp\" (UID: \"f747f534-4e45-446f-948e-c66ce316db5d\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zdsqp"
Nov 03 20:26:09.919922 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:09.636396    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6768\" (UniqueName: \"kubernetes.io/projected/f747f534-4e45-446f-948e-c66ce316db5d-kube-api-access-w6768\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-zdsqp\" (UID: \"f747f534-4e45-446f-948e-c66ce316db5d\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zdsqp"
Nov 03 20:26:09.919922 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:09.756221    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f747f534-4e45-446f-948e-c66ce316db5d-istio-token\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-zdsqp\" (UID: \"f747f534-4e45-446f-948e-c66ce316db5d\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zdsqp"
Nov 03 20:26:09.919922 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:09.756452    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f747f534-4e45-446f-948e-c66ce316db5d-workload-socket\") pod \"e2e-mt-eb87f429-openshift-default-64649dfdb5-zdsqp\" (UID: \"f747f534-4e45-446f-948e-c66ce316db5d\") " pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zdsqp"
Nov 03 20:26:10.030750 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:10.030711    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-126d4c8e-f5fm4"]
Nov 03 20:26:10.039509 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:10.039475    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zdsqp"
Nov 03 20:26:10.046419 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:10.042088    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-126d4c8e-f5fm4"]
Nov 03 20:26:10.375029 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:26:10.374982    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedc8b223_be7f_441c_aaa6_d22974f93443.slice/crio-8a608bbd6d689f30d739676e402a3b638b6a820d4c0cbaffabffa7db08fce9ae WatchSource:0}: Error finding container 8a608bbd6d689f30d739676e402a3b638b6a820d4c0cbaffabffa7db08fce9ae: Status 404 returned error can't find the container with id 8a608bbd6d689f30d739676e402a3b638b6a820d4c0cbaffabffa7db08fce9ae
Nov 03 20:26:11.014593 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:10.936928    5570 patch_prober.go:29] interesting pod/e2e-mt-eb87f429-openshift-default-64649dfdb5-nsjcd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:26:11.014593 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:10.937000    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-nsjcd" podUID="7b57a22a-cf28-4099-8883-c04c61807dfa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:26:11.014593 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:11.012402    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-dc7b94-777ff98d5f-4l8x2" event={"ID":"edc8b223-be7f-441c-aaa6-d22974f93443","Type":"ContainerStarted","Data":"8a608bbd6d689f30d739676e402a3b638b6a820d4c0cbaffabffa7db08fce9ae"}
Nov 03 20:26:12.066940 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:12.049538    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56932084-ddac-401e-87d3-8c75064e4498" path="/var/lib/kubelet/pods/56932084-ddac-401e-87d3-8c75064e4498/volumes"
Nov 03 20:26:12.066940 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:12.050190    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zdsqp"]
Nov 03 20:26:12.066940 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:12.050296    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-rbvqw"]
Nov 03 20:26:12.066940 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:12.050327    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-gd5bg"]
Nov 03 20:26:12.082950 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:12.082829    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-dc7b94-777ff98d5f-4l8x2" event={"ID":"edc8b223-be7f-441c-aaa6-d22974f93443","Type":"ContainerStarted","Data":"8ec3dd7ee1d9b56d9c6952e0000ca593092b590016ce9c7d7c857c04aeb26adf"}
Nov 03 20:26:12.085039 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:12.085002    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-dc7b94-777ff98d5f-4l8x2"
Nov 03 20:26:12.095487 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:26:12.095449    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf747f534_4e45_446f_948e_c66ce316db5d.slice/crio-039c1406177aa8b42956f35632c28b10a3b7b6ce94d0b16815450f8a41ea113d WatchSource:0}: Error finding container 039c1406177aa8b42956f35632c28b10a3b7b6ce94d0b16815450f8a41ea113d: Status 404 returned error can't find the container with id 039c1406177aa8b42956f35632c28b10a3b7b6ce94d0b16815450f8a41ea113d
Nov 03 20:26:12.095992 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:26:12.095957    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0037c462_11bc_4f13_9d5a_b4caaec482f5.slice/crio-e0bec6f8ef33f54f4ddf20b0ae68eb130ebf01b75b170c88730ed027f75c1349 WatchSource:0}: Error finding container e0bec6f8ef33f54f4ddf20b0ae68eb130ebf01b75b170c88730ed027f75c1349: Status 404 returned error can't find the container with id e0bec6f8ef33f54f4ddf20b0ae68eb130ebf01b75b170c88730ed027f75c1349
Nov 03 20:26:12.104792 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:12.104205    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:26:12.104792 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:12.104550    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:26:12.104792 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:12.104613    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:26:12.129706 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:12.114620    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:26:12.129706 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:12.114705    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:26:12.129706 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:12.114751    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:26:12.129706 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:12.126579    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:26:12.129706 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:12.126679    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:26:12.129706 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:12.126727    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:26:12.144185 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:12.141401    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-j7dz6"]
Nov 03 20:26:12.218505 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:12.188291    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-dc7b94-777ff98d5f-4l8x2" podStartSLOduration=25.188268002 podStartE2EDuration="25.188268002s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:25:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:26:12.133638034 +0000 UTC m=+1306.286299924" watchObservedRunningTime="2026-11-03 20:26:12.188268002 +0000 UTC m=+1306.340929890"
Nov 03 20:26:12.304089 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:12.304045    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-g262x"]
Nov 03 20:26:12.304359 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:12.304337    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-j7dz6"
Nov 03 20:26:12.404750 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:12.388243    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22ea3ab0-6acc-4558-9158-eaa289924f35-workload-socket\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-j7dz6\" (UID: \"22ea3ab0-6acc-4558-9158-eaa289924f35\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-j7dz6"
Nov 03 20:26:12.404750 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:12.388324    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22ea3ab0-6acc-4558-9158-eaa289924f35-istio-token\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-j7dz6\" (UID: \"22ea3ab0-6acc-4558-9158-eaa289924f35\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-j7dz6"
Nov 03 20:26:12.404750 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:12.388373    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22ea3ab0-6acc-4558-9158-eaa289924f35-istiod-ca-cert\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-j7dz6\" (UID: \"22ea3ab0-6acc-4558-9158-eaa289924f35\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-j7dz6"
Nov 03 20:26:12.404750 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:12.388483    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22ea3ab0-6acc-4558-9158-eaa289924f35-istio-data\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-j7dz6\" (UID: \"22ea3ab0-6acc-4558-9158-eaa289924f35\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-j7dz6"
Nov 03 20:26:12.404750 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:12.388552    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4rvl\" (UniqueName: \"kubernetes.io/projected/22ea3ab0-6acc-4558-9158-eaa289924f35-kube-api-access-m4rvl\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-j7dz6\" (UID: \"22ea3ab0-6acc-4558-9158-eaa289924f35\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-j7dz6"
Nov 03 20:26:12.404750 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:12.388579    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22ea3ab0-6acc-4558-9158-eaa289924f35-workload-certs\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-j7dz6\" (UID: \"22ea3ab0-6acc-4558-9158-eaa289924f35\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-j7dz6"
Nov 03 20:26:12.404750 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:12.389917    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22ea3ab0-6acc-4558-9158-eaa289924f35-istio-envoy\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-j7dz6\" (UID: \"22ea3ab0-6acc-4558-9158-eaa289924f35\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-j7dz6"
Nov 03 20:26:12.404750 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:12.389944    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22ea3ab0-6acc-4558-9158-eaa289924f35-credential-socket\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-j7dz6\" (UID: \"22ea3ab0-6acc-4558-9158-eaa289924f35\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-j7dz6"
Nov 03 20:26:12.404750 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:12.390067    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22ea3ab0-6acc-4558-9158-eaa289924f35-istio-podinfo\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-j7dz6\" (UID: \"22ea3ab0-6acc-4558-9158-eaa289924f35\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-j7dz6"
Nov 03 20:26:12.492056 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:12.492028    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22ea3ab0-6acc-4558-9158-eaa289924f35-istio-data\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-j7dz6\" (UID: \"22ea3ab0-6acc-4558-9158-eaa289924f35\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-j7dz6"
Nov 03 20:26:12.492145 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:12.492099    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4rvl\" (UniqueName: \"kubernetes.io/projected/22ea3ab0-6acc-4558-9158-eaa289924f35-kube-api-access-m4rvl\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-j7dz6\" (UID: \"22ea3ab0-6acc-4558-9158-eaa289924f35\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-j7dz6"
Nov 03 20:26:12.492145 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:12.492127    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22ea3ab0-6acc-4558-9158-eaa289924f35-workload-certs\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-j7dz6\" (UID: \"22ea3ab0-6acc-4558-9158-eaa289924f35\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-j7dz6"
Nov 03 20:26:12.492214 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:12.492185    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22ea3ab0-6acc-4558-9158-eaa289924f35-istio-envoy\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-j7dz6\" (UID: \"22ea3ab0-6acc-4558-9158-eaa289924f35\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-j7dz6"
Nov 03 20:26:12.492261 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:12.492214    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22ea3ab0-6acc-4558-9158-eaa289924f35-credential-socket\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-j7dz6\" (UID: \"22ea3ab0-6acc-4558-9158-eaa289924f35\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-j7dz6"
Nov 03 20:26:12.492489 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:12.492375    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22ea3ab0-6acc-4558-9158-eaa289924f35-istio-podinfo\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-j7dz6\" (UID: \"22ea3ab0-6acc-4558-9158-eaa289924f35\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-j7dz6"
Nov 03 20:26:12.492611 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:12.492592    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22ea3ab0-6acc-4558-9158-eaa289924f35-workload-socket\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-j7dz6\" (UID: \"22ea3ab0-6acc-4558-9158-eaa289924f35\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-j7dz6"
Nov 03 20:26:12.492673 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:12.492660    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22ea3ab0-6acc-4558-9158-eaa289924f35-istio-token\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-j7dz6\" (UID: \"22ea3ab0-6acc-4558-9158-eaa289924f35\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-j7dz6"
Nov 03 20:26:12.492715 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:12.492698    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22ea3ab0-6acc-4558-9158-eaa289924f35-istiod-ca-cert\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-j7dz6\" (UID: \"22ea3ab0-6acc-4558-9158-eaa289924f35\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-j7dz6"
Nov 03 20:26:12.509191 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:12.508945    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-j7dz6"]
Nov 03 20:26:12.509191 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:12.508995    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-g262x"]
Nov 03 20:26:12.509191 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:12.509101    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-g262x"
Nov 03 20:26:12.527132 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:12.527070    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22ea3ab0-6acc-4558-9158-eaa289924f35-istio-data\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-j7dz6\" (UID: \"22ea3ab0-6acc-4558-9158-eaa289924f35\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-j7dz6"
Nov 03 20:26:12.528094 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:12.528006    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22ea3ab0-6acc-4558-9158-eaa289924f35-workload-certs\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-j7dz6\" (UID: \"22ea3ab0-6acc-4558-9158-eaa289924f35\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-j7dz6"
Nov 03 20:26:12.528682 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:12.528656    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22ea3ab0-6acc-4558-9158-eaa289924f35-credential-socket\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-j7dz6\" (UID: \"22ea3ab0-6acc-4558-9158-eaa289924f35\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-j7dz6"
Nov 03 20:26:12.529895 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:12.529865    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22ea3ab0-6acc-4558-9158-eaa289924f35-istiod-ca-cert\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-j7dz6\" (UID: \"22ea3ab0-6acc-4558-9158-eaa289924f35\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-j7dz6"
Nov 03 20:26:12.530010 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:12.529986    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22ea3ab0-6acc-4558-9158-eaa289924f35-workload-socket\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-j7dz6\" (UID: \"22ea3ab0-6acc-4558-9158-eaa289924f35\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-j7dz6"
Nov 03 20:26:12.532754 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:12.532696    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22ea3ab0-6acc-4558-9158-eaa289924f35-istio-envoy\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-j7dz6\" (UID: \"22ea3ab0-6acc-4558-9158-eaa289924f35\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-j7dz6"
Nov 03 20:26:12.535116 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:12.535075    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22ea3ab0-6acc-4558-9158-eaa289924f35-istio-token\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-j7dz6\" (UID: \"22ea3ab0-6acc-4558-9158-eaa289924f35\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-j7dz6"
Nov 03 20:26:12.537901 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:12.537877    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22ea3ab0-6acc-4558-9158-eaa289924f35-istio-podinfo\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-j7dz6\" (UID: \"22ea3ab0-6acc-4558-9158-eaa289924f35\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-j7dz6"
Nov 03 20:26:12.544919 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:12.544866    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4rvl\" (UniqueName: \"kubernetes.io/projected/22ea3ab0-6acc-4558-9158-eaa289924f35-kube-api-access-m4rvl\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-j7dz6\" (UID: \"22ea3ab0-6acc-4558-9158-eaa289924f35\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-j7dz6"
Nov 03 20:26:12.594572 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:12.594532    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8aa4c48e-92c6-49c8-acd3-4bc441ac8ca7-istiod-ca-cert\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-g262x\" (UID: \"8aa4c48e-92c6-49c8-acd3-4bc441ac8ca7\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-g262x"
Nov 03 20:26:12.595482 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:12.595396    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8aa4c48e-92c6-49c8-acd3-4bc441ac8ca7-istio-data\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-g262x\" (UID: \"8aa4c48e-92c6-49c8-acd3-4bc441ac8ca7\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-g262x"
Nov 03 20:26:12.595626 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:12.595564    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmw5r\" (UniqueName: \"kubernetes.io/projected/8aa4c48e-92c6-49c8-acd3-4bc441ac8ca7-kube-api-access-xmw5r\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-g262x\" (UID: \"8aa4c48e-92c6-49c8-acd3-4bc441ac8ca7\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-g262x"
Nov 03 20:26:12.595705 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:12.595664    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8aa4c48e-92c6-49c8-acd3-4bc441ac8ca7-istio-podinfo\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-g262x\" (UID: \"8aa4c48e-92c6-49c8-acd3-4bc441ac8ca7\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-g262x"
Nov 03 20:26:12.595705 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:12.595699    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8aa4c48e-92c6-49c8-acd3-4bc441ac8ca7-workload-socket\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-g262x\" (UID: \"8aa4c48e-92c6-49c8-acd3-4bc441ac8ca7\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-g262x"
Nov 03 20:26:12.595862 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:12.595769    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8aa4c48e-92c6-49c8-acd3-4bc441ac8ca7-istio-token\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-g262x\" (UID: \"8aa4c48e-92c6-49c8-acd3-4bc441ac8ca7\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-g262x"
Nov 03 20:26:12.595927 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:12.595886    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8aa4c48e-92c6-49c8-acd3-4bc441ac8ca7-istio-envoy\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-g262x\" (UID: \"8aa4c48e-92c6-49c8-acd3-4bc441ac8ca7\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-g262x"
Nov 03 20:26:12.596034 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:12.596014    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8aa4c48e-92c6-49c8-acd3-4bc441ac8ca7-workload-certs\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-g262x\" (UID: \"8aa4c48e-92c6-49c8-acd3-4bc441ac8ca7\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-g262x"
Nov 03 20:26:12.596179 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:12.596157    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8aa4c48e-92c6-49c8-acd3-4bc441ac8ca7-credential-socket\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-g262x\" (UID: \"8aa4c48e-92c6-49c8-acd3-4bc441ac8ca7\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-g262x"
Nov 03 20:26:12.698410 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:12.698368    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8aa4c48e-92c6-49c8-acd3-4bc441ac8ca7-workload-certs\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-g262x\" (UID: \"8aa4c48e-92c6-49c8-acd3-4bc441ac8ca7\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-g262x"
Nov 03 20:26:12.698566 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:12.698522    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8aa4c48e-92c6-49c8-acd3-4bc441ac8ca7-credential-socket\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-g262x\" (UID: \"8aa4c48e-92c6-49c8-acd3-4bc441ac8ca7\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-g262x"
Nov 03 20:26:12.698733 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:12.698707    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8aa4c48e-92c6-49c8-acd3-4bc441ac8ca7-istiod-ca-cert\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-g262x\" (UID: \"8aa4c48e-92c6-49c8-acd3-4bc441ac8ca7\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-g262x"
Nov 03 20:26:12.698845 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:12.698821    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8aa4c48e-92c6-49c8-acd3-4bc441ac8ca7-istio-data\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-g262x\" (UID: \"8aa4c48e-92c6-49c8-acd3-4bc441ac8ca7\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-g262x"
Nov 03 20:26:12.699112 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:12.698937    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmw5r\" (UniqueName: \"kubernetes.io/projected/8aa4c48e-92c6-49c8-acd3-4bc441ac8ca7-kube-api-access-xmw5r\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-g262x\" (UID: \"8aa4c48e-92c6-49c8-acd3-4bc441ac8ca7\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-g262x"
Nov 03 20:26:12.699112 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:12.699005    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8aa4c48e-92c6-49c8-acd3-4bc441ac8ca7-istio-podinfo\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-g262x\" (UID: \"8aa4c48e-92c6-49c8-acd3-4bc441ac8ca7\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-g262x"
Nov 03 20:26:12.699112 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:12.699037    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8aa4c48e-92c6-49c8-acd3-4bc441ac8ca7-workload-socket\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-g262x\" (UID: \"8aa4c48e-92c6-49c8-acd3-4bc441ac8ca7\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-g262x"
Nov 03 20:26:12.700551 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:12.700216    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8aa4c48e-92c6-49c8-acd3-4bc441ac8ca7-workload-certs\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-g262x\" (UID: \"8aa4c48e-92c6-49c8-acd3-4bc441ac8ca7\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-g262x"
Nov 03 20:26:12.700551 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:12.700496    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8aa4c48e-92c6-49c8-acd3-4bc441ac8ca7-istio-data\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-g262x\" (UID: \"8aa4c48e-92c6-49c8-acd3-4bc441ac8ca7\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-g262x"
Nov 03 20:26:12.700551 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:12.700507    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8aa4c48e-92c6-49c8-acd3-4bc441ac8ca7-credential-socket\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-g262x\" (UID: \"8aa4c48e-92c6-49c8-acd3-4bc441ac8ca7\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-g262x"
Nov 03 20:26:12.701069 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:12.701040    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8aa4c48e-92c6-49c8-acd3-4bc441ac8ca7-istio-token\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-g262x\" (UID: \"8aa4c48e-92c6-49c8-acd3-4bc441ac8ca7\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-g262x"
Nov 03 20:26:12.701191 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:12.701131    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8aa4c48e-92c6-49c8-acd3-4bc441ac8ca7-istio-envoy\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-g262x\" (UID: \"8aa4c48e-92c6-49c8-acd3-4bc441ac8ca7\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-g262x"
Nov 03 20:26:12.705798 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:12.705317    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8aa4c48e-92c6-49c8-acd3-4bc441ac8ca7-workload-socket\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-g262x\" (UID: \"8aa4c48e-92c6-49c8-acd3-4bc441ac8ca7\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-g262x"
Nov 03 20:26:12.706545 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:12.706495    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8aa4c48e-92c6-49c8-acd3-4bc441ac8ca7-istio-podinfo\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-g262x\" (UID: \"8aa4c48e-92c6-49c8-acd3-4bc441ac8ca7\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-g262x"
Nov 03 20:26:12.706741 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:12.706704    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8aa4c48e-92c6-49c8-acd3-4bc441ac8ca7-istiod-ca-cert\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-g262x\" (UID: \"8aa4c48e-92c6-49c8-acd3-4bc441ac8ca7\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-g262x"
Nov 03 20:26:12.710112 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:12.710027    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8aa4c48e-92c6-49c8-acd3-4bc441ac8ca7-istio-envoy\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-g262x\" (UID: \"8aa4c48e-92c6-49c8-acd3-4bc441ac8ca7\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-g262x"
Nov 03 20:26:12.720091 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:12.720067    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmw5r\" (UniqueName: \"kubernetes.io/projected/8aa4c48e-92c6-49c8-acd3-4bc441ac8ca7-kube-api-access-xmw5r\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-g262x\" (UID: \"8aa4c48e-92c6-49c8-acd3-4bc441ac8ca7\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-g262x"
Nov 03 20:26:12.721477 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:12.721453    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8aa4c48e-92c6-49c8-acd3-4bc441ac8ca7-istio-token\") pod \"e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-g262x\" (UID: \"8aa4c48e-92c6-49c8-acd3-4bc441ac8ca7\") " pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-g262x"
Nov 03 20:26:12.835064 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:12.835027    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-j7dz6"
Nov 03 20:26:12.841099 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:12.841067    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-g262x"
Nov 03 20:26:13.196852 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:13.196386    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zdsqp" event={"ID":"f747f534-4e45-446f-948e-c66ce316db5d","Type":"ContainerStarted","Data":"1b29b47590450ba9dbd2e2bf29919ee55c3af4cce616436f3a421e68c2808b25"}
Nov 03 20:26:13.196852 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:13.196488    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zdsqp" event={"ID":"f747f534-4e45-446f-948e-c66ce316db5d","Type":"ContainerStarted","Data":"039c1406177aa8b42956f35632c28b10a3b7b6ce94d0b16815450f8a41ea113d"}
Nov 03 20:26:13.200633 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:13.199726    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-gd5bg" event={"ID":"0037c462-11bc-4f13-9d5a-b4caaec482f5","Type":"ContainerStarted","Data":"dfbabe837991c35498bcfdd8e6a2b1187ffc2d56f090498d2294563f36c5f235"}
Nov 03 20:26:13.200633 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:13.199803    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-gd5bg" event={"ID":"0037c462-11bc-4f13-9d5a-b4caaec482f5","Type":"ContainerStarted","Data":"e0bec6f8ef33f54f4ddf20b0ae68eb130ebf01b75b170c88730ed027f75c1349"}
Nov 03 20:26:13.208059 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:13.207676    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-rbvqw" event={"ID":"ae5a764e-25f7-492e-b4aa-8f2696e22f37","Type":"ContainerStarted","Data":"d1909a45904a0d779c9eee7a33e1d1b228fb1289b0a14179aaa2f840cae28632"}
Nov 03 20:26:13.208059 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:13.207731    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-rbvqw" event={"ID":"ae5a764e-25f7-492e-b4aa-8f2696e22f37","Type":"ContainerStarted","Data":"1da53265a6710cf9dd25041f57e3eb4b0f2da2c40cb98f13a8473405a95ec4e5"}
Nov 03 20:26:13.240520 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:13.240453    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-j7dz6"]
Nov 03 20:26:13.246423 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:13.246205    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zdsqp" podStartSLOduration=6.246187126 podStartE2EDuration="6.246187126s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:26:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:26:13.240975111 +0000 UTC m=+1307.393637011" watchObservedRunningTime="2026-11-03 20:26:13.246187126 +0000 UTC m=+1307.398849018"
Nov 03 20:26:13.322510 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:13.322091    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:26:13.322510 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:13.322188    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:26:13.322510 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:13.322221    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:26:13.325129 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:13.323502    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-gd5bg" podStartSLOduration=6.323482204 podStartE2EDuration="6.323482204s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:26:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:26:13.315343396 +0000 UTC m=+1307.468005285" watchObservedRunningTime="2026-11-03 20:26:13.323482204 +0000 UTC m=+1307.476144092"
Nov 03 20:26:13.393282 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:13.393199    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-rbvqw" podStartSLOduration=16.39314833 podStartE2EDuration="16.39314833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:25:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:26:13.358995914 +0000 UTC m=+1307.511657803" watchObservedRunningTime="2026-11-03 20:26:13.39314833 +0000 UTC m=+1307.545810218"
Nov 03 20:26:13.398556 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:13.398530    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-rbvqw"
Nov 03 20:26:13.400333 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:13.400305    5570 patch_prober.go:29] interesting pod/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-rbvqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body=
Nov 03 20:26:13.400907 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:13.400530    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-rbvqw" podUID="ae5a764e-25f7-492e-b4aa-8f2696e22f37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused"
Nov 03 20:26:13.447398 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:13.447329    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-gd5bg"
Nov 03 20:26:13.450286 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:13.450262    5570 patch_prober.go:29] interesting pod/e2e-mt-eb87f429-openshift-default-64649dfdb5-gd5bg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body=
Nov 03 20:26:13.450783 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:13.450497    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-gd5bg" podUID="0037c462-11bc-4f13-9d5a-b4caaec482f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused"
Nov 03 20:26:13.525114 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:13.525081    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-g262x"]
Nov 03 20:26:13.652973 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:13.652939    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-dc7b94-667b5756b8-ns2z5"]
Nov 03 20:26:13.685838 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:26:13.670958    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8aa4c48e_92c6_49c8_acd3_4bc441ac8ca7.slice/crio-894672646f01bfbb4093e78ccb59e3eab82f7fe17ad848b02b9f19036678c792 WatchSource:0}: Error finding container 894672646f01bfbb4093e78ccb59e3eab82f7fe17ad848b02b9f19036678c792: Status 404 returned error can't find the container with id 894672646f01bfbb4093e78ccb59e3eab82f7fe17ad848b02b9f19036678c792
Nov 03 20:26:13.685838 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:13.683099    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:26:13.685838 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:13.683195    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:26:13.685838 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:13.683246    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:26:13.715657 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:13.694043    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dc7b94-667b5756b8-ns2z5"
Nov 03 20:26:13.715657 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:13.710106    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-dc7b94-667b5756b8-ns2z5"]
Nov 03 20:26:13.830844 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:13.778035    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-dc7b94-serving-cert\""
Nov 03 20:26:13.938104 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:13.881054    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2635dffb-14f1-49cb-8d32-8f539c28ed30-maas-api-tls\") pod \"maas-api-e2e-api-a-dc7b94-667b5756b8-ns2z5\" (UID: \"2635dffb-14f1-49cb-8d32-8f539c28ed30\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dc7b94-667b5756b8-ns2z5"
Nov 03 20:26:13.938104 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:13.881239    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2635dffb-14f1-49cb-8d32-8f539c28ed30-openshift-service-ca\") pod \"maas-api-e2e-api-a-dc7b94-667b5756b8-ns2z5\" (UID: \"2635dffb-14f1-49cb-8d32-8f539c28ed30\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dc7b94-667b5756b8-ns2z5"
Nov 03 20:26:13.938104 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:13.881368    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxmdh\" (UniqueName: \"kubernetes.io/projected/2635dffb-14f1-49cb-8d32-8f539c28ed30-kube-api-access-mxmdh\") pod \"maas-api-e2e-api-a-dc7b94-667b5756b8-ns2z5\" (UID: \"2635dffb-14f1-49cb-8d32-8f539c28ed30\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dc7b94-667b5756b8-ns2z5"
Nov 03 20:26:13.938104 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:13.881489    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2635dffb-14f1-49cb-8d32-8f539c28ed30-metrics-certs\") pod \"maas-api-e2e-api-a-dc7b94-667b5756b8-ns2z5\" (UID: \"2635dffb-14f1-49cb-8d32-8f539c28ed30\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dc7b94-667b5756b8-ns2z5"
Nov 03 20:26:14.001668 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:14.001492    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2635dffb-14f1-49cb-8d32-8f539c28ed30-maas-api-tls\") pod \"maas-api-e2e-api-a-dc7b94-667b5756b8-ns2z5\" (UID: \"2635dffb-14f1-49cb-8d32-8f539c28ed30\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dc7b94-667b5756b8-ns2z5"
Nov 03 20:26:14.002226 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:14.002120    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2635dffb-14f1-49cb-8d32-8f539c28ed30-openshift-service-ca\") pod \"maas-api-e2e-api-a-dc7b94-667b5756b8-ns2z5\" (UID: \"2635dffb-14f1-49cb-8d32-8f539c28ed30\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dc7b94-667b5756b8-ns2z5"
Nov 03 20:26:14.002682 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:14.002554    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxmdh\" (UniqueName: \"kubernetes.io/projected/2635dffb-14f1-49cb-8d32-8f539c28ed30-kube-api-access-mxmdh\") pod \"maas-api-e2e-api-a-dc7b94-667b5756b8-ns2z5\" (UID: \"2635dffb-14f1-49cb-8d32-8f539c28ed30\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dc7b94-667b5756b8-ns2z5"
Nov 03 20:26:14.003155 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:14.003032    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2635dffb-14f1-49cb-8d32-8f539c28ed30-metrics-certs\") pod \"maas-api-e2e-api-a-dc7b94-667b5756b8-ns2z5\" (UID: \"2635dffb-14f1-49cb-8d32-8f539c28ed30\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dc7b94-667b5756b8-ns2z5"
Nov 03 20:26:14.032923 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:14.032890    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2635dffb-14f1-49cb-8d32-8f539c28ed30-metrics-certs\") pod \"maas-api-e2e-api-a-dc7b94-667b5756b8-ns2z5\" (UID: \"2635dffb-14f1-49cb-8d32-8f539c28ed30\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dc7b94-667b5756b8-ns2z5"
Nov 03 20:26:14.109798 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:14.051093    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2635dffb-14f1-49cb-8d32-8f539c28ed30-openshift-service-ca\") pod \"maas-api-e2e-api-a-dc7b94-667b5756b8-ns2z5\" (UID: \"2635dffb-14f1-49cb-8d32-8f539c28ed30\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dc7b94-667b5756b8-ns2z5"
Nov 03 20:26:14.109798 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:14.064968    5570 patch_prober.go:29] interesting pod/e2e-mt-eb87f429-openshift-default-64649dfdb5-zdsqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body=
Nov 03 20:26:14.109798 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:14.065025    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zdsqp" podUID="f747f534-4e45-446f-948e-c66ce316db5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused"
Nov 03 20:26:14.170863 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:14.170796    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2635dffb-14f1-49cb-8d32-8f539c28ed30-maas-api-tls\") pod \"maas-api-e2e-api-a-dc7b94-667b5756b8-ns2z5\" (UID: \"2635dffb-14f1-49cb-8d32-8f539c28ed30\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dc7b94-667b5756b8-ns2z5"
Nov 03 20:26:14.175981 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:14.175899    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zdsqp"
Nov 03 20:26:14.176692 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:14.176630    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxmdh\" (UniqueName: \"kubernetes.io/projected/2635dffb-14f1-49cb-8d32-8f539c28ed30-kube-api-access-mxmdh\") pod \"maas-api-e2e-api-a-dc7b94-667b5756b8-ns2z5\" (UID: \"2635dffb-14f1-49cb-8d32-8f539c28ed30\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dc7b94-667b5756b8-ns2z5"
Nov 03 20:26:14.247176 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:14.247138    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-j7dz6" event={"ID":"22ea3ab0-6acc-4558-9158-eaa289924f35","Type":"ContainerStarted","Data":"c35d6c2c4780ada7348150b2cfc4cd1948d3f225714d8b58219a774af050416b"}
Nov 03 20:26:14.252028 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:14.251952    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-j7dz6" event={"ID":"22ea3ab0-6acc-4558-9158-eaa289924f35","Type":"ContainerStarted","Data":"14b3d9881b512a53a263be0260dab7169a79def626efceacb6eb0073305b2136"}
Nov 03 20:26:14.259902 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:14.257819    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-g262x" event={"ID":"8aa4c48e-92c6-49c8-acd3-4bc441ac8ca7","Type":"ContainerStarted","Data":"894672646f01bfbb4093e78ccb59e3eab82f7fe17ad848b02b9f19036678c792"}
Nov 03 20:26:14.298482 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:14.298419    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-j7dz6" podStartSLOduration=2.298400234 podStartE2EDuration="2.298400234s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:26:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:26:14.291815524 +0000 UTC m=+1308.444477412" watchObservedRunningTime="2026-11-03 20:26:14.298400234 +0000 UTC m=+1308.451062122"
Nov 03 20:26:14.331995 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:14.331627    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dc7b94-667b5756b8-ns2z5"
Nov 03 20:26:14.382873 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:14.378371    5570 patch_prober.go:29] interesting pod/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-rbvqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body=
Nov 03 20:26:14.382873 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:14.378428    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-rbvqw" podUID="ae5a764e-25f7-492e-b4aa-8f2696e22f37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused"
Nov 03 20:26:14.445961 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:14.445305    5570 patch_prober.go:29] interesting pod/e2e-mt-eb87f429-openshift-default-64649dfdb5-gd5bg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body=
Nov 03 20:26:14.445961 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:14.445365    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-gd5bg" podUID="0037c462-11bc-4f13-9d5a-b4caaec482f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused"
Nov 03 20:26:14.850971 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:14.850885    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-j7dz6"
Nov 03 20:26:14.857813 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:14.857420    5570 patch_prober.go:29] interesting pod/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-j7dz6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body=
Nov 03 20:26:14.857813 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:14.857499    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-j7dz6" podUID="22ea3ab0-6acc-4558-9158-eaa289924f35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused"
Nov 03 20:26:15.049793 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:15.049152    5570 patch_prober.go:29] interesting pod/e2e-mt-eb87f429-openshift-default-64649dfdb5-zdsqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body=
Nov 03 20:26:15.049793 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:15.049218    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zdsqp" podUID="f747f534-4e45-446f-948e-c66ce316db5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused"
Nov 03 20:26:15.383331 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:15.382526    5570 patch_prober.go:29] interesting pod/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-rbvqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body=
Nov 03 20:26:15.383331 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:15.382588    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-rbvqw" podUID="ae5a764e-25f7-492e-b4aa-8f2696e22f37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused"
Nov 03 20:26:15.556505 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:15.513308    5570 patch_prober.go:29] interesting pod/e2e-mt-eb87f429-openshift-default-64649dfdb5-gd5bg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body=
Nov 03 20:26:15.556505 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:15.513373    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-gd5bg" podUID="0037c462-11bc-4f13-9d5a-b4caaec482f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused"
Nov 03 20:26:15.753809 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:15.747136    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/simulator-w0-e40816-kserve-688f79796f-2tvw5" podUID="c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Nov 03 20:26:15.778136 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:15.777670    5570 patch_prober.go:29] interesting pod/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-jshb8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:26:15.778136 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:15.777731    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-jshb8" podUID="2137488e-8e6c-46f7-a0ed-b4721d4f2645" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:26:15.958456 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:15.857555    5570 patch_prober.go:29] interesting pod/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-j7dz6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body=
Nov 03 20:26:15.958456 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:15.857622    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-j7dz6" podUID="22ea3ab0-6acc-4558-9158-eaa289924f35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused"
Nov 03 20:26:16.050825 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:16.044054    5570 patch_prober.go:29] interesting pod/e2e-mt-eb87f429-openshift-default-64649dfdb5-zdsqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body=
Nov 03 20:26:16.050825 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:16.044129    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zdsqp" podUID="f747f534-4e45-446f-948e-c66ce316db5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused"
Nov 03 20:26:16.064541 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:16.064483    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/distinct-2-w4-e80203-kserve-756f9b9bc6-69gxj" podUID="ccf5dc2f-f116-4917-bb1a-b136d8ed17f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:26:16.738749 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:16.691155    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/distinct-2-w3-482722-kserve-74bf58cf7d-v9cjw" podUID="9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded"
Nov 03 20:26:16.819811 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:16.809191    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-2-w6-9e1bcb-kserve-7d96d9f6f6-4f7fl" podUID="56353b0f-08ee-4ada-9dc0-ac5afd5a59c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:26:17.130422 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:17.079580    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz" podUID="3a715ef1-38e9-4386-9eec-e10e8df11a10" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": context deadline exceeded"
Nov 03 20:26:17.130422 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:17.080154    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/simulator-w0-e40816-kserve-688f79796f-2tvw5" podUID="c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Nov 03 20:26:17.130422 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:17.081286    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/premium-w2-9973d9-kserve-5c6dd4669c-5gv8x" podUID="cefc89ad-7ae1-433b-979d-6601678a6d98" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Nov 03 20:26:17.773794 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:17.765263    5570 patch_prober.go:29] interesting pod/e2e-mt-eb87f429-openshift-default-64649dfdb5-gd5bg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body=
Nov 03 20:26:17.773794 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:17.765337    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-gd5bg" podUID="0037c462-11bc-4f13-9d5a-b4caaec482f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused"
Nov 03 20:26:17.799029 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:17.776887    5570 patch_prober.go:29] interesting pod/e2e-mt-eb87f429-openshift-default-64649dfdb5-gd5bg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body=
Nov 03 20:26:17.799029 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:17.776955    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-gd5bg" podUID="0037c462-11bc-4f13-9d5a-b4caaec482f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused"
Nov 03 20:26:18.119863 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:18.086272    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/distinct-2-w0-e40816-kserve-79fb46654d-s242x" podUID="4ad9b048-925c-4d01-a839-a5f25d4dd02d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:26:18.464105 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:18.120030    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/premium-w1-6fc748-kserve-7474db9cd7-nf2xk" podUID="830e062e-83fc-4de7-91a7-5534dac1d46b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Nov 03 20:26:18.464105 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:26:18.199806    5570 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="2.137s"
Nov 03 20:26:18.464105 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:18.199928    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e40816/simulator-w0-e40816-kserve-688f79796f-2tvw5"
Nov 03 20:26:18.464105 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:18.380941    5570 patch_prober.go:29] interesting pod/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-j7dz6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body=
Nov 03 20:26:18.464105 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:18.394785    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-j7dz6" podUID="22ea3ab0-6acc-4558-9158-eaa289924f35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused"
Nov 03 20:26:18.464105 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:18.395219    5570 patch_prober.go:29] interesting pod/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-rbvqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body=
Nov 03 20:26:18.464105 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:18.395275    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-rbvqw" podUID="ae5a764e-25f7-492e-b4aa-8f2696e22f37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused"
Nov 03 20:26:18.464105 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:18.460688    5570 patch_prober.go:29] interesting pod/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-rbvqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body=
Nov 03 20:26:18.464105 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:18.460760    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-rbvqw" podUID="ae5a764e-25f7-492e-b4aa-8f2696e22f37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused"
Nov 03 20:26:18.464105 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:18.460882    5570 patch_prober.go:29] interesting pod/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-j7dz6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body=
Nov 03 20:26:18.464105 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:18.460921    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-j7dz6" podUID="22ea3ab0-6acc-4558-9158-eaa289924f35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused"
Nov 03 20:26:18.665650 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:18.665152    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/distinct-2-w4-e80203-kserve-756f9b9bc6-69gxj" podUID="ccf5dc2f-f116-4917-bb1a-b136d8ed17f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:26:18.665650 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:18.665372    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/premium-w4-e80203-kserve-5f9f445456-bms9d" podUID="6ad76efc-20c2-4951-b22a-c5a39cd39ed4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:26:18.699225 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:18.698917    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/distinct-w3-482722-kserve-d876967c-kd488" podUID="9a97f9cc-2261-471d-9053-51ee965850bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:26:18.699225 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:18.699096    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh" podUID="2d3647f2-89e9-4908-a8ee-d84e527abeb0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Nov 03 20:26:18.741615 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:18.741436    5570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-2-w6-9e1bcb-kserve-7d96d9f6f6-4f7fl" podUID="56353b0f-08ee-4ada-9dc0-ac5afd5a59c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:26:18.752089 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:18.494392    5570 patch_prober.go:29] interesting pod/e2e-mt-eb87f429-openshift-default-64649dfdb5-zdsqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body=
Nov 03 20:26:18.764568 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:18.752283    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zdsqp" podUID="f747f534-4e45-446f-948e-c66ce316db5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused"
Nov 03 20:26:18.929120 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:18.929062    5570 patch_prober.go:29] interesting pod/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-j7dz6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body=
Nov 03 20:26:18.929848 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:18.929811    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-j7dz6" podUID="22ea3ab0-6acc-4558-9158-eaa289924f35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused"
Nov 03 20:26:18.935624 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:18.932546    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-cpfzl" podUID="30b21f09-fd9e-4836-9fc3-c19f818c9022" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:26:19.049854 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:19.049218    5570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-9e1bcb/premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh" podUID="2d3647f2-89e9-4908-a8ee-d84e527abeb0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:26:19.122808 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:19.097961    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/distinct-2-w0-e40816-kserve-79fb46654d-s242x" podUID="4ad9b048-925c-4d01-a839-a5f25d4dd02d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:26:19.183835 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:19.129599    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/premium-w1-6fc748-kserve-7474db9cd7-nf2xk" podUID="830e062e-83fc-4de7-91a7-5534dac1d46b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:26:19.412860 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:19.394567    5570 patch_prober.go:29] interesting pod/e2e-mt-eb87f429-openshift-default-64649dfdb5-gd5bg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body=
Nov 03 20:26:19.412860 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:19.394624    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-gd5bg" podUID="0037c462-11bc-4f13-9d5a-b4caaec482f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused"
Nov 03 20:26:19.441832 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:19.430383    5570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-e40816/simulator-w0-e40816-kserve-688f79796f-2tvw5" podUID="c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:26:19.442150 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:19.435985    5570 patch_prober.go:29] interesting pod/e2e-mt-eb87f429-openshift-default-64649dfdb5-zdsqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body=
Nov 03 20:26:19.442284 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:19.442259    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zdsqp" podUID="f747f534-4e45-446f-948e-c66ce316db5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused"
Nov 03 20:26:19.451808 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:19.442650    5570 patch_prober.go:29] interesting pod/e2e-mt-eb87f429-openshift-default-64649dfdb5-zdsqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body=
Nov 03 20:26:19.452102 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:19.452068    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zdsqp" podUID="f747f534-4e45-446f-948e-c66ce316db5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused"
Nov 03 20:26:19.512943 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:19.452487    5570 patch_prober.go:29] interesting pod/e2e-mt-eb87f429-openshift-default-64649dfdb5-gd5bg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body=
Nov 03 20:26:19.512943 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:19.452523    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-gd5bg" podUID="0037c462-11bc-4f13-9d5a-b4caaec482f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused"
Nov 03 20:26:19.512943 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:19.494206    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/distinct-2-w3-482722-kserve-74bf58cf7d-v9cjw" podUID="9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded"
Nov 03 20:26:19.512943 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:19.494516    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-2-w6-9e1bcb-kserve-7d96d9f6f6-4f7fl" podUID="56353b0f-08ee-4ada-9dc0-ac5afd5a59c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded"
Nov 03 20:26:19.512943 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:19.494686    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/premium-w3-482722-kserve-689dc7bd5d-x8rjp" podUID="369045c3-4d6a-49a7-9bc6-f2939883bee8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Nov 03 20:26:19.512943 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:19.494866    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz" podUID="3a715ef1-38e9-4386-9eec-e10e8df11a10" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": context deadline exceeded"
Nov 03 20:26:19.512943 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:19.495029    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/simulator-w0-e40816-kserve-688f79796f-2tvw5" podUID="c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Nov 03 20:26:19.512943 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:19.495197    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/premium-w2-9973d9-kserve-5c6dd4669c-5gv8x" podUID="cefc89ad-7ae1-433b-979d-6601678a6d98" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Nov 03 20:26:19.667901 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:19.665866    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/distinct-2-w4-e80203-kserve-756f9b9bc6-69gxj" podUID="ccf5dc2f-f116-4917-bb1a-b136d8ed17f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded"
Nov 03 20:26:19.667901 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:19.666223    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/premium-w4-e80203-kserve-5f9f445456-bms9d" podUID="6ad76efc-20c2-4951-b22a-c5a39cd39ed4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Nov 03 20:26:19.705162 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:26:19.696956    5570 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.497s"
Nov 03 20:26:19.705162 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:19.697031    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-dc7b94-667b5756b8-ns2z5"]
Nov 03 20:26:19.705162 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:19.697332    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fca5b006-j488g"]
Nov 03 20:26:19.705162 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:19.697425    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-e80203/distinct-2-w4-e80203-kserve-756f9b9bc6-69gxj"
Nov 03 20:26:19.705162 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:19.697467    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-g262x" event={"ID":"8aa4c48e-92c6-49c8-acd3-4bc441ac8ca7","Type":"ContainerStarted","Data":"83026a35e395a00762993ad189f351188a419bba8a38d57433b5dae409cf0502"}
Nov 03 20:26:19.705162 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:19.700856    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e40816/distinct-2-w0-e40816-kserve-79fb46654d-s242x"
Nov 03 20:26:19.778148 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:19.700941    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6fc748/premium-w1-6fc748-kserve-7474db9cd7-nf2xk"
Nov 03 20:26:19.778148 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:19.715996    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-482722/distinct-2-w3-482722-kserve-74bf58cf7d-v9cjw"
Nov 03 20:26:19.778148 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:19.716069    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-2-w6-9e1bcb-kserve-7d96d9f6f6-4f7fl"
Nov 03 20:26:19.778148 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:19.716110    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz"
Nov 03 20:26:19.778148 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:19.716169    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9973d9/premium-w2-9973d9-kserve-5c6dd4669c-5gv8x"
Nov 03 20:26:19.778148 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:19.716229    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-e80203/premium-w4-e80203-kserve-5f9f445456-bms9d"
Nov 03 20:26:19.778148 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:19.716312    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh" podUID="2d3647f2-89e9-4908-a8ee-d84e527abeb0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:26:19.778148 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:19.749494    5570 patch_prober.go:29] interesting pod/e2e-worker-w2-9973d9-openshift-default-ffb6475-kmqsw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:26:19.778148 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:19.749573    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-kmqsw" podUID="28aada86-170c-4935-80ed-e8c5ee503c14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:26:19.820718 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:19.780266    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-9e1bcb/premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh"
Nov 03 20:26:19.820718 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:19.780443    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/distinct-w3-482722-kserve-d876967c-kd488" podUID="9a97f9cc-2261-471d-9053-51ee965850bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded"
Nov 03 20:26:19.820718 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:19.812423    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-482722/distinct-w3-482722-kserve-d876967c-kd488"
Nov 03 20:26:19.880803 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:19.869451    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-g262x"
Nov 03 20:26:19.913891 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:19.913845    5570 patch_prober.go:29] interesting pod/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sf2lg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:26:20.071581 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:20.056645    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sf2lg" podUID="07093a96-d60c-490f-910b-734f41752c17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:26:20.110952 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:20.110910    5570 patch_prober.go:29] interesting pod/e2e-mt-eb87f429-openshift-default-64649dfdb5-zdsqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body=
Nov 03 20:26:20.111168 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:20.110975    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zdsqp" podUID="f747f534-4e45-446f-948e-c66ce316db5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused"
Nov 03 20:26:20.124110 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:20.123375    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/distinct-2-w0-e40816-kserve-79fb46654d-s242x" podUID="4ad9b048-925c-4d01-a839-a5f25d4dd02d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:26:20.124447 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:20.124118    5570 patch_prober.go:29] interesting pod/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-g262x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body=
Nov 03 20:26:20.124447 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:20.124177    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-g262x" podUID="8aa4c48e-92c6-49c8-acd3-4bc441ac8ca7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused"
Nov 03 20:26:20.157020 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:20.125841    5570 patch_prober.go:29] interesting pod/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-rbvqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body=
Nov 03 20:26:20.157020 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:20.125893    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-rbvqw" podUID="ae5a764e-25f7-492e-b4aa-8f2696e22f37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused"
Nov 03 20:26:20.157020 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:20.155500    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/premium-w1-6fc748-kserve-7474db9cd7-nf2xk" podUID="830e062e-83fc-4de7-91a7-5534dac1d46b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:26:20.263012 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:20.262951    5570 patch_prober.go:29] interesting pod/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-j7dz6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body=
Nov 03 20:26:20.263330 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:20.263285    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-j7dz6" podUID="22ea3ab0-6acc-4558-9158-eaa289924f35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused"
Nov 03 20:26:20.413968 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:20.413198    5570 patch_prober.go:29] interesting pod/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-rbvqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body=
Nov 03 20:26:20.413968 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:20.413263    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-rbvqw" podUID="ae5a764e-25f7-492e-b4aa-8f2696e22f37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused"
Nov 03 20:26:20.540200 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:20.540147    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/premium-w2-9973d9-kserve-5c6dd4669c-5gv8x" podUID="cefc89ad-7ae1-433b-979d-6601678a6d98" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:26:20.541164 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:20.541089    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/distinct-2-w3-482722-kserve-74bf58cf7d-v9cjw" podUID="9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded"
Nov 03 20:26:20.541636 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:20.541611    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-2-w6-9e1bcb-kserve-7d96d9f6f6-4f7fl" podUID="56353b0f-08ee-4ada-9dc0-ac5afd5a59c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded"
Nov 03 20:26:20.542072 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:20.542048    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/premium-w3-482722-kserve-689dc7bd5d-x8rjp" podUID="369045c3-4d6a-49a7-9bc6-f2939883bee8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Nov 03 20:26:20.654799 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:20.544558    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz" podUID="3a715ef1-38e9-4386-9eec-e10e8df11a10" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": context deadline exceeded"
Nov 03 20:26:20.766823 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:20.544603    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/simulator-w0-e40816-kserve-688f79796f-2tvw5" podUID="c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Nov 03 20:26:20.767454 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:20.564265    5570 patch_prober.go:29] interesting pod/e2e-mt-eb87f429-openshift-default-64649dfdb5-gd5bg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body=
Nov 03 20:26:20.767671 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:20.767641    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-gd5bg" podUID="0037c462-11bc-4f13-9d5a-b4caaec482f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused"
Nov 03 20:26:20.767805 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:20.720591    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh" podUID="2d3647f2-89e9-4908-a8ee-d84e527abeb0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Nov 03 20:26:20.771798 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:20.694520    5570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-e80203/distinct-2-w4-e80203-kserve-756f9b9bc6-69gxj" podUID="ccf5dc2f-f116-4917-bb1a-b136d8ed17f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/health\": context deadline exceeded"
Nov 03 20:26:20.927714 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:20.927677    5570 patch_prober.go:29] interesting pod/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-g262x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body=
Nov 03 20:26:20.941934 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:20.927935    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-g262x" podUID="8aa4c48e-92c6-49c8-acd3-4bc441ac8ca7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused"
Nov 03 20:26:20.941934 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:20.791933    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/distinct-w3-482722-kserve-d876967c-kd488" podUID="9a97f9cc-2261-471d-9053-51ee965850bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded"
Nov 03 20:26:20.941934 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:20.803319    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/premium-w4-e80203-kserve-5f9f445456-bms9d" podUID="6ad76efc-20c2-4951-b22a-c5a39cd39ed4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Nov 03 20:26:20.993208 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:20.803274    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/distinct-2-w4-e80203-kserve-756f9b9bc6-69gxj" podUID="ccf5dc2f-f116-4917-bb1a-b136d8ed17f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded"
Nov 03 20:26:21.367803 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:26:21.367317    5570 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.349s"
Nov 03 20:26:21.367803 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:21.367457    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fca5b006-j488g"]
Nov 03 20:26:21.367803 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:21.367525    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-482722/premium-w3-482722-kserve-689dc7bd5d-x8rjp"
Nov 03 20:26:21.589798 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:21.525662    5570 patch_prober.go:29] interesting pod/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-j7dz6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body=
Nov 03 20:26:21.589798 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:21.525728    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-j7dz6" podUID="22ea3ab0-6acc-4558-9158-eaa289924f35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused"
Nov 03 20:26:21.702519 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:21.702465    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/distinct-2-w0-e40816-kserve-79fb46654d-s242x" podUID="4ad9b048-925c-4d01-a839-a5f25d4dd02d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded"
Nov 03 20:26:21.736919 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:21.736875    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/premium-w1-6fc748-kserve-7474db9cd7-nf2xk" podUID="830e062e-83fc-4de7-91a7-5534dac1d46b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Nov 03 20:26:21.738306 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:21.738275    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/premium-w3-482722-kserve-689dc7bd5d-x8rjp" podUID="369045c3-4d6a-49a7-9bc6-f2939883bee8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Nov 03 20:26:21.779939 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:21.779880    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh" podUID="2d3647f2-89e9-4908-a8ee-d84e527abeb0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:26:21.785814 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:21.785754    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz" podUID="3a715ef1-38e9-4386-9eec-e10e8df11a10" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": context deadline exceeded"
Nov 03 20:26:21.800924 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:21.792970    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/simulator-w0-e40816-kserve-688f79796f-2tvw5" podUID="c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Nov 03 20:26:21.854537 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:21.854492    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/premium-w2-9973d9-kserve-5c6dd4669c-5gv8x" podUID="cefc89ad-7ae1-433b-979d-6601678a6d98" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Nov 03 20:26:21.909651 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:21.909602    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/distinct-2-w3-482722-kserve-74bf58cf7d-v9cjw" podUID="9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded"
Nov 03 20:26:21.942726 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:21.939223    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/premium-w4-e80203-kserve-5f9f445456-bms9d" podUID="6ad76efc-20c2-4951-b22a-c5a39cd39ed4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:26:21.954293 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:21.854969    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-g262x" podStartSLOduration=9.854947257 podStartE2EDuration="9.854947257s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:26:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:26:19.52219509 +0000 UTC m=+1313.674857001" watchObservedRunningTime="2026-11-03 20:26:21.854947257 +0000 UTC m=+1316.007609145"
Nov 03 20:26:22.145395 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:22.145349    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/distinct-2-w4-e80203-kserve-756f9b9bc6-69gxj" podUID="ccf5dc2f-f116-4917-bb1a-b136d8ed17f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded"
Nov 03 20:26:22.190885 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:22.020343    5570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-482722/distinct-2-w3-482722-kserve-74bf58cf7d-v9cjw" podUID="9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/health\": context deadline exceeded"
Nov 03 20:26:22.194797 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:21.884951    5570 patch_prober.go:29] interesting pod/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-j7dz6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body=
Nov 03 20:26:22.215734 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:22.215624    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-j7dz6" podUID="22ea3ab0-6acc-4558-9158-eaa289924f35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused"
Nov 03 20:26:22.216014 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:21.887191    5570 patch_prober.go:29] interesting pod/e2e-mt-eb87f429-openshift-default-64649dfdb5-gd5bg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body=
Nov 03 20:26:22.216179 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:22.216143    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-gd5bg" podUID="0037c462-11bc-4f13-9d5a-b4caaec482f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused"
Nov 03 20:26:22.216277 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:21.887947    5570 patch_prober.go:29] interesting pod/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-g262x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body=
Nov 03 20:26:22.216366 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:22.216348    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-g262x" podUID="8aa4c48e-92c6-49c8-acd3-4bc441ac8ca7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused"
Nov 03 20:26:22.216530 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:21.957130    5570 patch_prober.go:29] interesting pod/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-rbvqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body=
Nov 03 20:26:22.216649 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:22.216629    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-rbvqw" podUID="ae5a764e-25f7-492e-b4aa-8f2696e22f37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused"
Nov 03 20:26:22.224029 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:21.987176    5570 patch_prober.go:29] interesting pod/e2e-mt-eb87f429-openshift-default-64649dfdb5-zdsqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body=
Nov 03 20:26:22.224278 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:22.224247    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zdsqp" podUID="f747f534-4e45-446f-948e-c66ce316db5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused"
Nov 03 20:26:22.234817 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:22.234088    5570 patch_prober.go:29] interesting pod/e2e-mt-eb87f429-openshift-default-64649dfdb5-zdsqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body=
Nov 03 20:26:22.234817 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:22.234164    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zdsqp" podUID="f747f534-4e45-446f-948e-c66ce316db5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused"
Nov 03 20:26:22.409823 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:21.986915    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/distinct-w3-482722-kserve-d876967c-kd488" podUID="9a97f9cc-2261-471d-9053-51ee965850bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded"
Nov 03 20:26:22.503448 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:22.476071    5570 patch_prober.go:29] interesting pod/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-rbvqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body=
Nov 03 20:26:22.503448 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:22.476138    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-rbvqw" podUID="ae5a764e-25f7-492e-b4aa-8f2696e22f37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused"
Nov 03 20:26:22.503448 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:22.476702    5570 patch_prober.go:29] interesting pod/e2e-mt-eb87f429-openshift-default-64649dfdb5-gd5bg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body=
Nov 03 20:26:22.503448 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:22.476740    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-gd5bg" podUID="0037c462-11bc-4f13-9d5a-b4caaec482f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused"
Nov 03 20:26:22.720995 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:22.720941    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/distinct-2-w0-e40816-kserve-79fb46654d-s242x" podUID="4ad9b048-925c-4d01-a839-a5f25d4dd02d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:26:22.829087 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:22.737385    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/premium-w1-6fc748-kserve-7474db9cd7-nf2xk" podUID="830e062e-83fc-4de7-91a7-5534dac1d46b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:26:22.975115 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:22.975050    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/premium-w2-9973d9-kserve-5c6dd4669c-5gv8x" podUID="cefc89ad-7ae1-433b-979d-6601678a6d98" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Nov 03 20:26:23.015126 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:23.015075    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/distinct-2-w3-482722-kserve-74bf58cf7d-v9cjw" podUID="9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:26:23.088922 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:22.739637    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/premium-w3-482722-kserve-689dc7bd5d-x8rjp" podUID="369045c3-4d6a-49a7-9bc6-f2939883bee8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:26:23.156980 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:23.156931    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/distinct-2-w4-e80203-kserve-756f9b9bc6-69gxj" podUID="ccf5dc2f-f116-4917-bb1a-b136d8ed17f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:26:23.284884 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:23.280007    5570 patch_prober.go:29] interesting pod/e2e-mt-eb87f429-openshift-default-64649dfdb5-zdsqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body=
Nov 03 20:26:23.284884 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:23.280092    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zdsqp" podUID="f747f534-4e45-446f-948e-c66ce316db5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused"
Nov 03 20:26:23.284884 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:23.239106    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/premium-w4-e80203-kserve-5f9f445456-bms9d" podUID="6ad76efc-20c2-4951-b22a-c5a39cd39ed4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Nov 03 20:26:23.326977 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:23.326928    5570 patch_prober.go:29] interesting pod/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-j7dz6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body=
Nov 03 20:26:23.327301 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:23.327252    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-j7dz6" podUID="22ea3ab0-6acc-4558-9158-eaa289924f35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused"
Nov 03 20:26:23.460835 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:23.215414    5570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-6fc748/premium-w1-6fc748-kserve-7474db9cd7-nf2xk" podUID="830e062e-83fc-4de7-91a7-5534dac1d46b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:26:23.468122 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:23.238005    5570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-e40816/distinct-2-w0-e40816-kserve-79fb46654d-s242x" podUID="4ad9b048-925c-4d01-a839-a5f25d4dd02d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/health\": context deadline exceeded"
Nov 03 20:26:23.491998 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:23.491945    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-2-w6-9e1bcb-kserve-7d96d9f6f6-4f7fl" podUID="56353b0f-08ee-4ada-9dc0-ac5afd5a59c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded"
Nov 03 20:26:23.594609 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:23.594549    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/distinct-w3-482722-kserve-d876967c-kd488" podUID="9a97f9cc-2261-471d-9053-51ee965850bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:26:23.727229 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:23.693217    5570 patch_prober.go:29] interesting pod/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-rbvqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body=
Nov 03 20:26:23.727229 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:23.693288    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-rbvqw" podUID="ae5a764e-25f7-492e-b4aa-8f2696e22f37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused"
Nov 03 20:26:23.727229 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:23.698188    5570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-e80203/premium-w4-e80203-kserve-5f9f445456-bms9d" podUID="6ad76efc-20c2-4951-b22a-c5a39cd39ed4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": context deadline exceeded"
Nov 03 20:26:23.784843 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:23.780969    5570 patch_prober.go:29] interesting pod/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-g262x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body=
Nov 03 20:26:23.784843 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:23.782068    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-g262x" podUID="8aa4c48e-92c6-49c8-acd3-4bc441ac8ca7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused"
Nov 03 20:26:23.887840 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:23.887783    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/premium-w1-6fc748-kserve-7474db9cd7-nf2xk" podUID="830e062e-83fc-4de7-91a7-5534dac1d46b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:26:24.263093 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:24.018076    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/distinct-2-w0-e40816-kserve-79fb46654d-s242x" podUID="4ad9b048-925c-4d01-a839-a5f25d4dd02d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:26:24.283095 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:24.098956    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/premium-w3-482722-kserve-689dc7bd5d-x8rjp" podUID="369045c3-4d6a-49a7-9bc6-f2939883bee8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:26:24.283395 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:24.161762    5570 patch_prober.go:29] interesting pod/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-j7dz6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body=
Nov 03 20:26:24.300058 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:24.283543    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-j7dz6" podUID="22ea3ab0-6acc-4558-9158-eaa289924f35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused"
Nov 03 20:26:24.300058 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:24.262436    5570 patch_prober.go:29] interesting pod/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-g262x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body=
Nov 03 20:26:24.300058 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:24.283631    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-g262x" podUID="8aa4c48e-92c6-49c8-acd3-4bc441ac8ca7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused"
Nov 03 20:26:24.300058 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:24.263108    5570 patch_prober.go:29] interesting pod/e2e-mt-eb87f429-openshift-default-64649dfdb5-zdsqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body=
Nov 03 20:26:24.300058 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:24.283693    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zdsqp" podUID="f747f534-4e45-446f-948e-c66ce316db5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused"
Nov 03 20:26:24.503910 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:24.497479    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc403049-5b3e-45fa-80b8-6d1f53ab171c" path="/var/lib/kubelet/pods/fc403049-5b3e-45fa-80b8-6d1f53ab171c/volumes"
Nov 03 20:26:24.503910 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:26:24.498155    5570 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="2.514s"
Nov 03 20:26:24.503910 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:24.498389    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dc7b94-667b5756b8-ns2z5" event={"ID":"2635dffb-14f1-49cb-8d32-8f539c28ed30","Type":"ContainerStarted","Data":"a995b94940a51a8f8665ed1cf56377df2d8e2210b8165c602e9a00aaf73e7127"}
Nov 03 20:26:24.503910 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:24.498483    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-2-w6-9e1bcb-kserve-7d96d9f6f6-4f7fl"
Nov 03 20:26:24.503910 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:24.498533    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-rbvqw"
Nov 03 20:26:24.649912 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:24.649868    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/distinct-2-w4-e80203-kserve-756f9b9bc6-69gxj" podUID="ccf5dc2f-f116-4917-bb1a-b136d8ed17f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:26:24.785394 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:24.651854    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/premium-w4-e80203-kserve-5f9f445456-bms9d" podUID="6ad76efc-20c2-4951-b22a-c5a39cd39ed4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Nov 03 20:26:24.785394 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:24.654275    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh" podUID="2d3647f2-89e9-4908-a8ee-d84e527abeb0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Nov 03 20:26:24.785394 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:24.764514    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/distinct-w3-482722-kserve-d876967c-kd488" podUID="9a97f9cc-2261-471d-9053-51ee965850bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:26:24.785394 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:24.654366    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/simulator-w0-e40816-kserve-688f79796f-2tvw5" podUID="c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Nov 03 20:26:24.785394 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:24.766396    5570 patch_prober.go:29] interesting pod/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-rbvqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body=
Nov 03 20:26:24.785394 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:24.766444    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-rbvqw" podUID="ae5a764e-25f7-492e-b4aa-8f2696e22f37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused"
Nov 03 20:26:24.810373 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:24.654318    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz" podUID="3a715ef1-38e9-4386-9eec-e10e8df11a10" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:26:24.813068 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:24.813036    5570 patch_prober.go:29] interesting pod/e2e-mt-eb87f429-openshift-default-64649dfdb5-gd5bg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body=
Nov 03 20:26:24.813314 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:24.813270    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-gd5bg" podUID="0037c462-11bc-4f13-9d5a-b4caaec482f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused"
Nov 03 20:26:24.892810 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:24.888647    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/premium-w2-9973d9-kserve-5c6dd4669c-5gv8x" podUID="cefc89ad-7ae1-433b-979d-6601678a6d98" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Nov 03 20:26:24.919801 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:24.894711    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/distinct-2-w3-482722-kserve-74bf58cf7d-v9cjw" podUID="9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded"
Nov 03 20:26:24.966818 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:24.946143    5570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-482722/distinct-w3-482722-kserve-d876967c-kd488" podUID="9a97f9cc-2261-471d-9053-51ee965850bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/health\": context deadline exceeded"
Nov 03 20:26:24.966818 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:24.933000    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-2-w6-9e1bcb-kserve-7d96d9f6f6-4f7fl" podUID="56353b0f-08ee-4ada-9dc0-ac5afd5a59c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded"
Nov 03 20:26:24.966818 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:24.946484    5570 patch_prober.go:29] interesting pod/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-g262x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body=
Nov 03 20:26:24.966818 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:24.946542    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-g262x" podUID="8aa4c48e-92c6-49c8-acd3-4bc441ac8ca7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused"
Nov 03 20:26:24.985327 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:24.968219    5570 patch_prober.go:29] interesting pod/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-j7dz6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body=
Nov 03 20:26:24.985327 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:24.968324    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-j7dz6" podUID="22ea3ab0-6acc-4558-9158-eaa289924f35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused"
Nov 03 20:26:25.324800 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:25.309099    5570 patch_prober.go:29] interesting pod/e2e-mt-eb87f429-openshift-default-64649dfdb5-zdsqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body=
Nov 03 20:26:25.324800 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:25.309174    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zdsqp" podUID="f747f534-4e45-446f-948e-c66ce316db5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused"
Nov 03 20:26:25.507602 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:25.494876    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/premium-w3-482722-kserve-689dc7bd5d-x8rjp" podUID="369045c3-4d6a-49a7-9bc6-f2939883bee8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Nov 03 20:26:25.507602 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:25.495267    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/premium-w1-6fc748-kserve-7474db9cd7-nf2xk" podUID="830e062e-83fc-4de7-91a7-5534dac1d46b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Nov 03 20:26:25.507602 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:25.495560    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/distinct-2-w0-e40816-kserve-79fb46654d-s242x" podUID="4ad9b048-925c-4d01-a839-a5f25d4dd02d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded"
Nov 03 20:26:25.574426 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:25.572170    5570 patch_prober.go:29] interesting pod/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-rbvqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body=
Nov 03 20:26:25.579948 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:25.579902    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-rbvqw" podUID="ae5a764e-25f7-492e-b4aa-8f2696e22f37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused"
Nov 03 20:26:25.677155 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:25.677114    5570 patch_prober.go:29] interesting pod/e2e-mt-eb87f429-openshift-default-64649dfdb5-gd5bg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body=
Nov 03 20:26:25.677429 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:25.677400    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-gd5bg" podUID="0037c462-11bc-4f13-9d5a-b4caaec482f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused"
Nov 03 20:26:25.797854 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:25.776213    5570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-482722/premium-w3-482722-kserve-689dc7bd5d-x8rjp" podUID="369045c3-4d6a-49a7-9bc6-f2939883bee8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:26:25.797854 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:25.776333    5570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-9973d9/premium-w2-9973d9-kserve-5c6dd4669c-5gv8x" podUID="cefc89ad-7ae1-433b-979d-6601678a6d98" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": context deadline exceeded"
Nov 03 20:26:25.822816 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:25.798224    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/simulator-w0-e40816-kserve-688f79796f-2tvw5" podUID="c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:26:25.849790 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:25.849715    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/distinct-2-w4-e80203-kserve-756f9b9bc6-69gxj" podUID="ccf5dc2f-f116-4917-bb1a-b136d8ed17f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded"
Nov 03 20:26:25.857266 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:25.857210    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/premium-w4-e80203-kserve-5f9f445456-bms9d" podUID="6ad76efc-20c2-4951-b22a-c5a39cd39ed4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Nov 03 20:26:25.957658 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:25.889345    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh" podUID="2d3647f2-89e9-4908-a8ee-d84e527abeb0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Nov 03 20:26:25.957658 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:25.950103    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/premium-w2-9973d9-kserve-5c6dd4669c-5gv8x" podUID="cefc89ad-7ae1-433b-979d-6601678a6d98" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:26:25.957658 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:25.950472    5570 patch_prober.go:29] interesting pod/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-j7dz6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body=
Nov 03 20:26:25.957658 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:25.950520    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-j7dz6" podUID="22ea3ab0-6acc-4558-9158-eaa289924f35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused"
Nov 03 20:26:25.962485 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:25.962436    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz" podUID="3a715ef1-38e9-4386-9eec-e10e8df11a10" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": context deadline exceeded"
Nov 03 20:26:25.973701 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:25.973652    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-2-w6-9e1bcb-kserve-7d96d9f6f6-4f7fl" podUID="56353b0f-08ee-4ada-9dc0-ac5afd5a59c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:26:25.978799 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:25.974823    5570 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz" podUID="3a715ef1-38e9-4386-9eec-e10e8df11a10" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/health\": context deadline exceeded"
Nov 03 20:26:25.989372 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:25.889481    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/distinct-w3-482722-kserve-d876967c-kd488" podUID="9a97f9cc-2261-471d-9053-51ee965850bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded"
Nov 03 20:26:26.133858 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:26.019687    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/distinct-2-w3-482722-kserve-74bf58cf7d-v9cjw" podUID="9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:26:26.145400 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:26.145352    5570 patch_prober.go:29] interesting pod/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-g262x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body=
Nov 03 20:26:26.145733 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:26.145681    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-g262x" podUID="8aa4c48e-92c6-49c8-acd3-4bc441ac8ca7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused"
Nov 03 20:26:26.326856 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:26.325853    5570 patch_prober.go:29] interesting pod/e2e-mt-eb87f429-openshift-default-64649dfdb5-nsjcd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:26:26.364174 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:26.350871    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-nsjcd" podUID="7b57a22a-cf28-4099-8883-c04c61807dfa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:26:26.364174 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:26.337074    5570 patch_prober.go:29] interesting pod/e2e-mt-eb87f429-openshift-default-64649dfdb5-zdsqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body=
Nov 03 20:26:26.364174 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:26.350978    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zdsqp" podUID="f747f534-4e45-446f-948e-c66ce316db5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused"
Nov 03 20:26:26.509830 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:26.505936    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/premium-w3-482722-kserve-689dc7bd5d-x8rjp" podUID="369045c3-4d6a-49a7-9bc6-f2939883bee8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Nov 03 20:26:26.509830 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:26.506326    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/premium-w1-6fc748-kserve-7474db9cd7-nf2xk" podUID="830e062e-83fc-4de7-91a7-5534dac1d46b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Nov 03 20:26:26.509830 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:26.506584    5570 patch_prober.go:29] interesting pod/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-rbvqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body=
Nov 03 20:26:26.509830 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:26.506846    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-rbvqw" podUID="ae5a764e-25f7-492e-b4aa-8f2696e22f37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused"
Nov 03 20:26:26.522009 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:26.521737    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/distinct-2-w0-e40816-kserve-79fb46654d-s242x" podUID="4ad9b048-925c-4d01-a839-a5f25d4dd02d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:26:26.522769 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:26:26.522744    5570 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.657s"
Nov 03 20:26:26.557935 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:26.548884    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-gd5bg"
Nov 03 20:26:26.557935 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:26.550050    5570 patch_prober.go:29] interesting pod/e2e-mt-eb87f429-openshift-default-64649dfdb5-gd5bg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body=
Nov 03 20:26:26.557935 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:26.550358    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-gd5bg" podUID="0037c462-11bc-4f13-9d5a-b4caaec482f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused"
Nov 03 20:26:26.806242 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:26.802861    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/simulator-w0-e40816-kserve-688f79796f-2tvw5" podUID="c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:26:26.895899 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:26.867847    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-2-w6-9e1bcb-kserve-7d96d9f6f6-4f7fl"
Nov 03 20:26:26.895899 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:26.868000    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-dc7b94-777ff98d5f-4l8x2"
Nov 03 20:26:26.895899 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:26.868045    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zdsqp"
Nov 03 20:26:26.895899 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:26.877676    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/distinct-2-w4-e80203-kserve-756f9b9bc6-69gxj" podUID="ccf5dc2f-f116-4917-bb1a-b136d8ed17f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:26:26.897355 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:26.897311    5570 patch_prober.go:29] interesting pod/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-j7dz6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body=
Nov 03 20:26:26.924194 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:26.924148    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-j7dz6" podUID="22ea3ab0-6acc-4558-9158-eaa289924f35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused"
Nov 03 20:26:26.939990 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:26.939154    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh" podUID="2d3647f2-89e9-4908-a8ee-d84e527abeb0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:26:26.939990 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:26.939679    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/premium-w4-e80203-kserve-5f9f445456-bms9d" podUID="6ad76efc-20c2-4951-b22a-c5a39cd39ed4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:26:26.953292 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:26.950561    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/premium-w2-9973d9-kserve-5c6dd4669c-5gv8x" podUID="cefc89ad-7ae1-433b-979d-6601678a6d98" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:26:26.977184 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:26.968143    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz" podUID="3a715ef1-38e9-4386-9eec-e10e8df11a10" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:26:26.984613 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:26.980146    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-2-w6-9e1bcb-kserve-7d96d9f6f6-4f7fl" podUID="56353b0f-08ee-4ada-9dc0-ac5afd5a59c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:26:27.028817 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:27.026477    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/distinct-w3-482722-kserve-d876967c-kd488" podUID="9a97f9cc-2261-471d-9053-51ee965850bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded"
Nov 03 20:26:27.154333 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:27.143226    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/distinct-2-w3-482722-kserve-74bf58cf7d-v9cjw" podUID="9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded"
Nov 03 20:26:27.154522 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:27.154400    5570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-2-w6-9e1bcb-kserve-7d96d9f6f6-4f7fl" podUID="56353b0f-08ee-4ada-9dc0-ac5afd5a59c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:26:27.415877 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:27.401003    5570 patch_prober.go:29] interesting pod/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-rbvqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body=
Nov 03 20:26:27.415877 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:27.401074    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-rbvqw" podUID="ae5a764e-25f7-492e-b4aa-8f2696e22f37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused"
Nov 03 20:26:27.526277 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:27.526211    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/premium-w3-482722-kserve-689dc7bd5d-x8rjp" podUID="369045c3-4d6a-49a7-9bc6-f2939883bee8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Nov 03 20:26:27.527205 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:27.527176    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/premium-w1-6fc748-kserve-7474db9cd7-nf2xk" podUID="830e062e-83fc-4de7-91a7-5534dac1d46b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Nov 03 20:26:27.594922 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:27.592438    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e036ad7-5g2sj"]
Nov 03 20:26:27.595277 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:27.595255    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-g262x"
Nov 03 20:26:27.595455 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:27.593253    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/distinct-2-w0-e40816-kserve-79fb46654d-s242x" podUID="4ad9b048-925c-4d01-a839-a5f25d4dd02d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:26:27.730154 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:27.728210    5570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-9e1bcb/premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh" podUID="2d3647f2-89e9-4908-a8ee-d84e527abeb0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": context deadline exceeded"
Nov 03 20:26:27.809283 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:26:27.809236    5570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8bwhz" containerName="extractor"
Nov 03 20:26:27.835597 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:27.812549    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e40816/simulator-w0-e40816-kserve-688f79796f-2tvw5"
Nov 03 20:26:27.835799 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:27.835716    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zdsqp"
Nov 03 20:26:27.835872 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:27.835819    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6fc748/premium-w1-6fc748-kserve-7474db9cd7-nf2xk"
Nov 03 20:26:27.835934 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:27.835883    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-482722/distinct-2-w3-482722-kserve-74bf58cf7d-v9cjw"
Nov 03 20:26:27.836030 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:27.835937    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-482722/distinct-w3-482722-kserve-d876967c-kd488"
Nov 03 20:26:27.836030 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:27.835989    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e40816/distinct-2-w0-e40816-kserve-79fb46654d-s242x"
Nov 03 20:26:27.836147 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:27.836042    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-e80203/distinct-2-w4-e80203-kserve-756f9b9bc6-69gxj"
Nov 03 20:26:27.836147 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:27.836067    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e036ad7-5g2sj"]
Nov 03 20:26:27.948704 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:27.947045    5570 patch_prober.go:29] interesting pod/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-j7dz6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body=
Nov 03 20:26:27.948704 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:27.947121    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-j7dz6" podUID="22ea3ab0-6acc-4558-9158-eaa289924f35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused"
Nov 03 20:26:28.010935 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:27.998129    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-2-w6-9e1bcb-kserve-7d96d9f6f6-4f7fl" podUID="56353b0f-08ee-4ada-9dc0-ac5afd5a59c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:26:28.010935 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:28.003752    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-482722/premium-w3-482722-kserve-689dc7bd5d-x8rjp"
Nov 03 20:26:28.010935 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:28.004215    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz" podUID="3a715ef1-38e9-4386-9eec-e10e8df11a10" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:26:28.032517 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:28.032103    5570 scope.go:122] "RemoveContainer" containerID="3417a92520cda35689055bd3a5c9c034576455ff70c4a138c5f6fd0f3141d5f7"
Nov 03 20:26:28.043180 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:28.042845    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf457154-f1db-4bf4-9f65-3604c92f6032" path="/var/lib/kubelet/pods/bf457154-f1db-4bf4-9f65-3604c92f6032/volumes"
Nov 03 20:26:28.043454 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:28.043408    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dc7b94-667b5756b8-ns2z5" event={"ID":"2635dffb-14f1-49cb-8d32-8f539c28ed30","Type":"ContainerStarted","Data":"e2d6437514e7ae9649085f2ab58a2c8c15874dadd45dda93d9e35c8a43ca8260"}
Nov 03 20:26:28.044255 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:28.044235    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-gd5bg"
Nov 03 20:26:28.047418 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:28.047340    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-e80203/premium-w4-e80203-kserve-5f9f445456-bms9d"
Nov 03 20:26:28.047418 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:28.047395    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-g262x"
Nov 03 20:26:28.047418 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:28.047422    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-j7dz6"
Nov 03 20:26:28.052115 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:28.047463    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-9e1bcb/premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh"
Nov 03 20:26:28.052115 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:28.047493    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-g262x"
Nov 03 20:26:28.052115 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:28.047528    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9973d9/premium-w2-9973d9-kserve-5c6dd4669c-5gv8x"
Nov 03 20:26:28.052115 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:28.047558    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-2-w6-9e1bcb-kserve-7d96d9f6f6-4f7fl"
Nov 03 20:26:28.052115 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:28.047634    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz"
Nov 03 20:26:28.052115 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:28.047703    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dc7b94-667b5756b8-ns2z5"
Nov 03 20:26:28.052115 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:28.047791    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zdsqp"
Nov 03 20:26:28.053611 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:28.053274    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-gd5bg"
Nov 03 20:26:28.080648 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:28.080611    5570 scope.go:122] "RemoveContainer" containerID="3c87f84b45f8993dd0fa9408f5defa5c488f5624343541f2832e37a68a4a4fbe"
Nov 03 20:26:28.385227 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:28.385139    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-rbvqw"
Nov 03 20:26:28.386342 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:28.386314    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-rbvqw"
Nov 03 20:26:28.551352 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:28.551281    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dc7b94-667b5756b8-ns2z5" podStartSLOduration=10.609631601 podStartE2EDuration="15.551259609s" totalImagesPullingTime="4.941628008s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:26:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:26:28.542444767 +0000 UTC m=+1322.695106656" watchObservedRunningTime="2026-11-03 20:26:28.551259609 +0000 UTC m=+1322.703921500"
Nov 03 20:26:28.841673 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:28.841637    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-j7dz6"
Nov 03 20:26:29.080814 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:29.075465    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-j7dz6"
Nov 03 20:26:30.423947 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:30.423914    5570 patch_prober.go:29] interesting pod/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-jshb8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": read tcp 10.134.0.2:39728->10.134.0.108:15021: read: connection reset by peer" start-of-body=
Nov 03 20:26:30.424389 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:30.423980    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-jshb8" podUID="2137488e-8e6c-46f7-a0ed-b4721d4f2645" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": read tcp 10.134.0.2:39728->10.134.0.108:15021: read: connection reset by peer"
Nov 03 20:26:30.424389 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:30.424107    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-jshb8"
Nov 03 20:26:30.424732 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:30.424710    5570 patch_prober.go:29] interesting pod/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-jshb8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body=
Nov 03 20:26:30.424833 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:30.424764    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-jshb8" podUID="2137488e-8e6c-46f7-a0ed-b4721d4f2645" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused"
Nov 03 20:26:31.089869 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:31.089831    5570 generic.go:356] "Generic (PLEG): container finished" podID="2137488e-8e6c-46f7-a0ed-b4721d4f2645" containerID="117061888c8f2f87481828900cffe3e409615fd7d0abfd26850b40bf839788ee" exitCode=0
Nov 03 20:26:31.090084 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:31.089905    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-jshb8" event={"ID":"2137488e-8e6c-46f7-a0ed-b4721d4f2645","Type":"ContainerDied","Data":"117061888c8f2f87481828900cffe3e409615fd7d0abfd26850b40bf839788ee"}
Nov 03 20:26:31.090484 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:31.090466    5570 scope.go:122] "RemoveContainer" containerID="117061888c8f2f87481828900cffe3e409615fd7d0abfd26850b40bf839788ee"
Nov 03 20:26:31.091805 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:31.091784    5570 generic.go:356] "Generic (PLEG): container finished" podID="7b57a22a-cf28-4099-8883-c04c61807dfa" containerID="e0359761c7f58587a7e9e0c308a42f2df94efc0661527121e1458f5db9608599" exitCode=0
Nov 03 20:26:31.091920 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:31.091832    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-nsjcd" event={"ID":"7b57a22a-cf28-4099-8883-c04c61807dfa","Type":"ContainerDied","Data":"e0359761c7f58587a7e9e0c308a42f2df94efc0661527121e1458f5db9608599"}
Nov 03 20:26:31.092322 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:31.092304    5570 scope.go:122] "RemoveContainer" containerID="e0359761c7f58587a7e9e0c308a42f2df94efc0661527121e1458f5db9608599"
Nov 03 20:26:31.092748 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:31.092716    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:26:31.092850 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:31.092831    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:26:31.092903 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:31.092881    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:26:31.094127 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:31.094104    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:26:31.094196 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:31.094168    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:26:31.094232 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:31.094200    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:26:32.108847 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:32.108763    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-nsjcd" event={"ID":"7b57a22a-cf28-4099-8883-c04c61807dfa","Type":"ContainerStarted","Data":"77b59920db5fd3798fd079759ead749d52749bd7e92d88348b87940208da84d0"}
Nov 03 20:26:32.111054 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:32.111029    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-jshb8" event={"ID":"2137488e-8e6c-46f7-a0ed-b4721d4f2645","Type":"ContainerStarted","Data":"e5c7ec1d14a857d58b25629abd68ececf4e69dc30db9346800e1c54320f24671"}
Nov 03 20:26:32.382860 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:32.382254    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-jshb8"
Nov 03 20:26:32.382860 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:32.382408    5570 patch_prober.go:29] interesting pod/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-jshb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body=
Nov 03 20:26:32.382860 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:32.382467    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-jshb8" podUID="2137488e-8e6c-46f7-a0ed-b4721d4f2645" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused"
Nov 03 20:26:32.616149 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:32.616115    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-nsjcd"
Nov 03 20:26:32.616715 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:32.616690    5570 patch_prober.go:29] interesting pod/e2e-mt-eb87f429-openshift-default-64649dfdb5-nsjcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body=
Nov 03 20:26:32.616943 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:32.616921    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-nsjcd" podUID="7b57a22a-cf28-4099-8883-c04c61807dfa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused"
Nov 03 20:26:33.385801 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:33.382367    5570 patch_prober.go:29] interesting pod/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-jshb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body=
Nov 03 20:26:33.385801 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:33.382456    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-jshb8" podUID="2137488e-8e6c-46f7-a0ed-b4721d4f2645" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused"
Nov 03 20:26:33.616285 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:33.616246    5570 patch_prober.go:29] interesting pod/e2e-mt-eb87f429-openshift-default-64649dfdb5-nsjcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body=
Nov 03 20:26:33.616585 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:33.616552    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-nsjcd" podUID="7b57a22a-cf28-4099-8883-c04c61807dfa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused"
Nov 03 20:26:34.077804 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:34.076910    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-852e3ec1-6vq7t"]
Nov 03 20:26:34.131504 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:34.129996    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-852e3ec1-6vq7t"]
Nov 03 20:26:34.182800 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:34.172857    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/premium-w1-6fc748-kserve-7474db9cd7-nf2xk" podUID="830e062e-83fc-4de7-91a7-5534dac1d46b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:26:34.378869 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:34.374244    5570 patch_prober.go:29] interesting pod/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-jshb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body=
Nov 03 20:26:34.378869 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:34.374329    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-jshb8" podUID="2137488e-8e6c-46f7-a0ed-b4721d4f2645" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused"
Nov 03 20:26:34.615530 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:34.615489    5570 patch_prober.go:29] interesting pod/e2e-mt-eb87f429-openshift-default-64649dfdb5-nsjcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body=
Nov 03 20:26:34.615948 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:34.615550    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-nsjcd" podUID="7b57a22a-cf28-4099-8883-c04c61807dfa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused"
Nov 03 20:26:35.094102 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:35.094066    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dc7b94-667b5756b8-ns2z5"
Nov 03 20:26:35.476861 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:35.427921    5570 patch_prober.go:29] interesting pod/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-jshb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body=
Nov 03 20:26:35.476861 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:35.427982    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-jshb8" podUID="2137488e-8e6c-46f7-a0ed-b4721d4f2645" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused"
Nov 03 20:26:35.618421 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:35.617313    5570 patch_prober.go:29] interesting pod/e2e-mt-eb87f429-openshift-default-64649dfdb5-nsjcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body=
Nov 03 20:26:35.618421 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:35.617384    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-nsjcd" podUID="7b57a22a-cf28-4099-8883-c04c61807dfa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused"
Nov 03 20:26:36.129926 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:36.129888    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10788aba-4e6e-44dd-b3c0-eacc930678f9" path="/var/lib/kubelet/pods/10788aba-4e6e-44dd-b3c0-eacc930678f9/volumes"
Nov 03 20:26:36.386854 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:36.386177    5570 patch_prober.go:29] interesting pod/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-jshb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body=
Nov 03 20:26:36.386854 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:36.386241    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-jshb8" podUID="2137488e-8e6c-46f7-a0ed-b4721d4f2645" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused"
Nov 03 20:26:36.429898 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:36.424231    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g69f8"]
Nov 03 20:26:36.618934 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:36.618729    5570 patch_prober.go:29] interesting pod/e2e-mt-eb87f429-openshift-default-64649dfdb5-nsjcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body=
Nov 03 20:26:36.619560 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:36.619518    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-nsjcd" podUID="7b57a22a-cf28-4099-8883-c04c61807dfa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused"
Nov 03 20:26:37.384801 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:37.383806    5570 patch_prober.go:29] interesting pod/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-jshb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body=
Nov 03 20:26:37.384801 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:37.383897    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-jshb8" podUID="2137488e-8e6c-46f7-a0ed-b4721d4f2645" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused"
Nov 03 20:26:37.560036 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:37.559990    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/premium-w2-9973d9-kserve-5c6dd4669c-5gv8x" podUID="cefc89ad-7ae1-433b-979d-6601678a6d98" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Nov 03 20:26:37.646854 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:37.624260    5570 patch_prober.go:29] interesting pod/e2e-mt-eb87f429-openshift-default-64649dfdb5-nsjcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body=
Nov 03 20:26:37.646854 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:37.624375    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-nsjcd" podUID="7b57a22a-cf28-4099-8883-c04c61807dfa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused"
Nov 03 20:26:38.024798 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:38.020099    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-2-w6-9e1bcb-kserve-7d96d9f6f6-4f7fl" podUID="56353b0f-08ee-4ada-9dc0-ac5afd5a59c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:26:38.181802 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:38.170447    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/distinct-2-w0-e40816-kserve-79fb46654d-s242x" podUID="4ad9b048-925c-4d01-a839-a5f25d4dd02d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:26:38.363913 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:38.363807    5570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-e40816/simulator-w0-e40816-kserve-688f79796f-2tvw5" podUID="c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:26:38.378384 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:38.378352    5570 patch_prober.go:29] interesting pod/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-jshb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body=
Nov 03 20:26:38.378668 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:38.378630    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-jshb8" podUID="2137488e-8e6c-46f7-a0ed-b4721d4f2645" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused"
Nov 03 20:26:38.508999 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:38.508941    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/simulator-w0-e40816-kserve-688f79796f-2tvw5" podUID="c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:26:38.567956 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:38.567896    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/premium-w2-9973d9-kserve-5c6dd4669c-5gv8x" podUID="cefc89ad-7ae1-433b-979d-6601678a6d98" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Nov 03 20:26:38.568867 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:38.568840    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9973d9/premium-w2-9973d9-kserve-5c6dd4669c-5gv8x"
Nov 03 20:26:38.648858 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:38.647868    5570 patch_prober.go:29] interesting pod/e2e-mt-eb87f429-openshift-default-64649dfdb5-nsjcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body=
Nov 03 20:26:38.648858 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:38.647940    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-nsjcd" podUID="7b57a22a-cf28-4099-8883-c04c61807dfa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused"
Nov 03 20:26:38.845452 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:38.845406    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh" podUID="2d3647f2-89e9-4908-a8ee-d84e527abeb0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Nov 03 20:26:39.077413 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:38.916511    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/premium-w3-482722-kserve-689dc7bd5d-x8rjp" podUID="369045c3-4d6a-49a7-9bc6-f2939883bee8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:26:39.100800 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:39.060992    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/distinct-w3-482722-kserve-d876967c-kd488" podUID="9a97f9cc-2261-471d-9053-51ee965850bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded"
Nov 03 20:26:39.122201 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:39.061029    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/premium-w4-e80203-kserve-5f9f445456-bms9d" podUID="6ad76efc-20c2-4951-b22a-c5a39cd39ed4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Nov 03 20:26:39.144854 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:39.130827    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-2-w6-9e1bcb-kserve-7d96d9f6f6-4f7fl" podUID="56353b0f-08ee-4ada-9dc0-ac5afd5a59c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:26:39.174988 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:39.174943    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/distinct-2-w0-e40816-kserve-79fb46654d-s242x" podUID="4ad9b048-925c-4d01-a839-a5f25d4dd02d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded"
Nov 03 20:26:39.198346 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:39.130856    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/distinct-2-w4-e80203-kserve-756f9b9bc6-69gxj" podUID="ccf5dc2f-f116-4917-bb1a-b136d8ed17f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:26:39.198346 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:39.197107    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-2-w6-9e1bcb-kserve-7d96d9f6f6-4f7fl"
Nov 03 20:26:39.198346 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:39.197172    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e40816/distinct-2-w0-e40816-kserve-79fb46654d-s242x"
Nov 03 20:26:39.202261 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:39.200212    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/premium-w1-6fc748-kserve-7474db9cd7-nf2xk" podUID="830e062e-83fc-4de7-91a7-5534dac1d46b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:26:39.633450 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:39.633401    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/distinct-2-w3-482722-kserve-74bf58cf7d-v9cjw" podUID="9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:26:40.120801 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:39.686986    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/simulator-w0-e40816-kserve-688f79796f-2tvw5" podUID="c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Nov 03 20:26:40.120801 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:39.703707    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/premium-w2-9973d9-kserve-5c6dd4669c-5gv8x" podUID="cefc89ad-7ae1-433b-979d-6601678a6d98" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Nov 03 20:26:40.213392 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:40.213343    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh" podUID="2d3647f2-89e9-4908-a8ee-d84e527abeb0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Nov 03 20:26:40.214283 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:40.214248    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/premium-w3-482722-kserve-689dc7bd5d-x8rjp" podUID="369045c3-4d6a-49a7-9bc6-f2939883bee8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Nov 03 20:26:40.214979 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:39.721012    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-baa8c991-tz94q"]
Nov 03 20:26:40.215939 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:40.215217    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e40816/simulator-w0-e40816-kserve-688f79796f-2tvw5"
Nov 03 20:26:40.215939 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:40.215292    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-2-w6-9e1bcb-kserve-7d96d9f6f6-4f7fl"
Nov 03 20:26:40.215939 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:40.215351    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-9e1bcb/premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh"
Nov 03 20:26:40.215939 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:40.215387    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-baa8c991-tz94q"]
Nov 03 20:26:40.215939 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:40.215464    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-482722/premium-w3-482722-kserve-689dc7bd5d-x8rjp"
Nov 03 20:26:40.254799 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:40.245203    5570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-e80203/distinct-2-w4-e80203-kserve-756f9b9bc6-69gxj" podUID="ccf5dc2f-f116-4917-bb1a-b136d8ed17f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:26:40.254799 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:39.704440    5570 patch_prober.go:29] interesting pod/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-jshb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body=
Nov 03 20:26:40.254799 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:40.245333    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-jshb8" podUID="2137488e-8e6c-46f7-a0ed-b4721d4f2645" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused"
Nov 03 20:26:40.254799 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:39.720039    5570 patch_prober.go:29] interesting pod/e2e-mt-eb87f429-openshift-default-64649dfdb5-nsjcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body=
Nov 03 20:26:40.254799 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:40.245385    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-nsjcd" podUID="7b57a22a-cf28-4099-8883-c04c61807dfa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused"
Nov 03 20:26:40.254799 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:40.248492    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/distinct-w3-482722-kserve-d876967c-kd488" podUID="9a97f9cc-2261-471d-9053-51ee965850bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded"
Nov 03 20:26:40.254799 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:40.249109    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/premium-w4-e80203-kserve-5f9f445456-bms9d" podUID="6ad76efc-20c2-4951-b22a-c5a39cd39ed4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:26:40.254799 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:40.250481    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-482722/distinct-w3-482722-kserve-d876967c-kd488"
Nov 03 20:26:40.254799 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:40.250576    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-e80203/premium-w4-e80203-kserve-5f9f445456-bms9d"
Nov 03 20:26:40.332643 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:40.330956    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/distinct-2-w0-e40816-kserve-79fb46654d-s242x" podUID="4ad9b048-925c-4d01-a839-a5f25d4dd02d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:26:40.332643 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:40.331350    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/distinct-2-w4-e80203-kserve-756f9b9bc6-69gxj" podUID="ccf5dc2f-f116-4917-bb1a-b136d8ed17f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:26:40.393863 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:40.372980    5570 patch_prober.go:29] interesting pod/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-jshb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body=
Nov 03 20:26:40.393863 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:40.373058    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-jshb8" podUID="2137488e-8e6c-46f7-a0ed-b4721d4f2645" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused"
Nov 03 20:26:40.730801 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:40.700389    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87s5mf" podUID="b7306cb9-eeae-447b-b834-870c8d49a5aa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": context deadline exceeded"
Nov 03 20:26:40.730801 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:40.701440    5570 patch_prober.go:29] interesting pod/e2e-mt-eb87f429-openshift-default-64649dfdb5-nsjcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body=
Nov 03 20:26:40.730801 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:40.701493    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-nsjcd" podUID="7b57a22a-cf28-4099-8883-c04c61807dfa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused"
Nov 03 20:26:40.867427 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:40.867384    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f41cff1-9c8c-48a8-a719-6bb52f8095a7" path="/var/lib/kubelet/pods/8f41cff1-9c8c-48a8-a719-6bb52f8095a7/volumes"
Nov 03 20:26:40.868964 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:40.868925    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-nsjcd"
Nov 03 20:26:40.869261 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:40.869244    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-9e1bcb/premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh"
Nov 03 20:26:40.869421 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:40.869408    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-e80203/distinct-2-w4-e80203-kserve-756f9b9bc6-69gxj"
Nov 03 20:26:40.869546 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:40.869535    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9973d9/premium-w2-9973d9-kserve-5c6dd4669c-5gv8x"
Nov 03 20:26:40.869666 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:40.869656    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e40816/simulator-w0-e40816-kserve-688f79796f-2tvw5"
Nov 03 20:26:40.870005 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:40.869989    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-e80203/premium-w4-e80203-kserve-5f9f445456-bms9d"
Nov 03 20:26:40.870161 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:40.870149    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-482722/premium-w3-482722-kserve-689dc7bd5d-x8rjp"
Nov 03 20:26:40.870277 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:40.870266    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-482722/distinct-w3-482722-kserve-d876967c-kd488"
Nov 03 20:26:40.870386 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:40.870375    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e40816/distinct-2-w0-e40816-kserve-79fb46654d-s242x"
Nov 03 20:26:40.906796 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:40.903394    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-e80203/distinct-2-w4-e80203-kserve-756f9b9bc6-69gxj"
Nov 03 20:26:41.405697 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:41.405669    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-jshb8"
Nov 03 20:26:41.408482 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:41.408457    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-jshb8"
Nov 03 20:26:41.409984 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:41.409939    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-jshb8"
Nov 03 20:26:41.913645 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:41.913618    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-nsjcd"
Nov 03 20:26:41.917503 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:41.917441    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-nsjcd"
Nov 03 20:26:46.308463 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:46.308419    5570 patch_prober.go:29] interesting pod/e2e-shared-a-w3-76b894-openshift-default-6669d4d94f-vrq8z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:26:46.308935 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:46.308507    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-76b894-openshift-default-6669d4d94f-vrq8z" podUID="3db8ac79-76fc-4d46-ac4f-6536fe192b8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:26:56.034818 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:56.034746    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aafe6908-shqjq"]
Nov 03 20:26:56.557267 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:56.048206    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aafe6908-shqjq"]
Nov 03 20:26:57.967263 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:26:57.967224    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1aec61be-2b12-40cb-ada3-88c05f790cfb" path="/var/lib/kubelet/pods/1aec61be-2b12-40cb-ada3-88c05f790cfb/volumes"
Nov 03 20:26:58.559945 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:26:58.559899    5570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8bwhz" containerName="extractor"
Nov 03 20:27:01.562277 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:01.562219    5570 patch_prober.go:29] interesting pod/e2e-shared-a-w3-76b894-openshift-default-6669d4d94f-vrq8z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:27:01.640015 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:01.639960    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-76b894-openshift-default-6669d4d94f-vrq8z" podUID="3db8ac79-76fc-4d46-ac4f-6536fe192b8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:27:05.230243 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:05.230203    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4cdbeb-7c468dc747-9kqqx"]
Nov 03 20:27:05.384495 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:05.384453    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-4cdbeb-77575c99cb-7pz9r"]
Nov 03 20:27:05.404800 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:05.404738    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4cdbeb-7c468dc747-9kqqx"
Nov 03 20:27:05.487409 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:05.487314    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrhgz\" (UniqueName: \"kubernetes.io/projected/880ebe10-49c7-4960-9fb2-2c699047ebb6-kube-api-access-rrhgz\") pod \"payload-pre-processing-e2e-api-b-4cdbeb-7c468dc747-9kqqx\" (UID: \"880ebe10-49c7-4960-9fb2-2c699047ebb6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4cdbeb-7c468dc747-9kqqx"
Nov 03 20:27:05.532166 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:05.490875    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/880ebe10-49c7-4960-9fb2-2c699047ebb6-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-4cdbeb-7c468dc747-9kqqx\" (UID: \"880ebe10-49c7-4960-9fb2-2c699047ebb6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4cdbeb-7c468dc747-9kqqx"
Nov 03 20:27:05.551351 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:05.551315    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-4cdbeb-77575c99cb-7pz9r"
Nov 03 20:27:05.602818 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:05.593298    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrhgz\" (UniqueName: \"kubernetes.io/projected/880ebe10-49c7-4960-9fb2-2c699047ebb6-kube-api-access-rrhgz\") pod \"payload-pre-processing-e2e-api-b-4cdbeb-7c468dc747-9kqqx\" (UID: \"880ebe10-49c7-4960-9fb2-2c699047ebb6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4cdbeb-7c468dc747-9kqqx"
Nov 03 20:27:05.602818 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:05.593387    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3716281c-9234-41d2-96c3-dbc38c692bbb-plugins-config-volume\") pod \"payload-processing-e2e-api-b-4cdbeb-77575c99cb-7pz9r\" (UID: \"3716281c-9234-41d2-96c3-dbc38c692bbb\") " pod="openshift-ingress/payload-processing-e2e-api-b-4cdbeb-77575c99cb-7pz9r"
Nov 03 20:27:05.602818 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:05.593482    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/880ebe10-49c7-4960-9fb2-2c699047ebb6-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-4cdbeb-7c468dc747-9kqqx\" (UID: \"880ebe10-49c7-4960-9fb2-2c699047ebb6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4cdbeb-7c468dc747-9kqqx"
Nov 03 20:27:05.602818 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:05.593802    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwb9b\" (UniqueName: \"kubernetes.io/projected/3716281c-9234-41d2-96c3-dbc38c692bbb-kube-api-access-kwb9b\") pod \"payload-processing-e2e-api-b-4cdbeb-77575c99cb-7pz9r\" (UID: \"3716281c-9234-41d2-96c3-dbc38c692bbb\") " pod="openshift-ingress/payload-processing-e2e-api-b-4cdbeb-77575c99cb-7pz9r"
Nov 03 20:27:05.676951 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:05.673095    5570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-4cdbeb\""
Nov 03 20:27:05.676951 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:05.674731    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/880ebe10-49c7-4960-9fb2-2c699047ebb6-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-4cdbeb-7c468dc747-9kqqx\" (UID: \"880ebe10-49c7-4960-9fb2-2c699047ebb6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4cdbeb-7c468dc747-9kqqx"
Nov 03 20:27:05.699876 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:05.699348    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3716281c-9234-41d2-96c3-dbc38c692bbb-plugins-config-volume\") pod \"payload-processing-e2e-api-b-4cdbeb-77575c99cb-7pz9r\" (UID: \"3716281c-9234-41d2-96c3-dbc38c692bbb\") " pod="openshift-ingress/payload-processing-e2e-api-b-4cdbeb-77575c99cb-7pz9r"
Nov 03 20:27:05.699876 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:05.699741    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwb9b\" (UniqueName: \"kubernetes.io/projected/3716281c-9234-41d2-96c3-dbc38c692bbb-kube-api-access-kwb9b\") pod \"payload-processing-e2e-api-b-4cdbeb-77575c99cb-7pz9r\" (UID: \"3716281c-9234-41d2-96c3-dbc38c692bbb\") " pod="openshift-ingress/payload-processing-e2e-api-b-4cdbeb-77575c99cb-7pz9r"
Nov 03 20:27:05.742542 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:05.701278    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3716281c-9234-41d2-96c3-dbc38c692bbb-plugins-config-volume\") pod \"payload-processing-e2e-api-b-4cdbeb-77575c99cb-7pz9r\" (UID: \"3716281c-9234-41d2-96c3-dbc38c692bbb\") " pod="openshift-ingress/payload-processing-e2e-api-b-4cdbeb-77575c99cb-7pz9r"
Nov 03 20:27:05.744653 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:05.744619    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-4cdbeb-dockercfg-tn2vz\""
Nov 03 20:27:06.084004 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:06.083915    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwb9b\" (UniqueName: \"kubernetes.io/projected/3716281c-9234-41d2-96c3-dbc38c692bbb-kube-api-access-kwb9b\") pod \"payload-processing-e2e-api-b-4cdbeb-77575c99cb-7pz9r\" (UID: \"3716281c-9234-41d2-96c3-dbc38c692bbb\") " pod="openshift-ingress/payload-processing-e2e-api-b-4cdbeb-77575c99cb-7pz9r"
Nov 03 20:27:06.084004 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:06.083922    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrhgz\" (UniqueName: \"kubernetes.io/projected/880ebe10-49c7-4960-9fb2-2c699047ebb6-kube-api-access-rrhgz\") pod \"payload-pre-processing-e2e-api-b-4cdbeb-7c468dc747-9kqqx\" (UID: \"880ebe10-49c7-4960-9fb2-2c699047ebb6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4cdbeb-7c468dc747-9kqqx"
Nov 03 20:27:06.135599 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:06.135545    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4cdbeb-7c468dc747-9kqqx"]
Nov 03 20:27:06.199151 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:06.199121    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-4cdbeb-77575c99cb-7pz9r"]
Nov 03 20:27:06.221699 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:06.220569    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4cdbeb-7c468dc747-9kqqx"
Nov 03 20:27:06.272971 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:06.272923    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-4cdbeb-77575c99cb-7pz9r"
Nov 03 20:27:09.276311 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:09.276269    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-4cdbeb-77575c99cb-7pz9r"]
Nov 03 20:27:09.284971 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:09.284945    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4cdbeb-7c468dc747-9kqqx"]
Nov 03 20:27:09.287532 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:27:09.287368    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3716281c_9234_41d2_96c3_dbc38c692bbb.slice/crio-d992d82915b787a6337392b782cf3a127f25831eac12d01d25cfb43a6bd0ac16 WatchSource:0}: Error finding container d992d82915b787a6337392b782cf3a127f25831eac12d01d25cfb43a6bd0ac16: Status 404 returned error can't find the container with id d992d82915b787a6337392b782cf3a127f25831eac12d01d25cfb43a6bd0ac16
Nov 03 20:27:09.297151 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:09.297128    5570 generic.go:356] "Generic (PLEG): container finished" podID="3db8ac79-76fc-4d46-ac4f-6536fe192b8f" containerID="e71f0ebc0297767f2edd7a26e6152c5a29917591acf45764b21d98a005b7a272" exitCode=0
Nov 03 20:27:09.297225 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:09.297204    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-76b894-openshift-default-6669d4d94f-vrq8z" event={"ID":"3db8ac79-76fc-4d46-ac4f-6536fe192b8f","Type":"ContainerDied","Data":"e71f0ebc0297767f2edd7a26e6152c5a29917591acf45764b21d98a005b7a272"}
Nov 03 20:27:09.297769 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:09.297754    5570 scope.go:122] "RemoveContainer" containerID="e71f0ebc0297767f2edd7a26e6152c5a29917591acf45764b21d98a005b7a272"
Nov 03 20:27:09.309104 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:09.309049    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:27:09.309710 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:09.309661    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:27:09.309975 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:09.309945    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:27:10.408581 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:10.408543    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-76b894-openshift-default-6669d4d94f-vrq8z" event={"ID":"3db8ac79-76fc-4d46-ac4f-6536fe192b8f","Type":"ContainerStarted","Data":"08e6c055f3b06fb5e23c8d8740789d057e79fe1612f7eeefc46ba3e8200254e9"}
Nov 03 20:27:10.435659 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:10.435516    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-4cdbeb-77575c99cb-7pz9r" event={"ID":"3716281c-9234-41d2-96c3-dbc38c692bbb","Type":"ContainerStarted","Data":"4be17df6901c7c6d2804455f4b35dab727aa9a50b40068440dca691f0ba75af3"}
Nov 03 20:27:10.435659 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:10.435595    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-4cdbeb-77575c99cb-7pz9r"
Nov 03 20:27:10.435659 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:10.435621    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-4cdbeb-77575c99cb-7pz9r" event={"ID":"3716281c-9234-41d2-96c3-dbc38c692bbb","Type":"ContainerStarted","Data":"d992d82915b787a6337392b782cf3a127f25831eac12d01d25cfb43a6bd0ac16"}
Nov 03 20:27:10.532622 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:10.528919    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4cdbeb-7c468dc747-9kqqx" event={"ID":"880ebe10-49c7-4960-9fb2-2c699047ebb6","Type":"ContainerStarted","Data":"5f27f224d717a2b67dd1594ec6e976ea0589888f16a8aa5e5a94c06420e38d46"}
Nov 03 20:27:10.532622 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:10.528994    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4cdbeb-7c468dc747-9kqqx" event={"ID":"880ebe10-49c7-4960-9fb2-2c699047ebb6","Type":"ContainerStarted","Data":"f5547614565e9560552ed9bd9f9655ff568060ec86e37f003aa1073fe240508c"}
Nov 03 20:27:10.532622 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:10.530333    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4cdbeb-7c468dc747-9kqqx"
Nov 03 20:27:10.684424 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:10.684331    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-76b894-openshift-default-6669d4d94f-vrq8z"
Nov 03 20:27:10.695966 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:10.695928    5570 patch_prober.go:29] interesting pod/e2e-shared-a-w3-76b894-openshift-default-6669d4d94f-vrq8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body=
Nov 03 20:27:10.696255 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:10.696224    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-76b894-openshift-default-6669d4d94f-vrq8z" podUID="3db8ac79-76fc-4d46-ac4f-6536fe192b8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused"
Nov 03 20:27:10.697349 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:10.697292    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-4cdbeb-77575c99cb-7pz9r" podStartSLOduration=5.697267517 podStartE2EDuration="5.697267517s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:27:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:27:10.590365239 +0000 UTC m=+1364.743027125" watchObservedRunningTime="2026-11-03 20:27:10.697267517 +0000 UTC m=+1364.849929404"
Nov 03 20:27:10.707913 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:10.706574    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4cdbeb-7c468dc747-9kqqx" podStartSLOduration=5.706550744 podStartE2EDuration="5.706550744s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:27:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:27:10.659944883 +0000 UTC m=+1364.812606768" watchObservedRunningTime="2026-11-03 20:27:10.706550744 +0000 UTC m=+1364.859212632"
Nov 03 20:27:11.671799 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:11.671243    5570 patch_prober.go:29] interesting pod/e2e-shared-a-w3-76b894-openshift-default-6669d4d94f-vrq8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body=
Nov 03 20:27:11.671799 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:11.671313    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-76b894-openshift-default-6669d4d94f-vrq8z" podUID="3db8ac79-76fc-4d46-ac4f-6536fe192b8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused"
Nov 03 20:27:12.663740 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:12.663696    5570 patch_prober.go:29] interesting pod/e2e-shared-a-w3-76b894-openshift-default-6669d4d94f-vrq8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body=
Nov 03 20:27:12.664551 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:12.663761    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-76b894-openshift-default-6669d4d94f-vrq8z" podUID="3db8ac79-76fc-4d46-ac4f-6536fe192b8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused"
Nov 03 20:27:13.664403 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:13.664366    5570 patch_prober.go:29] interesting pod/e2e-shared-a-w3-76b894-openshift-default-6669d4d94f-vrq8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body=
Nov 03 20:27:13.664909 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:13.664423    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-76b894-openshift-default-6669d4d94f-vrq8z" podUID="3db8ac79-76fc-4d46-ac4f-6536fe192b8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused"
Nov 03 20:27:14.664445 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:14.664396    5570 patch_prober.go:29] interesting pod/e2e-shared-a-w3-76b894-openshift-default-6669d4d94f-vrq8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body=
Nov 03 20:27:14.665917 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:14.664467    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-76b894-openshift-default-6669d4d94f-vrq8z" podUID="3db8ac79-76fc-4d46-ac4f-6536fe192b8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused"
Nov 03 20:27:15.171059 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:15.171013    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-xcxnz"]
Nov 03 20:27:15.172231 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:15.171968    5570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-xcxnz" podUID="63727774-9118-4b83-bede-04208e8d59b9" containerName="istio-proxy" containerID="cri-o://c6574ca71b7f10f0ce3bcd5398a085b8f18c8a09ee7a532e8935c39af660977c" gracePeriod=30
Nov 03 20:27:15.181648 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:15.180882    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-mt7dd"]
Nov 03 20:27:15.181648 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:15.181119    5570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-mt7dd" podUID="55945440-d40e-4235-8b84-d1e2a013372d" containerName="istio-proxy" containerID="cri-o://3846399a93c8fddbeb80930fda59b34d3625c7cf8f08fb90f62591110fad7bec" gracePeriod=30
Nov 03 20:27:15.185676 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:15.185436    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-wv4cn"]
Nov 03 20:27:15.185676 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:15.185585    5570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-wv4cn" podUID="c1e2eadf-ede0-4ea2-9eea-f25a39779795" containerName="istio-proxy" containerID="cri-o://5c6d90f6dff94d54503259d98624356b90f1622e4c025457fbf45267a1efac96" gracePeriod=30
Nov 03 20:27:15.297650 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:15.297616    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-76b894-openshift-default-6669d4d94f-vrq8z"
Nov 03 20:27:15.604755 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:15.604713    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-wg864"]
Nov 03 20:27:15.605013 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:15.604987    5570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-wg864" podUID="6a1b3d29-b647-46e1-9005-e540e937f85f" containerName="istio-proxy" containerID="cri-o://c112fdc1a8a3d21c59453de2ff24628f1b064fd91b46b2f5e53085450bd216c0" gracePeriod=30
Nov 03 20:27:15.610997 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:15.610963    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-9gvtf"]
Nov 03 20:27:15.611259 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:15.611229    5570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-9gvtf" podUID="5e7f17ab-0f1d-4a59-b415-fe844ae7c1e7" containerName="istio-proxy" containerID="cri-o://3e7c2142d5118a937b9c4c1db78402b22ba93d6da618fb6dea1c3717ae2ab38b" gracePeriod=30
Nov 03 20:27:15.618096 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:15.618062    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-jbtqd"]
Nov 03 20:27:15.618328 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:15.618293    5570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-jbtqd" podUID="baacee7f-b912-4964-9855-be2beb0094cb" containerName="istio-proxy" containerID="cri-o://f88e91ce1e2d9b540bdd3ec4f7167d68046dc7bc6e871d1e1f9efc08d87c6965" gracePeriod=30
Nov 03 20:27:15.636724 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:15.636459    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-wj4gp"]
Nov 03 20:27:15.648342 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:15.644913    5570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-wj4gp" podUID="ce7b1d98-c21f-4412-97ce-9dca59ce2649" containerName="istio-proxy" containerID="cri-o://2065ee88578f7048629b049802360f906227df9654abe61d45dbb67e25165dbb" gracePeriod=30
Nov 03 20:27:15.672387 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:15.672262    5570 patch_prober.go:29] interesting pod/e2e-shared-a-w3-76b894-openshift-default-6669d4d94f-vrq8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body=
Nov 03 20:27:15.672387 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:15.672324    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-76b894-openshift-default-6669d4d94f-vrq8z" podUID="3db8ac79-76fc-4d46-ac4f-6536fe192b8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused"
Nov 03 20:27:15.680362 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:15.680324    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-5bn7t"]
Nov 03 20:27:15.681241 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:15.680590    5570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-5bn7t" podUID="26831c71-2091-4090-843d-f7268d257114" containerName="istio-proxy" containerID="cri-o://50869a3fc874fdc6908769230eea560a48902e803a7a881735d87e7285089421" gracePeriod=30
Nov 03 20:27:15.687394 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:15.687363    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-5c82t"]
Nov 03 20:27:15.688680 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:15.687621    5570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-5c82t" podUID="6029825e-e953-4c43-9f4d-d19b5ca38623" containerName="istio-proxy" containerID="cri-o://61df14cbe40ac50825e634951cd7b5b5dbc121f9048dfffa32676dd32719d0b9" gracePeriod=30
Nov 03 20:27:15.692056 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:15.692029    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-xszg4"]
Nov 03 20:27:15.692308 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:15.692278    5570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-xszg4" podUID="4ee1819e-4649-4d3e-9cfc-0d2d72c1b481" containerName="istio-proxy" containerID="cri-o://c70dfe54e707de3f4706134c4a875053c59da1192e0ce8ce4a82e0c984572532" gracePeriod=30
Nov 03 20:27:15.696587 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:15.696554    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-fqjnh"]
Nov 03 20:27:15.696801 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:15.696760    5570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-fqjnh" podUID="51c1be09-aa2e-47bd-a863-97f7daf94ddb" containerName="istio-proxy" containerID="cri-o://8c8ff7ee0e824dae78ba0ae02863133dd5b8d0ddb46392cb8f4984679b96ed86" gracePeriod=30
Nov 03 20:27:16.666800 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:16.664964    5570 patch_prober.go:29] interesting pod/e2e-shared-a-w3-76b894-openshift-default-6669d4d94f-vrq8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body=
Nov 03 20:27:16.666800 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:16.665042    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-76b894-openshift-default-6669d4d94f-vrq8z" podUID="3db8ac79-76fc-4d46-ac4f-6536fe192b8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused"
Nov 03 20:27:17.671576 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:17.671543    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-76b894-openshift-default-6669d4d94f-vrq8z"
Nov 03 20:27:17.672685 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:17.672663    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-76b894-openshift-default-6669d4d94f-vrq8z"
Nov 03 20:27:17.791415 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:17.791202    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-wkmg8"]
Nov 03 20:27:17.792128 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:17.792093    5570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-wkmg8" podUID="e314286e-4b2d-4fae-bc59-528809a8dbce" containerName="istio-proxy" containerID="cri-o://39dce94b7e35773069c096fb3e6dee58c6fc5bf5e34a8cbdbb93fa453abea1cc" gracePeriod=30
Nov 03 20:27:17.800890 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:17.798577    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmkq7"]
Nov 03 20:27:17.800890 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:17.798827    5570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmkq7" podUID="157f80a0-a704-4bed-b544-205c3f2a52b3" containerName="istio-proxy" containerID="cri-o://73be03f9649446acd3f30a87e3b60fbb7889d073564dcf2859c9ed6c410dcaf5" gracePeriod=30
Nov 03 20:27:17.807205 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:17.807175    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-7458z"]
Nov 03 20:27:17.807395 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:17.807373    5570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-7458z" podUID="3c3d9f2d-dcbb-4fb3-a54f-ed9cdef4118e" containerName="istio-proxy" containerID="cri-o://5cfb10265f083a5dc0871a30cf040733f11efc2fb138b8ba9e9ecd8ad963fc77" gracePeriod=30
Nov 03 20:27:18.195673 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:18.195639    5570 patch_prober.go:29] interesting pod/e2e-worker-w1-6fc748-openshift-default-665546959f-jbtqd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 20:27:18.196419 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:18.195705    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-jbtqd" podUID="baacee7f-b912-4964-9855-be2beb0094cb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 20:27:18.971871 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:18.971830    5570 patch_prober.go:29] interesting pod/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-wv4cn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 20:27:18.972272 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:18.971903    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-wv4cn" podUID="c1e2eadf-ede0-4ea2-9eea-f25a39779795" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 20:27:19.211744 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:19.211708    5570 patch_prober.go:29] interesting pod/e2e-worker-w1-6fc748-openshift-default-665546959f-9gvtf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 20:27:19.211913 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:19.211766    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-9gvtf" podUID="5e7f17ab-0f1d-4a59-b415-fe844ae7c1e7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 20:27:19.248103 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:19.248030    5570 patch_prober.go:29] interesting pod/e2e-worker-w0-e40816-openshift-default-744c57967c-5bn7t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 20:27:19.248103 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:19.248094    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-5bn7t" podUID="26831c71-2091-4090-843d-f7268d257114" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 20:27:19.413205 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:19.413171    5570 patch_prober.go:29] interesting pod/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-wkmg8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 20:27:19.413404 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:19.413227    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-wkmg8" podUID="e314286e-4b2d-4fae-bc59-528809a8dbce" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 20:27:19.744790 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:19.744622    5570 patch_prober.go:29] interesting pod/e2e-worker-w3-482722-openshift-default-7645487df-fqjnh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 20:27:19.747148 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:19.744976    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-fqjnh" podUID="51c1be09-aa2e-47bd-a863-97f7daf94ddb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 20:27:20.466159 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:20.466073    5570 patch_prober.go:29] interesting pod/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmkq7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 20:27:20.466159 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:20.466144    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmkq7" podUID="157f80a0-a704-4bed-b544-205c3f2a52b3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 20:27:20.580370 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:20.579570    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-mt7dd"
Nov 03 20:27:20.637363 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:20.636940    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55945440-d40e-4235-8b84-d1e2a013372d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55945440-d40e-4235-8b84-d1e2a013372d-istio-envoy\") pod \"55945440-d40e-4235-8b84-d1e2a013372d\" (UID: \"55945440-d40e-4235-8b84-d1e2a013372d\") "
Nov 03 20:27:20.637363 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:20.637000    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/55945440-d40e-4235-8b84-d1e2a013372d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55945440-d40e-4235-8b84-d1e2a013372d-istio-podinfo\") pod \"55945440-d40e-4235-8b84-d1e2a013372d\" (UID: \"55945440-d40e-4235-8b84-d1e2a013372d\") "
Nov 03 20:27:20.637363 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:20.637092    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55945440-d40e-4235-8b84-d1e2a013372d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55945440-d40e-4235-8b84-d1e2a013372d-istio-data\") pod \"55945440-d40e-4235-8b84-d1e2a013372d\" (UID: \"55945440-d40e-4235-8b84-d1e2a013372d\") "
Nov 03 20:27:20.637363 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:20.637185    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55945440-d40e-4235-8b84-d1e2a013372d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55945440-d40e-4235-8b84-d1e2a013372d-istiod-ca-cert\") pod \"55945440-d40e-4235-8b84-d1e2a013372d\" (UID: \"55945440-d40e-4235-8b84-d1e2a013372d\") "
Nov 03 20:27:20.637363 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:20.637241    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55945440-d40e-4235-8b84-d1e2a013372d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55945440-d40e-4235-8b84-d1e2a013372d-workload-socket\") pod \"55945440-d40e-4235-8b84-d1e2a013372d\" (UID: \"55945440-d40e-4235-8b84-d1e2a013372d\") "
Nov 03 20:27:20.637363 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:20.637365    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55945440-d40e-4235-8b84-d1e2a013372d-istio-token\" (UniqueName: \"kubernetes.io/projected/55945440-d40e-4235-8b84-d1e2a013372d-istio-token\") pod \"55945440-d40e-4235-8b84-d1e2a013372d\" (UID: \"55945440-d40e-4235-8b84-d1e2a013372d\") "
Nov 03 20:27:20.639447 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:20.637411    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55945440-d40e-4235-8b84-d1e2a013372d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55945440-d40e-4235-8b84-d1e2a013372d-credential-socket\") pod \"55945440-d40e-4235-8b84-d1e2a013372d\" (UID: \"55945440-d40e-4235-8b84-d1e2a013372d\") "
Nov 03 20:27:20.639447 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:20.637587    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55945440-d40e-4235-8b84-d1e2a013372d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55945440-d40e-4235-8b84-d1e2a013372d-workload-certs\") pod \"55945440-d40e-4235-8b84-d1e2a013372d\" (UID: \"55945440-d40e-4235-8b84-d1e2a013372d\") "
Nov 03 20:27:20.639447 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:20.637807    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55945440-d40e-4235-8b84-d1e2a013372d-kube-api-access-blxqs\" (UniqueName: \"kubernetes.io/projected/55945440-d40e-4235-8b84-d1e2a013372d-kube-api-access-blxqs\") pod \"55945440-d40e-4235-8b84-d1e2a013372d\" (UID: \"55945440-d40e-4235-8b84-d1e2a013372d\") "
Nov 03 20:27:20.643000 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:20.642326    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55945440-d40e-4235-8b84-d1e2a013372d-credential-socket" pod "55945440-d40e-4235-8b84-d1e2a013372d" (UID: "55945440-d40e-4235-8b84-d1e2a013372d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:20.644765 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:20.644740    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-xcxnz"
Nov 03 20:27:20.646558 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:20.646528    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55945440-d40e-4235-8b84-d1e2a013372d-workload-certs" pod "55945440-d40e-4235-8b84-d1e2a013372d" (UID: "55945440-d40e-4235-8b84-d1e2a013372d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:20.646658 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:20.646625    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55945440-d40e-4235-8b84-d1e2a013372d-istiod-ca-cert" pod "55945440-d40e-4235-8b84-d1e2a013372d" (UID: "55945440-d40e-4235-8b84-d1e2a013372d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:27:20.647224 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:20.647194    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55945440-d40e-4235-8b84-d1e2a013372d-workload-socket" pod "55945440-d40e-4235-8b84-d1e2a013372d" (UID: "55945440-d40e-4235-8b84-d1e2a013372d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:20.649035 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:20.648543    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55945440-d40e-4235-8b84-d1e2a013372d-istio-data" pod "55945440-d40e-4235-8b84-d1e2a013372d" (UID: "55945440-d40e-4235-8b84-d1e2a013372d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:20.652055 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:20.652032    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-wv4cn"
Nov 03 20:27:20.660457 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:20.660423    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/55945440-d40e-4235-8b84-d1e2a013372d-istio-podinfo" pod "55945440-d40e-4235-8b84-d1e2a013372d" (UID: "55945440-d40e-4235-8b84-d1e2a013372d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:27:20.663155 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:20.662940    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55945440-d40e-4235-8b84-d1e2a013372d-istio-token" pod "55945440-d40e-4235-8b84-d1e2a013372d" (UID: "55945440-d40e-4235-8b84-d1e2a013372d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:27:20.709577 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:20.709497    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55945440-d40e-4235-8b84-d1e2a013372d-istio-envoy" pod "55945440-d40e-4235-8b84-d1e2a013372d" (UID: "55945440-d40e-4235-8b84-d1e2a013372d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:20.743897 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:20.742044    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55945440-d40e-4235-8b84-d1e2a013372d-kube-api-access-blxqs" pod "55945440-d40e-4235-8b84-d1e2a013372d" (UID: "55945440-d40e-4235-8b84-d1e2a013372d"). InnerVolumeSpecName "kube-api-access-blxqs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:27:20.744127 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:20.744084    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63727774-9118-4b83-bede-04208e8d59b9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63727774-9118-4b83-bede-04208e8d59b9-credential-socket\") pod \"63727774-9118-4b83-bede-04208e8d59b9\" (UID: \"63727774-9118-4b83-bede-04208e8d59b9\") "
Nov 03 20:27:20.744248 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:20.744152    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1e2eadf-ede0-4ea2-9eea-f25a39779795-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1e2eadf-ede0-4ea2-9eea-f25a39779795-credential-socket\") pod \"c1e2eadf-ede0-4ea2-9eea-f25a39779795\" (UID: \"c1e2eadf-ede0-4ea2-9eea-f25a39779795\") "
Nov 03 20:27:20.744248 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:20.744209    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/63727774-9118-4b83-bede-04208e8d59b9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63727774-9118-4b83-bede-04208e8d59b9-istio-podinfo\") pod \"63727774-9118-4b83-bede-04208e8d59b9\" (UID: \"63727774-9118-4b83-bede-04208e8d59b9\") "
Nov 03 20:27:20.744368 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:20.744265    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1e2eadf-ede0-4ea2-9eea-f25a39779795-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1e2eadf-ede0-4ea2-9eea-f25a39779795-istio-data\") pod \"c1e2eadf-ede0-4ea2-9eea-f25a39779795\" (UID: \"c1e2eadf-ede0-4ea2-9eea-f25a39779795\") "
Nov 03 20:27:20.744368 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:20.744323    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63727774-9118-4b83-bede-04208e8d59b9-istio-token\" (UniqueName: \"kubernetes.io/projected/63727774-9118-4b83-bede-04208e8d59b9-istio-token\") pod \"63727774-9118-4b83-bede-04208e8d59b9\" (UID: \"63727774-9118-4b83-bede-04208e8d59b9\") "
Nov 03 20:27:20.746023 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:20.744367    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1e2eadf-ede0-4ea2-9eea-f25a39779795-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1e2eadf-ede0-4ea2-9eea-f25a39779795-istio-envoy\") pod \"c1e2eadf-ede0-4ea2-9eea-f25a39779795\" (UID: \"c1e2eadf-ede0-4ea2-9eea-f25a39779795\") "
Nov 03 20:27:20.746023 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:20.744518    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63727774-9118-4b83-bede-04208e8d59b9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63727774-9118-4b83-bede-04208e8d59b9-istio-envoy\") pod \"63727774-9118-4b83-bede-04208e8d59b9\" (UID: \"63727774-9118-4b83-bede-04208e8d59b9\") "
Nov 03 20:27:20.746023 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:20.744555    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63727774-9118-4b83-bede-04208e8d59b9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63727774-9118-4b83-bede-04208e8d59b9-istiod-ca-cert\") pod \"63727774-9118-4b83-bede-04208e8d59b9\" (UID: \"63727774-9118-4b83-bede-04208e8d59b9\") "
Nov 03 20:27:20.746023 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:20.744584    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1e2eadf-ede0-4ea2-9eea-f25a39779795-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1e2eadf-ede0-4ea2-9eea-f25a39779795-workload-socket\") pod \"c1e2eadf-ede0-4ea2-9eea-f25a39779795\" (UID: \"c1e2eadf-ede0-4ea2-9eea-f25a39779795\") "
Nov 03 20:27:20.746023 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:20.744621    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1e2eadf-ede0-4ea2-9eea-f25a39779795-istio-token\" (UniqueName: \"kubernetes.io/projected/c1e2eadf-ede0-4ea2-9eea-f25a39779795-istio-token\") pod \"c1e2eadf-ede0-4ea2-9eea-f25a39779795\" (UID: \"c1e2eadf-ede0-4ea2-9eea-f25a39779795\") "
Nov 03 20:27:20.746023 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:20.744745    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63727774-9118-4b83-bede-04208e8d59b9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63727774-9118-4b83-bede-04208e8d59b9-istio-data\") pod \"63727774-9118-4b83-bede-04208e8d59b9\" (UID: \"63727774-9118-4b83-bede-04208e8d59b9\") "
Nov 03 20:27:20.746023 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:20.744762    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63727774-9118-4b83-bede-04208e8d59b9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63727774-9118-4b83-bede-04208e8d59b9-workload-socket\") pod \"63727774-9118-4b83-bede-04208e8d59b9\" (UID: \"63727774-9118-4b83-bede-04208e8d59b9\") "
Nov 03 20:27:20.746023 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:20.744845    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63727774-9118-4b83-bede-04208e8d59b9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63727774-9118-4b83-bede-04208e8d59b9-workload-certs\") pod \"63727774-9118-4b83-bede-04208e8d59b9\" (UID: \"63727774-9118-4b83-bede-04208e8d59b9\") "
Nov 03 20:27:20.746023 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:20.744890    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c1e2eadf-ede0-4ea2-9eea-f25a39779795-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1e2eadf-ede0-4ea2-9eea-f25a39779795-istio-podinfo\") pod \"c1e2eadf-ede0-4ea2-9eea-f25a39779795\" (UID: \"c1e2eadf-ede0-4ea2-9eea-f25a39779795\") "
Nov 03 20:27:20.746023 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:20.744947    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1e2eadf-ede0-4ea2-9eea-f25a39779795-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1e2eadf-ede0-4ea2-9eea-f25a39779795-workload-certs\") pod \"c1e2eadf-ede0-4ea2-9eea-f25a39779795\" (UID: \"c1e2eadf-ede0-4ea2-9eea-f25a39779795\") "
Nov 03 20:27:20.746023 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:20.744981    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1e2eadf-ede0-4ea2-9eea-f25a39779795-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1e2eadf-ede0-4ea2-9eea-f25a39779795-istiod-ca-cert\") pod \"c1e2eadf-ede0-4ea2-9eea-f25a39779795\" (UID: \"c1e2eadf-ede0-4ea2-9eea-f25a39779795\") "
Nov 03 20:27:20.746023 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:20.745058    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1e2eadf-ede0-4ea2-9eea-f25a39779795-kube-api-access-g6kv7\" (UniqueName: \"kubernetes.io/projected/c1e2eadf-ede0-4ea2-9eea-f25a39779795-kube-api-access-g6kv7\") pod \"c1e2eadf-ede0-4ea2-9eea-f25a39779795\" (UID: \"c1e2eadf-ede0-4ea2-9eea-f25a39779795\") "
Nov 03 20:27:20.746023 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:20.745126    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63727774-9118-4b83-bede-04208e8d59b9-kube-api-access-858bb\" (UniqueName: \"kubernetes.io/projected/63727774-9118-4b83-bede-04208e8d59b9-kube-api-access-858bb\") pod \"63727774-9118-4b83-bede-04208e8d59b9\" (UID: \"63727774-9118-4b83-bede-04208e8d59b9\") "
Nov 03 20:27:20.746833 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:20.746620    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1e2eadf-ede0-4ea2-9eea-f25a39779795-workload-socket" pod "c1e2eadf-ede0-4ea2-9eea-f25a39779795" (UID: "c1e2eadf-ede0-4ea2-9eea-f25a39779795"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:20.747624 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:20.747229    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1e2eadf-ede0-4ea2-9eea-f25a39779795-istio-envoy" pod "c1e2eadf-ede0-4ea2-9eea-f25a39779795" (UID: "c1e2eadf-ede0-4ea2-9eea-f25a39779795"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:20.749613 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:20.749553    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63727774-9118-4b83-bede-04208e8d59b9-workload-socket" pod "63727774-9118-4b83-bede-04208e8d59b9" (UID: "63727774-9118-4b83-bede-04208e8d59b9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:20.750560 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:20.750397    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1e2eadf-ede0-4ea2-9eea-f25a39779795-istio-data" pod "c1e2eadf-ede0-4ea2-9eea-f25a39779795" (UID: "c1e2eadf-ede0-4ea2-9eea-f25a39779795"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:20.755369 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:20.751047    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63727774-9118-4b83-bede-04208e8d59b9-workload-certs" pod "63727774-9118-4b83-bede-04208e8d59b9" (UID: "63727774-9118-4b83-bede-04208e8d59b9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:20.755369 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:20.751239    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63727774-9118-4b83-bede-04208e8d59b9-istiod-ca-cert" pod "63727774-9118-4b83-bede-04208e8d59b9" (UID: "63727774-9118-4b83-bede-04208e8d59b9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:27:20.755369 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:20.752192    5570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55945440-d40e-4235-8b84-d1e2a013372d-istio-data\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:20.755369 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:20.752217    5570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55945440-d40e-4235-8b84-d1e2a013372d-istiod-ca-cert\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:20.755369 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:20.752233    5570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55945440-d40e-4235-8b84-d1e2a013372d-workload-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:20.755369 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:20.752246    5570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55945440-d40e-4235-8b84-d1e2a013372d-istio-token\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:20.755369 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:20.752258    5570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1e2eadf-ede0-4ea2-9eea-f25a39779795-istio-data\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:20.755369 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:20.752271    5570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55945440-d40e-4235-8b84-d1e2a013372d-credential-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:20.755369 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:20.752283    5570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1e2eadf-ede0-4ea2-9eea-f25a39779795-istio-envoy\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:20.755369 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:20.752297    5570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55945440-d40e-4235-8b84-d1e2a013372d-workload-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:20.755369 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:20.752310    5570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63727774-9118-4b83-bede-04208e8d59b9-istiod-ca-cert\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:20.755369 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:20.752322    5570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1e2eadf-ede0-4ea2-9eea-f25a39779795-workload-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:20.755369 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:20.752336    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-blxqs\" (UniqueName: \"kubernetes.io/projected/55945440-d40e-4235-8b84-d1e2a013372d-kube-api-access-blxqs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:20.755369 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:20.752350    5570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63727774-9118-4b83-bede-04208e8d59b9-workload-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:20.755369 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:20.752363    5570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63727774-9118-4b83-bede-04208e8d59b9-workload-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:20.755369 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:20.752375    5570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55945440-d40e-4235-8b84-d1e2a013372d-istio-envoy\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:20.755369 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:20.752388    5570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55945440-d40e-4235-8b84-d1e2a013372d-istio-podinfo\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:20.755369 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:20.752459    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63727774-9118-4b83-bede-04208e8d59b9-istio-envoy" pod "63727774-9118-4b83-bede-04208e8d59b9" (UID: "63727774-9118-4b83-bede-04208e8d59b9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:20.755369 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:20.752793    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63727774-9118-4b83-bede-04208e8d59b9-istio-data" pod "63727774-9118-4b83-bede-04208e8d59b9" (UID: "63727774-9118-4b83-bede-04208e8d59b9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:20.755369 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:20.753089    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63727774-9118-4b83-bede-04208e8d59b9-kube-api-access-858bb" pod "63727774-9118-4b83-bede-04208e8d59b9" (UID: "63727774-9118-4b83-bede-04208e8d59b9"). InnerVolumeSpecName "kube-api-access-858bb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:27:20.755369 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:20.753568    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1e2eadf-ede0-4ea2-9eea-f25a39779795-workload-certs" pod "c1e2eadf-ede0-4ea2-9eea-f25a39779795" (UID: "c1e2eadf-ede0-4ea2-9eea-f25a39779795"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:20.755369 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:20.753092    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1e2eadf-ede0-4ea2-9eea-f25a39779795-istiod-ca-cert" pod "c1e2eadf-ede0-4ea2-9eea-f25a39779795" (UID: "c1e2eadf-ede0-4ea2-9eea-f25a39779795"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:27:20.755369 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:20.754093    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63727774-9118-4b83-bede-04208e8d59b9-credential-socket" pod "63727774-9118-4b83-bede-04208e8d59b9" (UID: "63727774-9118-4b83-bede-04208e8d59b9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:20.755369 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:20.754348    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/63727774-9118-4b83-bede-04208e8d59b9-istio-podinfo" pod "63727774-9118-4b83-bede-04208e8d59b9" (UID: "63727774-9118-4b83-bede-04208e8d59b9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:27:20.755369 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:20.755226    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1e2eadf-ede0-4ea2-9eea-f25a39779795-credential-socket" pod "c1e2eadf-ede0-4ea2-9eea-f25a39779795" (UID: "c1e2eadf-ede0-4ea2-9eea-f25a39779795"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:20.762239 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:20.759995    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1e2eadf-ede0-4ea2-9eea-f25a39779795-kube-api-access-g6kv7" pod "c1e2eadf-ede0-4ea2-9eea-f25a39779795" (UID: "c1e2eadf-ede0-4ea2-9eea-f25a39779795"). InnerVolumeSpecName "kube-api-access-g6kv7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:27:20.762239 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:20.760481    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1e2eadf-ede0-4ea2-9eea-f25a39779795-istio-token" pod "c1e2eadf-ede0-4ea2-9eea-f25a39779795" (UID: "c1e2eadf-ede0-4ea2-9eea-f25a39779795"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:27:20.762239 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:20.761108    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63727774-9118-4b83-bede-04208e8d59b9-istio-token" pod "63727774-9118-4b83-bede-04208e8d59b9" (UID: "63727774-9118-4b83-bede-04208e8d59b9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:27:20.763228 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:20.763205    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c1e2eadf-ede0-4ea2-9eea-f25a39779795-istio-podinfo" pod "c1e2eadf-ede0-4ea2-9eea-f25a39779795" (UID: "c1e2eadf-ede0-4ea2-9eea-f25a39779795"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:27:20.819032 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:20.817280    5570 generic.go:356] "Generic (PLEG): container finished" podID="55945440-d40e-4235-8b84-d1e2a013372d" containerID="3846399a93c8fddbeb80930fda59b34d3625c7cf8f08fb90f62591110fad7bec" exitCode=0
Nov 03 20:27:20.819032 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:20.817460    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-mt7dd"
Nov 03 20:27:20.819032 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:20.817919    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-mt7dd" event={"ID":"55945440-d40e-4235-8b84-d1e2a013372d","Type":"ContainerDied","Data":"3846399a93c8fddbeb80930fda59b34d3625c7cf8f08fb90f62591110fad7bec"}
Nov 03 20:27:20.819032 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:20.818021    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-mt7dd" event={"ID":"55945440-d40e-4235-8b84-d1e2a013372d","Type":"ContainerDied","Data":"8b1e9f15e462251ff6d98491599b0a0ed6b83cbdf0f9ba718552b9a2c23c3f79"}
Nov 03 20:27:20.819032 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:20.818056    5570 scope.go:122] "RemoveContainer" containerID="3846399a93c8fddbeb80930fda59b34d3625c7cf8f08fb90f62591110fad7bec"
Nov 03 20:27:20.824412 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:20.824383    5570 generic.go:356] "Generic (PLEG): container finished" podID="c1e2eadf-ede0-4ea2-9eea-f25a39779795" containerID="5c6d90f6dff94d54503259d98624356b90f1622e4c025457fbf45267a1efac96" exitCode=0
Nov 03 20:27:20.824526 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:20.824475    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-wv4cn" event={"ID":"c1e2eadf-ede0-4ea2-9eea-f25a39779795","Type":"ContainerDied","Data":"5c6d90f6dff94d54503259d98624356b90f1622e4c025457fbf45267a1efac96"}
Nov 03 20:27:20.824526 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:20.824520    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-wv4cn" event={"ID":"c1e2eadf-ede0-4ea2-9eea-f25a39779795","Type":"ContainerDied","Data":"a39a32213538df3773348b493f42a48dd7b4cba9e251838b18ed3c4dce7c42bb"}
Nov 03 20:27:20.824658 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:20.824642    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-wv4cn"
Nov 03 20:27:20.838227 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:20.838194    5570 generic.go:356] "Generic (PLEG): container finished" podID="63727774-9118-4b83-bede-04208e8d59b9" containerID="c6574ca71b7f10f0ce3bcd5398a085b8f18c8a09ee7a532e8935c39af660977c" exitCode=0
Nov 03 20:27:20.838471 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:20.838380    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-xcxnz" event={"ID":"63727774-9118-4b83-bede-04208e8d59b9","Type":"ContainerDied","Data":"c6574ca71b7f10f0ce3bcd5398a085b8f18c8a09ee7a532e8935c39af660977c"}
Nov 03 20:27:20.838471 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:20.838400    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-xcxnz"
Nov 03 20:27:20.838471 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:20.838433    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-xcxnz" event={"ID":"63727774-9118-4b83-bede-04208e8d59b9","Type":"ContainerDied","Data":"0e6626c5f047215d8796f5a752597e077e75813ee563b18ff17cc55aac9cd69c"}
Nov 03 20:27:20.855669 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:20.855637    5570 generic.go:356] "Generic (PLEG): container finished" podID="5e7f17ab-0f1d-4a59-b415-fe844ae7c1e7" containerID="3e7c2142d5118a937b9c4c1db78402b22ba93d6da618fb6dea1c3717ae2ab38b" exitCode=0
Nov 03 20:27:20.856178 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:20.856154    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-9gvtf" event={"ID":"5e7f17ab-0f1d-4a59-b415-fe844ae7c1e7","Type":"ContainerDied","Data":"3e7c2142d5118a937b9c4c1db78402b22ba93d6da618fb6dea1c3717ae2ab38b"}
Nov 03 20:27:20.858608 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:20.858581    5570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1e2eadf-ede0-4ea2-9eea-f25a39779795-workload-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:20.858736 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:20.858612    5570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1e2eadf-ede0-4ea2-9eea-f25a39779795-istiod-ca-cert\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:20.858736 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:20.858626    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g6kv7\" (UniqueName: \"kubernetes.io/projected/c1e2eadf-ede0-4ea2-9eea-f25a39779795-kube-api-access-g6kv7\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:20.858736 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:20.858639    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-858bb\" (UniqueName: \"kubernetes.io/projected/63727774-9118-4b83-bede-04208e8d59b9-kube-api-access-858bb\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:20.858736 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:20.858653    5570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63727774-9118-4b83-bede-04208e8d59b9-credential-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:20.858736 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:20.858667    5570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1e2eadf-ede0-4ea2-9eea-f25a39779795-credential-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:20.858736 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:20.858680    5570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63727774-9118-4b83-bede-04208e8d59b9-istio-podinfo\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:20.858736 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:20.858693    5570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63727774-9118-4b83-bede-04208e8d59b9-istio-token\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:20.858736 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:20.858704    5570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63727774-9118-4b83-bede-04208e8d59b9-istio-envoy\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:20.858736 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:20.858716    5570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1e2eadf-ede0-4ea2-9eea-f25a39779795-istio-token\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:20.858736 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:20.858729    5570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63727774-9118-4b83-bede-04208e8d59b9-istio-data\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:20.861268 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:20.858743    5570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1e2eadf-ede0-4ea2-9eea-f25a39779795-istio-podinfo\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:20.861268 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:20.860144    5570 generic.go:356] "Generic (PLEG): container finished" podID="6a1b3d29-b647-46e1-9005-e540e937f85f" containerID="c112fdc1a8a3d21c59453de2ff24628f1b064fd91b46b2f5e53085450bd216c0" exitCode=0
Nov 03 20:27:20.861268 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:20.860248    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-wg864" event={"ID":"6a1b3d29-b647-46e1-9005-e540e937f85f","Type":"ContainerDied","Data":"c112fdc1a8a3d21c59453de2ff24628f1b064fd91b46b2f5e53085450bd216c0"}
Nov 03 20:27:20.866354 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:20.866178    5570 generic.go:356] "Generic (PLEG): container finished" podID="baacee7f-b912-4964-9855-be2beb0094cb" containerID="f88e91ce1e2d9b540bdd3ec4f7167d68046dc7bc6e871d1e1f9efc08d87c6965" exitCode=0
Nov 03 20:27:20.866354 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:20.866287    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-jbtqd" event={"ID":"baacee7f-b912-4964-9855-be2beb0094cb","Type":"ContainerDied","Data":"f88e91ce1e2d9b540bdd3ec4f7167d68046dc7bc6e871d1e1f9efc08d87c6965"}
Nov 03 20:27:20.882255 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:20.879833    5570 generic.go:356] "Generic (PLEG): container finished" podID="ce7b1d98-c21f-4412-97ce-9dca59ce2649" containerID="2065ee88578f7048629b049802360f906227df9654abe61d45dbb67e25165dbb" exitCode=0
Nov 03 20:27:20.882255 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:20.879925    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-wj4gp" event={"ID":"ce7b1d98-c21f-4412-97ce-9dca59ce2649","Type":"ContainerDied","Data":"2065ee88578f7048629b049802360f906227df9654abe61d45dbb67e25165dbb"}
Nov 03 20:27:20.994674 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:20.994591    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-jbtqd"
Nov 03 20:27:21.016291 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.015985    5570 scope.go:122] "RemoveContainer" containerID="3846399a93c8fddbeb80930fda59b34d3625c7cf8f08fb90f62591110fad7bec"
Nov 03 20:27:21.018541 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:27:21.017875    5570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3846399a93c8fddbeb80930fda59b34d3625c7cf8f08fb90f62591110fad7bec\": container with ID starting with 3846399a93c8fddbeb80930fda59b34d3625c7cf8f08fb90f62591110fad7bec not found: ID does not exist" containerID="3846399a93c8fddbeb80930fda59b34d3625c7cf8f08fb90f62591110fad7bec"
Nov 03 20:27:21.018541 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.017915    5570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3846399a93c8fddbeb80930fda59b34d3625c7cf8f08fb90f62591110fad7bec"} err="failed to get container status \"3846399a93c8fddbeb80930fda59b34d3625c7cf8f08fb90f62591110fad7bec\": rpc error: code = NotFound desc = could not find container \"3846399a93c8fddbeb80930fda59b34d3625c7cf8f08fb90f62591110fad7bec\": container with ID starting with 3846399a93c8fddbeb80930fda59b34d3625c7cf8f08fb90f62591110fad7bec not found: ID does not exist"
Nov 03 20:27:21.018541 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.017944    5570 scope.go:122] "RemoveContainer" containerID="5c6d90f6dff94d54503259d98624356b90f1622e4c025457fbf45267a1efac96"
Nov 03 20:27:21.039170 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.038199    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-wv4cn"]
Nov 03 20:27:21.047285 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.047253    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-wv4cn"]
Nov 03 20:27:21.066883 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.065492    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/baacee7f-b912-4964-9855-be2beb0094cb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/baacee7f-b912-4964-9855-be2beb0094cb-workload-socket\") pod \"baacee7f-b912-4964-9855-be2beb0094cb\" (UID: \"baacee7f-b912-4964-9855-be2beb0094cb\") "
Nov 03 20:27:21.066883 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.065587    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/baacee7f-b912-4964-9855-be2beb0094cb-istio-token\" (UniqueName: \"kubernetes.io/projected/baacee7f-b912-4964-9855-be2beb0094cb-istio-token\") pod \"baacee7f-b912-4964-9855-be2beb0094cb\" (UID: \"baacee7f-b912-4964-9855-be2beb0094cb\") "
Nov 03 20:27:21.066883 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.065632    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/baacee7f-b912-4964-9855-be2beb0094cb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/baacee7f-b912-4964-9855-be2beb0094cb-credential-socket\") pod \"baacee7f-b912-4964-9855-be2beb0094cb\" (UID: \"baacee7f-b912-4964-9855-be2beb0094cb\") "
Nov 03 20:27:21.066883 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.065705    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/baacee7f-b912-4964-9855-be2beb0094cb-kube-api-access-cw925\" (UniqueName: \"kubernetes.io/projected/baacee7f-b912-4964-9855-be2beb0094cb-kube-api-access-cw925\") pod \"baacee7f-b912-4964-9855-be2beb0094cb\" (UID: \"baacee7f-b912-4964-9855-be2beb0094cb\") "
Nov 03 20:27:21.066883 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.065804    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/baacee7f-b912-4964-9855-be2beb0094cb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/baacee7f-b912-4964-9855-be2beb0094cb-istio-data\") pod \"baacee7f-b912-4964-9855-be2beb0094cb\" (UID: \"baacee7f-b912-4964-9855-be2beb0094cb\") "
Nov 03 20:27:21.066883 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.065838    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/baacee7f-b912-4964-9855-be2beb0094cb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/baacee7f-b912-4964-9855-be2beb0094cb-istio-envoy\") pod \"baacee7f-b912-4964-9855-be2beb0094cb\" (UID: \"baacee7f-b912-4964-9855-be2beb0094cb\") "
Nov 03 20:27:21.066883 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.066021    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/baacee7f-b912-4964-9855-be2beb0094cb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/baacee7f-b912-4964-9855-be2beb0094cb-istiod-ca-cert\") pod \"baacee7f-b912-4964-9855-be2beb0094cb\" (UID: \"baacee7f-b912-4964-9855-be2beb0094cb\") "
Nov 03 20:27:21.066883 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.066076    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/baacee7f-b912-4964-9855-be2beb0094cb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/baacee7f-b912-4964-9855-be2beb0094cb-istio-podinfo\") pod \"baacee7f-b912-4964-9855-be2beb0094cb\" (UID: \"baacee7f-b912-4964-9855-be2beb0094cb\") "
Nov 03 20:27:21.066883 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.066128    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/baacee7f-b912-4964-9855-be2beb0094cb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/baacee7f-b912-4964-9855-be2beb0094cb-workload-certs\") pod \"baacee7f-b912-4964-9855-be2beb0094cb\" (UID: \"baacee7f-b912-4964-9855-be2beb0094cb\") "
Nov 03 20:27:21.067577 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.067033    5570 scope.go:122] "RemoveContainer" containerID="5c6d90f6dff94d54503259d98624356b90f1622e4c025457fbf45267a1efac96"
Nov 03 20:27:21.070354 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.070325    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/baacee7f-b912-4964-9855-be2beb0094cb-istiod-ca-cert" pod "baacee7f-b912-4964-9855-be2beb0094cb" (UID: "baacee7f-b912-4964-9855-be2beb0094cb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:27:21.070516 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.070399    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baacee7f-b912-4964-9855-be2beb0094cb-istio-envoy" pod "baacee7f-b912-4964-9855-be2beb0094cb" (UID: "baacee7f-b912-4964-9855-be2beb0094cb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:21.070895 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.070870    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baacee7f-b912-4964-9855-be2beb0094cb-credential-socket" pod "baacee7f-b912-4964-9855-be2beb0094cb" (UID: "baacee7f-b912-4964-9855-be2beb0094cb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:21.071177 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:27:21.071151    5570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c6d90f6dff94d54503259d98624356b90f1622e4c025457fbf45267a1efac96\": container with ID starting with 5c6d90f6dff94d54503259d98624356b90f1622e4c025457fbf45267a1efac96 not found: ID does not exist" containerID="5c6d90f6dff94d54503259d98624356b90f1622e4c025457fbf45267a1efac96"
Nov 03 20:27:21.072128 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.071747    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baacee7f-b912-4964-9855-be2beb0094cb-istio-data" pod "baacee7f-b912-4964-9855-be2beb0094cb" (UID: "baacee7f-b912-4964-9855-be2beb0094cb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:21.072128 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.072090    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baacee7f-b912-4964-9855-be2beb0094cb-workload-socket" pod "baacee7f-b912-4964-9855-be2beb0094cb" (UID: "baacee7f-b912-4964-9855-be2beb0094cb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:21.076200 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.071828    5570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c6d90f6dff94d54503259d98624356b90f1622e4c025457fbf45267a1efac96"} err="failed to get container status \"5c6d90f6dff94d54503259d98624356b90f1622e4c025457fbf45267a1efac96\": rpc error: code = NotFound desc = could not find container \"5c6d90f6dff94d54503259d98624356b90f1622e4c025457fbf45267a1efac96\": container with ID starting with 5c6d90f6dff94d54503259d98624356b90f1622e4c025457fbf45267a1efac96 not found: ID does not exist"
Nov 03 20:27:21.076200 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.072279    5570 scope.go:122] "RemoveContainer" containerID="c6574ca71b7f10f0ce3bcd5398a085b8f18c8a09ee7a532e8935c39af660977c"
Nov 03 20:27:21.076200 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.072750    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baacee7f-b912-4964-9855-be2beb0094cb-workload-certs" pod "baacee7f-b912-4964-9855-be2beb0094cb" (UID: "baacee7f-b912-4964-9855-be2beb0094cb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:21.084481 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.084448    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/baacee7f-b912-4964-9855-be2beb0094cb-istio-podinfo" pod "baacee7f-b912-4964-9855-be2beb0094cb" (UID: "baacee7f-b912-4964-9855-be2beb0094cb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:27:21.084583 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.084484    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/baacee7f-b912-4964-9855-be2beb0094cb-kube-api-access-cw925" pod "baacee7f-b912-4964-9855-be2beb0094cb" (UID: "baacee7f-b912-4964-9855-be2beb0094cb"). InnerVolumeSpecName "kube-api-access-cw925". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:27:21.086535 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.085742    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/baacee7f-b912-4964-9855-be2beb0094cb-istio-token" pod "baacee7f-b912-4964-9855-be2beb0094cb" (UID: "baacee7f-b912-4964-9855-be2beb0094cb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:27:21.086875 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.086728    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-xcxnz"]
Nov 03 20:27:21.094437 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.094401    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-xcxnz"]
Nov 03 20:27:21.101349 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.101326    5570 scope.go:122] "RemoveContainer" containerID="c6574ca71b7f10f0ce3bcd5398a085b8f18c8a09ee7a532e8935c39af660977c"
Nov 03 20:27:21.102206 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:27:21.102177    5570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6574ca71b7f10f0ce3bcd5398a085b8f18c8a09ee7a532e8935c39af660977c\": container with ID starting with c6574ca71b7f10f0ce3bcd5398a085b8f18c8a09ee7a532e8935c39af660977c not found: ID does not exist" containerID="c6574ca71b7f10f0ce3bcd5398a085b8f18c8a09ee7a532e8935c39af660977c"
Nov 03 20:27:21.102347 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.102326    5570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6574ca71b7f10f0ce3bcd5398a085b8f18c8a09ee7a532e8935c39af660977c"} err="failed to get container status \"c6574ca71b7f10f0ce3bcd5398a085b8f18c8a09ee7a532e8935c39af660977c\": rpc error: code = NotFound desc = could not find container \"c6574ca71b7f10f0ce3bcd5398a085b8f18c8a09ee7a532e8935c39af660977c\": container with ID starting with c6574ca71b7f10f0ce3bcd5398a085b8f18c8a09ee7a532e8935c39af660977c not found: ID does not exist"
Nov 03 20:27:21.103105 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.102433    5570 scope.go:122] "RemoveContainer" containerID="1e34129bff2aa0dc0d4d3ecc12b6cc9633c75b5005270c89221a9df5b61b304d"
Nov 03 20:27:21.136908 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.136831    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-mt7dd"]
Nov 03 20:27:21.152499 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.152461    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-mt7dd"]
Nov 03 20:27:21.170740 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.170708    5570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/baacee7f-b912-4964-9855-be2beb0094cb-workload-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:21.170960 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.170746    5570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/baacee7f-b912-4964-9855-be2beb0094cb-istio-token\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:21.170960 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.170759    5570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/baacee7f-b912-4964-9855-be2beb0094cb-credential-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:21.170960 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.170796    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cw925\" (UniqueName: \"kubernetes.io/projected/baacee7f-b912-4964-9855-be2beb0094cb-kube-api-access-cw925\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:21.170960 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.170816    5570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/baacee7f-b912-4964-9855-be2beb0094cb-istio-data\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:21.170960 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.170829    5570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/baacee7f-b912-4964-9855-be2beb0094cb-istio-envoy\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:21.170960 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.170841    5570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/baacee7f-b912-4964-9855-be2beb0094cb-istiod-ca-cert\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:21.170960 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.170855    5570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/baacee7f-b912-4964-9855-be2beb0094cb-istio-podinfo\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:21.170960 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.170867    5570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/baacee7f-b912-4964-9855-be2beb0094cb-workload-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:21.192025 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.191888    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-9gvtf"
Nov 03 20:27:21.204187 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.204158    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-wg864"
Nov 03 20:27:21.273423 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.272851    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e7f17ab-0f1d-4a59-b415-fe844ae7c1e7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e7f17ab-0f1d-4a59-b415-fe844ae7c1e7-workload-certs\") pod \"5e7f17ab-0f1d-4a59-b415-fe844ae7c1e7\" (UID: \"5e7f17ab-0f1d-4a59-b415-fe844ae7c1e7\") "
Nov 03 20:27:21.273423 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.272932    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a1b3d29-b647-46e1-9005-e540e937f85f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a1b3d29-b647-46e1-9005-e540e937f85f-istiod-ca-cert\") pod \"6a1b3d29-b647-46e1-9005-e540e937f85f\" (UID: \"6a1b3d29-b647-46e1-9005-e540e937f85f\") "
Nov 03 20:27:21.273423 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.273117    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5e7f17ab-0f1d-4a59-b415-fe844ae7c1e7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e7f17ab-0f1d-4a59-b415-fe844ae7c1e7-istio-podinfo\") pod \"5e7f17ab-0f1d-4a59-b415-fe844ae7c1e7\" (UID: \"5e7f17ab-0f1d-4a59-b415-fe844ae7c1e7\") "
Nov 03 20:27:21.273423 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.273194    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a1b3d29-b647-46e1-9005-e540e937f85f-istio-token\" (UniqueName: \"kubernetes.io/projected/6a1b3d29-b647-46e1-9005-e540e937f85f-istio-token\") pod \"6a1b3d29-b647-46e1-9005-e540e937f85f\" (UID: \"6a1b3d29-b647-46e1-9005-e540e937f85f\") "
Nov 03 20:27:21.273423 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.273224    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e7f17ab-0f1d-4a59-b415-fe844ae7c1e7-kube-api-access-msl98\" (UniqueName: \"kubernetes.io/projected/5e7f17ab-0f1d-4a59-b415-fe844ae7c1e7-kube-api-access-msl98\") pod \"5e7f17ab-0f1d-4a59-b415-fe844ae7c1e7\" (UID: \"5e7f17ab-0f1d-4a59-b415-fe844ae7c1e7\") "
Nov 03 20:27:21.273423 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.273265    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a1b3d29-b647-46e1-9005-e540e937f85f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a1b3d29-b647-46e1-9005-e540e937f85f-istio-data\") pod \"6a1b3d29-b647-46e1-9005-e540e937f85f\" (UID: \"6a1b3d29-b647-46e1-9005-e540e937f85f\") "
Nov 03 20:27:21.273423 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.273336    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6a1b3d29-b647-46e1-9005-e540e937f85f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a1b3d29-b647-46e1-9005-e540e937f85f-istio-podinfo\") pod \"6a1b3d29-b647-46e1-9005-e540e937f85f\" (UID: \"6a1b3d29-b647-46e1-9005-e540e937f85f\") "
Nov 03 20:27:21.273423 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.273424    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e7f17ab-0f1d-4a59-b415-fe844ae7c1e7-istio-token\" (UniqueName: \"kubernetes.io/projected/5e7f17ab-0f1d-4a59-b415-fe844ae7c1e7-istio-token\") pod \"5e7f17ab-0f1d-4a59-b415-fe844ae7c1e7\" (UID: \"5e7f17ab-0f1d-4a59-b415-fe844ae7c1e7\") "
Nov 03 20:27:21.275301 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.273450    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e7f17ab-0f1d-4a59-b415-fe844ae7c1e7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e7f17ab-0f1d-4a59-b415-fe844ae7c1e7-istio-data\") pod \"5e7f17ab-0f1d-4a59-b415-fe844ae7c1e7\" (UID: \"5e7f17ab-0f1d-4a59-b415-fe844ae7c1e7\") "
Nov 03 20:27:21.275301 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.273475    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e7f17ab-0f1d-4a59-b415-fe844ae7c1e7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e7f17ab-0f1d-4a59-b415-fe844ae7c1e7-istiod-ca-cert\") pod \"5e7f17ab-0f1d-4a59-b415-fe844ae7c1e7\" (UID: \"5e7f17ab-0f1d-4a59-b415-fe844ae7c1e7\") "
Nov 03 20:27:21.275301 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.273556    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a1b3d29-b647-46e1-9005-e540e937f85f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a1b3d29-b647-46e1-9005-e540e937f85f-workload-socket\") pod \"6a1b3d29-b647-46e1-9005-e540e937f85f\" (UID: \"6a1b3d29-b647-46e1-9005-e540e937f85f\") "
Nov 03 20:27:21.275301 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.273622    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a1b3d29-b647-46e1-9005-e540e937f85f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a1b3d29-b647-46e1-9005-e540e937f85f-workload-certs\") pod \"6a1b3d29-b647-46e1-9005-e540e937f85f\" (UID: \"6a1b3d29-b647-46e1-9005-e540e937f85f\") "
Nov 03 20:27:21.275301 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.273654    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e7f17ab-0f1d-4a59-b415-fe844ae7c1e7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e7f17ab-0f1d-4a59-b415-fe844ae7c1e7-workload-socket\") pod \"5e7f17ab-0f1d-4a59-b415-fe844ae7c1e7\" (UID: \"5e7f17ab-0f1d-4a59-b415-fe844ae7c1e7\") "
Nov 03 20:27:21.275301 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.273685    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a1b3d29-b647-46e1-9005-e540e937f85f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a1b3d29-b647-46e1-9005-e540e937f85f-credential-socket\") pod \"6a1b3d29-b647-46e1-9005-e540e937f85f\" (UID: \"6a1b3d29-b647-46e1-9005-e540e937f85f\") "
Nov 03 20:27:21.275301 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.273733    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a1b3d29-b647-46e1-9005-e540e937f85f-kube-api-access-stxjs\" (UniqueName: \"kubernetes.io/projected/6a1b3d29-b647-46e1-9005-e540e937f85f-kube-api-access-stxjs\") pod \"6a1b3d29-b647-46e1-9005-e540e937f85f\" (UID: \"6a1b3d29-b647-46e1-9005-e540e937f85f\") "
Nov 03 20:27:21.275301 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.273789    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e7f17ab-0f1d-4a59-b415-fe844ae7c1e7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e7f17ab-0f1d-4a59-b415-fe844ae7c1e7-istio-envoy\") pod \"5e7f17ab-0f1d-4a59-b415-fe844ae7c1e7\" (UID: \"5e7f17ab-0f1d-4a59-b415-fe844ae7c1e7\") "
Nov 03 20:27:21.275301 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.273872    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e7f17ab-0f1d-4a59-b415-fe844ae7c1e7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e7f17ab-0f1d-4a59-b415-fe844ae7c1e7-credential-socket\") pod \"5e7f17ab-0f1d-4a59-b415-fe844ae7c1e7\" (UID: \"5e7f17ab-0f1d-4a59-b415-fe844ae7c1e7\") "
Nov 03 20:27:21.275301 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.273936    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a1b3d29-b647-46e1-9005-e540e937f85f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a1b3d29-b647-46e1-9005-e540e937f85f-istio-envoy\") pod \"6a1b3d29-b647-46e1-9005-e540e937f85f\" (UID: \"6a1b3d29-b647-46e1-9005-e540e937f85f\") "
Nov 03 20:27:21.277203 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.276424    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a1b3d29-b647-46e1-9005-e540e937f85f-credential-socket" pod "6a1b3d29-b647-46e1-9005-e540e937f85f" (UID: "6a1b3d29-b647-46e1-9005-e540e937f85f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:21.277203 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.277036    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e7f17ab-0f1d-4a59-b415-fe844ae7c1e7-istio-data" pod "5e7f17ab-0f1d-4a59-b415-fe844ae7c1e7" (UID: "5e7f17ab-0f1d-4a59-b415-fe844ae7c1e7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:21.280535 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.278651    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a1b3d29-b647-46e1-9005-e540e937f85f-workload-socket" pod "6a1b3d29-b647-46e1-9005-e540e937f85f" (UID: "6a1b3d29-b647-46e1-9005-e540e937f85f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:21.280535 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.279231    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e7f17ab-0f1d-4a59-b415-fe844ae7c1e7-workload-socket" pod "5e7f17ab-0f1d-4a59-b415-fe844ae7c1e7" (UID: "5e7f17ab-0f1d-4a59-b415-fe844ae7c1e7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:21.280535 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.279435    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a1b3d29-b647-46e1-9005-e540e937f85f-workload-certs" pod "6a1b3d29-b647-46e1-9005-e540e937f85f" (UID: "6a1b3d29-b647-46e1-9005-e540e937f85f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:21.280535 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.279739    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e7f17ab-0f1d-4a59-b415-fe844ae7c1e7-istiod-ca-cert" pod "5e7f17ab-0f1d-4a59-b415-fe844ae7c1e7" (UID: "5e7f17ab-0f1d-4a59-b415-fe844ae7c1e7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:27:21.280535 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.280042    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e7f17ab-0f1d-4a59-b415-fe844ae7c1e7-credential-socket" pod "5e7f17ab-0f1d-4a59-b415-fe844ae7c1e7" (UID: "5e7f17ab-0f1d-4a59-b415-fe844ae7c1e7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:21.283560 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.281203    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a1b3d29-b647-46e1-9005-e540e937f85f-istio-token" pod "6a1b3d29-b647-46e1-9005-e540e937f85f" (UID: "6a1b3d29-b647-46e1-9005-e540e937f85f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:27:21.283560 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.281634    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a1b3d29-b647-46e1-9005-e540e937f85f-istiod-ca-cert" pod "6a1b3d29-b647-46e1-9005-e540e937f85f" (UID: "6a1b3d29-b647-46e1-9005-e540e937f85f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:27:21.283560 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.282215    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e7f17ab-0f1d-4a59-b415-fe844ae7c1e7-workload-certs" pod "5e7f17ab-0f1d-4a59-b415-fe844ae7c1e7" (UID: "5e7f17ab-0f1d-4a59-b415-fe844ae7c1e7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:21.283560 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.282713    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a1b3d29-b647-46e1-9005-e540e937f85f-istio-data" pod "6a1b3d29-b647-46e1-9005-e540e937f85f" (UID: "6a1b3d29-b647-46e1-9005-e540e937f85f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:21.287666 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.287624    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a1b3d29-b647-46e1-9005-e540e937f85f-kube-api-access-stxjs" pod "6a1b3d29-b647-46e1-9005-e540e937f85f" (UID: "6a1b3d29-b647-46e1-9005-e540e937f85f"). InnerVolumeSpecName "kube-api-access-stxjs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:27:21.289976 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.289144    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e7f17ab-0f1d-4a59-b415-fe844ae7c1e7-istio-envoy" pod "5e7f17ab-0f1d-4a59-b415-fe844ae7c1e7" (UID: "5e7f17ab-0f1d-4a59-b415-fe844ae7c1e7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:21.301614 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.301382    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e7f17ab-0f1d-4a59-b415-fe844ae7c1e7-istio-token" pod "5e7f17ab-0f1d-4a59-b415-fe844ae7c1e7" (UID: "5e7f17ab-0f1d-4a59-b415-fe844ae7c1e7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:27:21.311127 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.311056    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a1b3d29-b647-46e1-9005-e540e937f85f-istio-envoy" pod "6a1b3d29-b647-46e1-9005-e540e937f85f" (UID: "6a1b3d29-b647-46e1-9005-e540e937f85f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:21.315564 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.315467    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e7f17ab-0f1d-4a59-b415-fe844ae7c1e7-kube-api-access-msl98" pod "5e7f17ab-0f1d-4a59-b415-fe844ae7c1e7" (UID: "5e7f17ab-0f1d-4a59-b415-fe844ae7c1e7"). InnerVolumeSpecName "kube-api-access-msl98". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:27:21.317223 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.317171    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6a1b3d29-b647-46e1-9005-e540e937f85f-istio-podinfo" pod "6a1b3d29-b647-46e1-9005-e540e937f85f" (UID: "6a1b3d29-b647-46e1-9005-e540e937f85f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:27:21.345964 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.326496    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5e7f17ab-0f1d-4a59-b415-fe844ae7c1e7-istio-podinfo" pod "5e7f17ab-0f1d-4a59-b415-fe844ae7c1e7" (UID: "5e7f17ab-0f1d-4a59-b415-fe844ae7c1e7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:27:21.380297 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.380022    5570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e7f17ab-0f1d-4a59-b415-fe844ae7c1e7-istio-podinfo\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:21.380297 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.380041    5570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a1b3d29-b647-46e1-9005-e540e937f85f-istio-token\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:21.380297 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.380050    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-msl98\" (UniqueName: \"kubernetes.io/projected/5e7f17ab-0f1d-4a59-b415-fe844ae7c1e7-kube-api-access-msl98\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:21.380297 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.380060    5570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a1b3d29-b647-46e1-9005-e540e937f85f-istio-data\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:21.380297 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.380068    5570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a1b3d29-b647-46e1-9005-e540e937f85f-istio-podinfo\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:21.380297 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.380076    5570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e7f17ab-0f1d-4a59-b415-fe844ae7c1e7-istio-token\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:21.380297 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.380086    5570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e7f17ab-0f1d-4a59-b415-fe844ae7c1e7-istio-data\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:21.380297 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.380098    5570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e7f17ab-0f1d-4a59-b415-fe844ae7c1e7-istiod-ca-cert\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:21.380297 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.380107    5570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a1b3d29-b647-46e1-9005-e540e937f85f-workload-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:21.380297 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.380115    5570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a1b3d29-b647-46e1-9005-e540e937f85f-workload-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:21.380297 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.380123    5570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e7f17ab-0f1d-4a59-b415-fe844ae7c1e7-workload-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:21.380297 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.380132    5570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a1b3d29-b647-46e1-9005-e540e937f85f-credential-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:21.380297 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.380140    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-stxjs\" (UniqueName: \"kubernetes.io/projected/6a1b3d29-b647-46e1-9005-e540e937f85f-kube-api-access-stxjs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:21.380297 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.380148    5570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e7f17ab-0f1d-4a59-b415-fe844ae7c1e7-istio-envoy\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:21.380297 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.380156    5570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e7f17ab-0f1d-4a59-b415-fe844ae7c1e7-credential-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:21.380297 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.380168    5570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a1b3d29-b647-46e1-9005-e540e937f85f-istio-envoy\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:21.380297 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.380179    5570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e7f17ab-0f1d-4a59-b415-fe844ae7c1e7-workload-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:21.380297 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.380188    5570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a1b3d29-b647-46e1-9005-e540e937f85f-istiod-ca-cert\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:21.393420 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.393086    5570 patch_prober.go:29] interesting pod/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-7458z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 20:27:21.393420 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.393160    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-7458z" podUID="3c3d9f2d-dcbb-4fb3-a54f-ed9cdef4118e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 20:27:21.496398 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.496370    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-xszg4"
Nov 03 20:27:21.523751 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.501083    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-5c82t"
Nov 03 20:27:21.523751 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.523618    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-5bn7t"
Nov 03 20:27:21.526863 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.526703    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-wj4gp"
Nov 03 20:27:21.535249 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.535174    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-fqjnh"
Nov 03 20:27:21.602372 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.602332    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce7b1d98-c21f-4412-97ce-9dca59ce2649-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce7b1d98-c21f-4412-97ce-9dca59ce2649-workload-socket\") pod \"ce7b1d98-c21f-4412-97ce-9dca59ce2649\" (UID: \"ce7b1d98-c21f-4412-97ce-9dca59ce2649\") "
Nov 03 20:27:21.602657 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.602633    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ee1819e-4649-4d3e-9cfc-0d2d72c1b481-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ee1819e-4649-4d3e-9cfc-0d2d72c1b481-istiod-ca-cert\") pod \"4ee1819e-4649-4d3e-9cfc-0d2d72c1b481\" (UID: \"4ee1819e-4649-4d3e-9cfc-0d2d72c1b481\") "
Nov 03 20:27:21.602847 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.602758    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce7b1d98-c21f-4412-97ce-9dca59ce2649-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce7b1d98-c21f-4412-97ce-9dca59ce2649-istiod-ca-cert\") pod \"ce7b1d98-c21f-4412-97ce-9dca59ce2649\" (UID: \"ce7b1d98-c21f-4412-97ce-9dca59ce2649\") "
Nov 03 20:27:21.602847 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.602824    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26831c71-2091-4090-843d-f7268d257114-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26831c71-2091-4090-843d-f7268d257114-istiod-ca-cert\") pod \"26831c71-2091-4090-843d-f7268d257114\" (UID: \"26831c71-2091-4090-843d-f7268d257114\") "
Nov 03 20:27:21.603016 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.602852    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ee1819e-4649-4d3e-9cfc-0d2d72c1b481-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ee1819e-4649-4d3e-9cfc-0d2d72c1b481-istio-envoy\") pod \"4ee1819e-4649-4d3e-9cfc-0d2d72c1b481\" (UID: \"4ee1819e-4649-4d3e-9cfc-0d2d72c1b481\") "
Nov 03 20:27:21.603016 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.602886    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26831c71-2091-4090-843d-f7268d257114-kube-api-access-9cljz\" (UniqueName: \"kubernetes.io/projected/26831c71-2091-4090-843d-f7268d257114-kube-api-access-9cljz\") pod \"26831c71-2091-4090-843d-f7268d257114\" (UID: \"26831c71-2091-4090-843d-f7268d257114\") "
Nov 03 20:27:21.603016 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.602932    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/26831c71-2091-4090-843d-f7268d257114-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26831c71-2091-4090-843d-f7268d257114-istio-podinfo\") pod \"26831c71-2091-4090-843d-f7268d257114\" (UID: \"26831c71-2091-4090-843d-f7268d257114\") "
Nov 03 20:27:21.603016 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.602972    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51c1be09-aa2e-47bd-a863-97f7daf94ddb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51c1be09-aa2e-47bd-a863-97f7daf94ddb-istio-envoy\") pod \"51c1be09-aa2e-47bd-a863-97f7daf94ddb\" (UID: \"51c1be09-aa2e-47bd-a863-97f7daf94ddb\") "
Nov 03 20:27:21.603016 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.602999    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6029825e-e953-4c43-9f4d-d19b5ca38623-istio-token\" (UniqueName: \"kubernetes.io/projected/6029825e-e953-4c43-9f4d-d19b5ca38623-istio-token\") pod \"6029825e-e953-4c43-9f4d-d19b5ca38623\" (UID: \"6029825e-e953-4c43-9f4d-d19b5ca38623\") "
Nov 03 20:27:21.604307 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.603032    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6029825e-e953-4c43-9f4d-d19b5ca38623-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6029825e-e953-4c43-9f4d-d19b5ca38623-istio-envoy\") pod \"6029825e-e953-4c43-9f4d-d19b5ca38623\" (UID: \"6029825e-e953-4c43-9f4d-d19b5ca38623\") "
Nov 03 20:27:21.604307 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.603078    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51c1be09-aa2e-47bd-a863-97f7daf94ddb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51c1be09-aa2e-47bd-a863-97f7daf94ddb-credential-socket\") pod \"51c1be09-aa2e-47bd-a863-97f7daf94ddb\" (UID: \"51c1be09-aa2e-47bd-a863-97f7daf94ddb\") "
Nov 03 20:27:21.604307 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.603111    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ee1819e-4649-4d3e-9cfc-0d2d72c1b481-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ee1819e-4649-4d3e-9cfc-0d2d72c1b481-istio-data\") pod \"4ee1819e-4649-4d3e-9cfc-0d2d72c1b481\" (UID: \"4ee1819e-4649-4d3e-9cfc-0d2d72c1b481\") "
Nov 03 20:27:21.604307 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.603159    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4ee1819e-4649-4d3e-9cfc-0d2d72c1b481-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ee1819e-4649-4d3e-9cfc-0d2d72c1b481-istio-podinfo\") pod \"4ee1819e-4649-4d3e-9cfc-0d2d72c1b481\" (UID: \"4ee1819e-4649-4d3e-9cfc-0d2d72c1b481\") "
Nov 03 20:27:21.604307 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.603197    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51c1be09-aa2e-47bd-a863-97f7daf94ddb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51c1be09-aa2e-47bd-a863-97f7daf94ddb-workload-socket\") pod \"51c1be09-aa2e-47bd-a863-97f7daf94ddb\" (UID: \"51c1be09-aa2e-47bd-a863-97f7daf94ddb\") "
Nov 03 20:27:21.604307 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.603252    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6029825e-e953-4c43-9f4d-d19b5ca38623-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6029825e-e953-4c43-9f4d-d19b5ca38623-istio-podinfo\") pod \"6029825e-e953-4c43-9f4d-d19b5ca38623\" (UID: \"6029825e-e953-4c43-9f4d-d19b5ca38623\") "
Nov 03 20:27:21.604307 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.603294    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ce7b1d98-c21f-4412-97ce-9dca59ce2649-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce7b1d98-c21f-4412-97ce-9dca59ce2649-istio-podinfo\") pod \"ce7b1d98-c21f-4412-97ce-9dca59ce2649\" (UID: \"ce7b1d98-c21f-4412-97ce-9dca59ce2649\") "
Nov 03 20:27:21.604307 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.603332    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26831c71-2091-4090-843d-f7268d257114-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26831c71-2091-4090-843d-f7268d257114-workload-socket\") pod \"26831c71-2091-4090-843d-f7268d257114\" (UID: \"26831c71-2091-4090-843d-f7268d257114\") "
Nov 03 20:27:21.604307 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.603372    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ee1819e-4649-4d3e-9cfc-0d2d72c1b481-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ee1819e-4649-4d3e-9cfc-0d2d72c1b481-workload-certs\") pod \"4ee1819e-4649-4d3e-9cfc-0d2d72c1b481\" (UID: \"4ee1819e-4649-4d3e-9cfc-0d2d72c1b481\") "
Nov 03 20:27:21.604307 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.603400    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce7b1d98-c21f-4412-97ce-9dca59ce2649-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce7b1d98-c21f-4412-97ce-9dca59ce2649-istio-envoy\") pod \"ce7b1d98-c21f-4412-97ce-9dca59ce2649\" (UID: \"ce7b1d98-c21f-4412-97ce-9dca59ce2649\") "
Nov 03 20:27:21.604307 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.603502    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce7b1d98-c21f-4412-97ce-9dca59ce2649-workload-socket" pod "ce7b1d98-c21f-4412-97ce-9dca59ce2649" (UID: "ce7b1d98-c21f-4412-97ce-9dca59ce2649"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:21.630760 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.604579    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51c1be09-aa2e-47bd-a863-97f7daf94ddb-credential-socket" pod "51c1be09-aa2e-47bd-a863-97f7daf94ddb" (UID: "51c1be09-aa2e-47bd-a863-97f7daf94ddb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:21.630760 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.607397    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ee1819e-4649-4d3e-9cfc-0d2d72c1b481-istio-data" pod "4ee1819e-4649-4d3e-9cfc-0d2d72c1b481" (UID: "4ee1819e-4649-4d3e-9cfc-0d2d72c1b481"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:21.630760 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.607631    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51c1be09-aa2e-47bd-a863-97f7daf94ddb-workload-socket" pod "51c1be09-aa2e-47bd-a863-97f7daf94ddb" (UID: "51c1be09-aa2e-47bd-a863-97f7daf94ddb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:21.630760 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.607703    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ee1819e-4649-4d3e-9cfc-0d2d72c1b481-workload-certs" pod "4ee1819e-4649-4d3e-9cfc-0d2d72c1b481" (UID: "4ee1819e-4649-4d3e-9cfc-0d2d72c1b481"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:21.630760 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.607792    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ee1819e-4649-4d3e-9cfc-0d2d72c1b481-kube-api-access-bbpvb\" (UniqueName: \"kubernetes.io/projected/4ee1819e-4649-4d3e-9cfc-0d2d72c1b481-kube-api-access-bbpvb\") pod \"4ee1819e-4649-4d3e-9cfc-0d2d72c1b481\" (UID: \"4ee1819e-4649-4d3e-9cfc-0d2d72c1b481\") "
Nov 03 20:27:21.630760 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.607864    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51c1be09-aa2e-47bd-a863-97f7daf94ddb-istio-token\" (UniqueName: \"kubernetes.io/projected/51c1be09-aa2e-47bd-a863-97f7daf94ddb-istio-token\") pod \"51c1be09-aa2e-47bd-a863-97f7daf94ddb\" (UID: \"51c1be09-aa2e-47bd-a863-97f7daf94ddb\") "
Nov 03 20:27:21.630760 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.607896    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6029825e-e953-4c43-9f4d-d19b5ca38623-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6029825e-e953-4c43-9f4d-d19b5ca38623-credential-socket\") pod \"6029825e-e953-4c43-9f4d-d19b5ca38623\" (UID: \"6029825e-e953-4c43-9f4d-d19b5ca38623\") "
Nov 03 20:27:21.630760 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.607957    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26831c71-2091-4090-843d-f7268d257114-istio-token\" (UniqueName: \"kubernetes.io/projected/26831c71-2091-4090-843d-f7268d257114-istio-token\") pod \"26831c71-2091-4090-843d-f7268d257114\" (UID: \"26831c71-2091-4090-843d-f7268d257114\") "
Nov 03 20:27:21.630760 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.608047    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce7b1d98-c21f-4412-97ce-9dca59ce2649-kube-api-access-56hdq\" (UniqueName: \"kubernetes.io/projected/ce7b1d98-c21f-4412-97ce-9dca59ce2649-kube-api-access-56hdq\") pod \"ce7b1d98-c21f-4412-97ce-9dca59ce2649\" (UID: \"ce7b1d98-c21f-4412-97ce-9dca59ce2649\") "
Nov 03 20:27:21.630760 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.608154    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ee1819e-4649-4d3e-9cfc-0d2d72c1b481-istio-token\" (UniqueName: \"kubernetes.io/projected/4ee1819e-4649-4d3e-9cfc-0d2d72c1b481-istio-token\") pod \"4ee1819e-4649-4d3e-9cfc-0d2d72c1b481\" (UID: \"4ee1819e-4649-4d3e-9cfc-0d2d72c1b481\") "
Nov 03 20:27:21.630760 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.608261    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6029825e-e953-4c43-9f4d-d19b5ca38623-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6029825e-e953-4c43-9f4d-d19b5ca38623-workload-certs\") pod \"6029825e-e953-4c43-9f4d-d19b5ca38623\" (UID: \"6029825e-e953-4c43-9f4d-d19b5ca38623\") "
Nov 03 20:27:21.630760 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.608389    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6029825e-e953-4c43-9f4d-d19b5ca38623-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6029825e-e953-4c43-9f4d-d19b5ca38623-istiod-ca-cert\") pod \"6029825e-e953-4c43-9f4d-d19b5ca38623\" (UID: \"6029825e-e953-4c43-9f4d-d19b5ca38623\") "
Nov 03 20:27:21.630760 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.608434    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6029825e-e953-4c43-9f4d-d19b5ca38623-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6029825e-e953-4c43-9f4d-d19b5ca38623-istio-data\") pod \"6029825e-e953-4c43-9f4d-d19b5ca38623\" (UID: \"6029825e-e953-4c43-9f4d-d19b5ca38623\") "
Nov 03 20:27:21.630760 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.608462    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ee1819e-4649-4d3e-9cfc-0d2d72c1b481-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ee1819e-4649-4d3e-9cfc-0d2d72c1b481-credential-socket\") pod \"4ee1819e-4649-4d3e-9cfc-0d2d72c1b481\" (UID: \"4ee1819e-4649-4d3e-9cfc-0d2d72c1b481\") "
Nov 03 20:27:21.630760 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.608491    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26831c71-2091-4090-843d-f7268d257114-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26831c71-2091-4090-843d-f7268d257114-workload-certs\") pod \"26831c71-2091-4090-843d-f7268d257114\" (UID: \"26831c71-2091-4090-843d-f7268d257114\") "
Nov 03 20:27:21.630760 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.608586    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/51c1be09-aa2e-47bd-a863-97f7daf94ddb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51c1be09-aa2e-47bd-a863-97f7daf94ddb-istio-podinfo\") pod \"51c1be09-aa2e-47bd-a863-97f7daf94ddb\" (UID: \"51c1be09-aa2e-47bd-a863-97f7daf94ddb\") "
Nov 03 20:27:21.630760 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.608640    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce7b1d98-c21f-4412-97ce-9dca59ce2649-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce7b1d98-c21f-4412-97ce-9dca59ce2649-istio-data\") pod \"ce7b1d98-c21f-4412-97ce-9dca59ce2649\" (UID: \"ce7b1d98-c21f-4412-97ce-9dca59ce2649\") "
Nov 03 20:27:21.630760 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.608720    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51c1be09-aa2e-47bd-a863-97f7daf94ddb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51c1be09-aa2e-47bd-a863-97f7daf94ddb-istio-data\") pod \"51c1be09-aa2e-47bd-a863-97f7daf94ddb\" (UID: \"51c1be09-aa2e-47bd-a863-97f7daf94ddb\") "
Nov 03 20:27:21.630760 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.608767    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51c1be09-aa2e-47bd-a863-97f7daf94ddb-kube-api-access-sc86d\" (UniqueName: \"kubernetes.io/projected/51c1be09-aa2e-47bd-a863-97f7daf94ddb-kube-api-access-sc86d\") pod \"51c1be09-aa2e-47bd-a863-97f7daf94ddb\" (UID: \"51c1be09-aa2e-47bd-a863-97f7daf94ddb\") "
Nov 03 20:27:21.630760 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.608829    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26831c71-2091-4090-843d-f7268d257114-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26831c71-2091-4090-843d-f7268d257114-istio-data\") pod \"26831c71-2091-4090-843d-f7268d257114\" (UID: \"26831c71-2091-4090-843d-f7268d257114\") "
Nov 03 20:27:21.630760 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.608871    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce7b1d98-c21f-4412-97ce-9dca59ce2649-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce7b1d98-c21f-4412-97ce-9dca59ce2649-credential-socket\") pod \"ce7b1d98-c21f-4412-97ce-9dca59ce2649\" (UID: \"ce7b1d98-c21f-4412-97ce-9dca59ce2649\") "
Nov 03 20:27:21.630760 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.608925    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce7b1d98-c21f-4412-97ce-9dca59ce2649-istio-token\" (UniqueName: \"kubernetes.io/projected/ce7b1d98-c21f-4412-97ce-9dca59ce2649-istio-token\") pod \"ce7b1d98-c21f-4412-97ce-9dca59ce2649\" (UID: \"ce7b1d98-c21f-4412-97ce-9dca59ce2649\") "
Nov 03 20:27:21.630760 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.608994    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce7b1d98-c21f-4412-97ce-9dca59ce2649-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce7b1d98-c21f-4412-97ce-9dca59ce2649-workload-certs\") pod \"ce7b1d98-c21f-4412-97ce-9dca59ce2649\" (UID: \"ce7b1d98-c21f-4412-97ce-9dca59ce2649\") "
Nov 03 20:27:21.630760 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.609063    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6029825e-e953-4c43-9f4d-d19b5ca38623-kube-api-access-ff2pz\" (UniqueName: \"kubernetes.io/projected/6029825e-e953-4c43-9f4d-d19b5ca38623-kube-api-access-ff2pz\") pod \"6029825e-e953-4c43-9f4d-d19b5ca38623\" (UID: \"6029825e-e953-4c43-9f4d-d19b5ca38623\") "
Nov 03 20:27:21.630760 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.609234    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26831c71-2091-4090-843d-f7268d257114-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26831c71-2091-4090-843d-f7268d257114-istio-envoy\") pod \"26831c71-2091-4090-843d-f7268d257114\" (UID: \"26831c71-2091-4090-843d-f7268d257114\") "
Nov 03 20:27:21.630760 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.609287    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51c1be09-aa2e-47bd-a863-97f7daf94ddb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51c1be09-aa2e-47bd-a863-97f7daf94ddb-workload-certs\") pod \"51c1be09-aa2e-47bd-a863-97f7daf94ddb\" (UID: \"51c1be09-aa2e-47bd-a863-97f7daf94ddb\") "
Nov 03 20:27:21.630760 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.609316    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6029825e-e953-4c43-9f4d-d19b5ca38623-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6029825e-e953-4c43-9f4d-d19b5ca38623-workload-socket\") pod \"6029825e-e953-4c43-9f4d-d19b5ca38623\" (UID: \"6029825e-e953-4c43-9f4d-d19b5ca38623\") "
Nov 03 20:27:21.630760 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.609380    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ee1819e-4649-4d3e-9cfc-0d2d72c1b481-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ee1819e-4649-4d3e-9cfc-0d2d72c1b481-workload-socket\") pod \"4ee1819e-4649-4d3e-9cfc-0d2d72c1b481\" (UID: \"4ee1819e-4649-4d3e-9cfc-0d2d72c1b481\") "
Nov 03 20:27:21.630760 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.609457    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51c1be09-aa2e-47bd-a863-97f7daf94ddb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51c1be09-aa2e-47bd-a863-97f7daf94ddb-istiod-ca-cert\") pod \"51c1be09-aa2e-47bd-a863-97f7daf94ddb\" (UID: \"51c1be09-aa2e-47bd-a863-97f7daf94ddb\") "
Nov 03 20:27:21.630760 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.609577    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26831c71-2091-4090-843d-f7268d257114-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26831c71-2091-4090-843d-f7268d257114-credential-socket\") pod \"26831c71-2091-4090-843d-f7268d257114\" (UID: \"26831c71-2091-4090-843d-f7268d257114\") "
Nov 03 20:27:21.630760 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.614992    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ee1819e-4649-4d3e-9cfc-0d2d72c1b481-istiod-ca-cert" pod "4ee1819e-4649-4d3e-9cfc-0d2d72c1b481" (UID: "4ee1819e-4649-4d3e-9cfc-0d2d72c1b481"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:27:21.630760 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.615574    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4ee1819e-4649-4d3e-9cfc-0d2d72c1b481-istio-podinfo" pod "4ee1819e-4649-4d3e-9cfc-0d2d72c1b481" (UID: "4ee1819e-4649-4d3e-9cfc-0d2d72c1b481"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:27:21.630760 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.629879    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce7b1d98-c21f-4412-97ce-9dca59ce2649-istiod-ca-cert" pod "ce7b1d98-c21f-4412-97ce-9dca59ce2649" (UID: "ce7b1d98-c21f-4412-97ce-9dca59ce2649"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:27:21.630760 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.630355    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26831c71-2091-4090-843d-f7268d257114-credential-socket" pod "26831c71-2091-4090-843d-f7268d257114" (UID: "26831c71-2091-4090-843d-f7268d257114"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:21.630760 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.630362    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6029825e-e953-4c43-9f4d-d19b5ca38623-workload-certs" pod "6029825e-e953-4c43-9f4d-d19b5ca38623" (UID: "6029825e-e953-4c43-9f4d-d19b5ca38623"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:21.630760 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.630507    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26831c71-2091-4090-843d-f7268d257114-istiod-ca-cert" pod "26831c71-2091-4090-843d-f7268d257114" (UID: "26831c71-2091-4090-843d-f7268d257114"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:27:21.643927 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.631088    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26831c71-2091-4090-843d-f7268d257114-workload-socket" pod "26831c71-2091-4090-843d-f7268d257114" (UID: "26831c71-2091-4090-843d-f7268d257114"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:21.643927 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.634399    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6029825e-e953-4c43-9f4d-d19b5ca38623-istio-token" pod "6029825e-e953-4c43-9f4d-d19b5ca38623" (UID: "6029825e-e953-4c43-9f4d-d19b5ca38623"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:27:21.643927 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.636396    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce7b1d98-c21f-4412-97ce-9dca59ce2649-kube-api-access-56hdq" pod "ce7b1d98-c21f-4412-97ce-9dca59ce2649" (UID: "ce7b1d98-c21f-4412-97ce-9dca59ce2649"). InnerVolumeSpecName "kube-api-access-56hdq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:27:21.643927 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.636811    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ee1819e-4649-4d3e-9cfc-0d2d72c1b481-kube-api-access-bbpvb" pod "4ee1819e-4649-4d3e-9cfc-0d2d72c1b481" (UID: "4ee1819e-4649-4d3e-9cfc-0d2d72c1b481"). InnerVolumeSpecName "kube-api-access-bbpvb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:27:21.643927 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.637313    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6029825e-e953-4c43-9f4d-d19b5ca38623-istio-data" pod "6029825e-e953-4c43-9f4d-d19b5ca38623" (UID: "6029825e-e953-4c43-9f4d-d19b5ca38623"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:21.643927 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.637335    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce7b1d98-c21f-4412-97ce-9dca59ce2649-istio-envoy" pod "ce7b1d98-c21f-4412-97ce-9dca59ce2649" (UID: "ce7b1d98-c21f-4412-97ce-9dca59ce2649"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:21.643927 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.637401    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6029825e-e953-4c43-9f4d-d19b5ca38623-credential-socket" pod "6029825e-e953-4c43-9f4d-d19b5ca38623" (UID: "6029825e-e953-4c43-9f4d-d19b5ca38623"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:21.643927 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.637420    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ee1819e-4649-4d3e-9cfc-0d2d72c1b481-istio-envoy" pod "4ee1819e-4649-4d3e-9cfc-0d2d72c1b481" (UID: "4ee1819e-4649-4d3e-9cfc-0d2d72c1b481"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:21.643927 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.637665    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ee1819e-4649-4d3e-9cfc-0d2d72c1b481-credential-socket" pod "4ee1819e-4649-4d3e-9cfc-0d2d72c1b481" (UID: "4ee1819e-4649-4d3e-9cfc-0d2d72c1b481"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:21.643927 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.638228    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26831c71-2091-4090-843d-f7268d257114-workload-certs" pod "26831c71-2091-4090-843d-f7268d257114" (UID: "26831c71-2091-4090-843d-f7268d257114"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:21.643927 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.638640    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26831c71-2091-4090-843d-f7268d257114-istio-token" pod "26831c71-2091-4090-843d-f7268d257114" (UID: "26831c71-2091-4090-843d-f7268d257114"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:27:21.643927 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.638877    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-4cdbeb-77575c99cb-7pz9r"
Nov 03 20:27:21.643927 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.639551    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ce7b1d98-c21f-4412-97ce-9dca59ce2649-istio-podinfo" pod "ce7b1d98-c21f-4412-97ce-9dca59ce2649" (UID: "ce7b1d98-c21f-4412-97ce-9dca59ce2649"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:27:21.643927 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.639868    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51c1be09-aa2e-47bd-a863-97f7daf94ddb-istio-data" pod "51c1be09-aa2e-47bd-a863-97f7daf94ddb" (UID: "51c1be09-aa2e-47bd-a863-97f7daf94ddb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:21.643927 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.640407    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6029825e-e953-4c43-9f4d-d19b5ca38623-istiod-ca-cert" pod "6029825e-e953-4c43-9f4d-d19b5ca38623" (UID: "6029825e-e953-4c43-9f4d-d19b5ca38623"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:27:21.643927 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.641137    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce7b1d98-c21f-4412-97ce-9dca59ce2649-istio-token" pod "ce7b1d98-c21f-4412-97ce-9dca59ce2649" (UID: "ce7b1d98-c21f-4412-97ce-9dca59ce2649"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:27:21.643927 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.641203    5570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ee1819e-4649-4d3e-9cfc-0d2d72c1b481-istio-podinfo\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:21.643927 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.641226    5570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51c1be09-aa2e-47bd-a863-97f7daf94ddb-workload-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:21.643927 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.641241    5570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce7b1d98-c21f-4412-97ce-9dca59ce2649-istio-podinfo\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:21.643927 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.641255    5570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26831c71-2091-4090-843d-f7268d257114-workload-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:21.643927 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.641269    5570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ee1819e-4649-4d3e-9cfc-0d2d72c1b481-workload-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:21.643927 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.641282    5570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce7b1d98-c21f-4412-97ce-9dca59ce2649-istio-envoy\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:21.643927 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.641295    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bbpvb\" (UniqueName: \"kubernetes.io/projected/4ee1819e-4649-4d3e-9cfc-0d2d72c1b481-kube-api-access-bbpvb\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:21.643927 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.641310    5570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6029825e-e953-4c43-9f4d-d19b5ca38623-credential-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:21.643927 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.641324    5570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26831c71-2091-4090-843d-f7268d257114-istio-token\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:21.643927 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.641338    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56hdq\" (UniqueName: \"kubernetes.io/projected/ce7b1d98-c21f-4412-97ce-9dca59ce2649-kube-api-access-56hdq\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:21.643927 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.641352    5570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6029825e-e953-4c43-9f4d-d19b5ca38623-workload-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:21.643927 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.641369    5570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6029825e-e953-4c43-9f4d-d19b5ca38623-istiod-ca-cert\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:21.643927 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.641383    5570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6029825e-e953-4c43-9f4d-d19b5ca38623-istio-data\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:21.643927 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.641397    5570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ee1819e-4649-4d3e-9cfc-0d2d72c1b481-credential-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:21.643927 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.641413    5570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26831c71-2091-4090-843d-f7268d257114-workload-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:21.643927 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.641429    5570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51c1be09-aa2e-47bd-a863-97f7daf94ddb-istio-data\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:21.643927 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.641442    5570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26831c71-2091-4090-843d-f7268d257114-credential-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:21.643927 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.641457    5570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce7b1d98-c21f-4412-97ce-9dca59ce2649-workload-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:21.643927 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.641475    5570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ee1819e-4649-4d3e-9cfc-0d2d72c1b481-istiod-ca-cert\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:21.643927 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.641489    5570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce7b1d98-c21f-4412-97ce-9dca59ce2649-istiod-ca-cert\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:21.643927 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.641505    5570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26831c71-2091-4090-843d-f7268d257114-istiod-ca-cert\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:21.643927 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.641519    5570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ee1819e-4649-4d3e-9cfc-0d2d72c1b481-istio-envoy\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:21.643927 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.641535    5570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6029825e-e953-4c43-9f4d-d19b5ca38623-istio-token\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:21.643927 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.641549    5570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51c1be09-aa2e-47bd-a863-97f7daf94ddb-credential-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:21.643927 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.641564    5570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ee1819e-4649-4d3e-9cfc-0d2d72c1b481-istio-data\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:21.643927 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.641809    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6029825e-e953-4c43-9f4d-d19b5ca38623-workload-socket" pod "6029825e-e953-4c43-9f4d-d19b5ca38623" (UID: "6029825e-e953-4c43-9f4d-d19b5ca38623"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:21.643927 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.641860    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51c1be09-aa2e-47bd-a863-97f7daf94ddb-workload-certs" pod "51c1be09-aa2e-47bd-a863-97f7daf94ddb" (UID: "51c1be09-aa2e-47bd-a863-97f7daf94ddb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:21.643927 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.642017    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ee1819e-4649-4d3e-9cfc-0d2d72c1b481-workload-socket" pod "4ee1819e-4649-4d3e-9cfc-0d2d72c1b481" (UID: "4ee1819e-4649-4d3e-9cfc-0d2d72c1b481"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:21.643927 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.642065    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce7b1d98-c21f-4412-97ce-9dca59ce2649-workload-certs" pod "ce7b1d98-c21f-4412-97ce-9dca59ce2649" (UID: "ce7b1d98-c21f-4412-97ce-9dca59ce2649"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:21.643927 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.642307    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51c1be09-aa2e-47bd-a863-97f7daf94ddb-istiod-ca-cert" pod "51c1be09-aa2e-47bd-a863-97f7daf94ddb" (UID: "51c1be09-aa2e-47bd-a863-97f7daf94ddb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:27:21.643927 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.642921    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/51c1be09-aa2e-47bd-a863-97f7daf94ddb-istio-podinfo" pod "51c1be09-aa2e-47bd-a863-97f7daf94ddb" (UID: "51c1be09-aa2e-47bd-a863-97f7daf94ddb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:27:21.643927 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.643499    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce7b1d98-c21f-4412-97ce-9dca59ce2649-credential-socket" pod "ce7b1d98-c21f-4412-97ce-9dca59ce2649" (UID: "ce7b1d98-c21f-4412-97ce-9dca59ce2649"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:21.643927 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.643541    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26831c71-2091-4090-843d-f7268d257114-istio-data" pod "26831c71-2091-4090-843d-f7268d257114" (UID: "26831c71-2091-4090-843d-f7268d257114"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:21.655179 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.644804    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51c1be09-aa2e-47bd-a863-97f7daf94ddb-istio-token" pod "51c1be09-aa2e-47bd-a863-97f7daf94ddb" (UID: "51c1be09-aa2e-47bd-a863-97f7daf94ddb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:27:21.655179 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.645989    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce7b1d98-c21f-4412-97ce-9dca59ce2649-istio-data" pod "ce7b1d98-c21f-4412-97ce-9dca59ce2649" (UID: "ce7b1d98-c21f-4412-97ce-9dca59ce2649"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:21.655179 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.646206    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6029825e-e953-4c43-9f4d-d19b5ca38623-istio-podinfo" pod "6029825e-e953-4c43-9f4d-d19b5ca38623" (UID: "6029825e-e953-4c43-9f4d-d19b5ca38623"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:27:21.655179 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.646855    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/26831c71-2091-4090-843d-f7268d257114-istio-podinfo" pod "26831c71-2091-4090-843d-f7268d257114" (UID: "26831c71-2091-4090-843d-f7268d257114"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:27:21.655179 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.647000    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51c1be09-aa2e-47bd-a863-97f7daf94ddb-istio-envoy" pod "51c1be09-aa2e-47bd-a863-97f7daf94ddb" (UID: "51c1be09-aa2e-47bd-a863-97f7daf94ddb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:21.655179 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.651195    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26831c71-2091-4090-843d-f7268d257114-istio-envoy" pod "26831c71-2091-4090-843d-f7268d257114" (UID: "26831c71-2091-4090-843d-f7268d257114"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:21.655179 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.651286    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26831c71-2091-4090-843d-f7268d257114-kube-api-access-9cljz" pod "26831c71-2091-4090-843d-f7268d257114" (UID: "26831c71-2091-4090-843d-f7268d257114"). InnerVolumeSpecName "kube-api-access-9cljz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:27:21.655179 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.651285    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6029825e-e953-4c43-9f4d-d19b5ca38623-istio-envoy" pod "6029825e-e953-4c43-9f4d-d19b5ca38623" (UID: "6029825e-e953-4c43-9f4d-d19b5ca38623"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:21.655179 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.651329    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ee1819e-4649-4d3e-9cfc-0d2d72c1b481-istio-token" pod "4ee1819e-4649-4d3e-9cfc-0d2d72c1b481" (UID: "4ee1819e-4649-4d3e-9cfc-0d2d72c1b481"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:27:21.655179 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.651383    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6029825e-e953-4c43-9f4d-d19b5ca38623-kube-api-access-ff2pz" pod "6029825e-e953-4c43-9f4d-d19b5ca38623" (UID: "6029825e-e953-4c43-9f4d-d19b5ca38623"). InnerVolumeSpecName "kube-api-access-ff2pz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:27:21.655878 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.655849    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51c1be09-aa2e-47bd-a863-97f7daf94ddb-kube-api-access-sc86d" pod "51c1be09-aa2e-47bd-a863-97f7daf94ddb" (UID: "51c1be09-aa2e-47bd-a863-97f7daf94ddb"). InnerVolumeSpecName "kube-api-access-sc86d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:27:21.743701 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.743661    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sc86d\" (UniqueName: \"kubernetes.io/projected/51c1be09-aa2e-47bd-a863-97f7daf94ddb-kube-api-access-sc86d\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:21.743701 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.743696    5570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26831c71-2091-4090-843d-f7268d257114-istio-data\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:21.743958 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.743712    5570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce7b1d98-c21f-4412-97ce-9dca59ce2649-credential-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:21.743958 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.743726    5570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce7b1d98-c21f-4412-97ce-9dca59ce2649-istio-token\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:21.743958 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.743741    5570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce7b1d98-c21f-4412-97ce-9dca59ce2649-workload-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:21.743958 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.743753    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ff2pz\" (UniqueName: \"kubernetes.io/projected/6029825e-e953-4c43-9f4d-d19b5ca38623-kube-api-access-ff2pz\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:21.743958 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.743768    5570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26831c71-2091-4090-843d-f7268d257114-istio-envoy\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:21.743958 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.743812    5570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51c1be09-aa2e-47bd-a863-97f7daf94ddb-workload-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:21.743958 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.743825    5570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6029825e-e953-4c43-9f4d-d19b5ca38623-workload-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:21.743958 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.743837    5570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ee1819e-4649-4d3e-9cfc-0d2d72c1b481-workload-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:21.743958 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.743850    5570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51c1be09-aa2e-47bd-a863-97f7daf94ddb-istiod-ca-cert\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:21.743958 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.743864    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9cljz\" (UniqueName: \"kubernetes.io/projected/26831c71-2091-4090-843d-f7268d257114-kube-api-access-9cljz\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:21.743958 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.743877    5570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26831c71-2091-4090-843d-f7268d257114-istio-podinfo\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:21.743958 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.743889    5570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51c1be09-aa2e-47bd-a863-97f7daf94ddb-istio-envoy\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:21.743958 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.743902    5570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6029825e-e953-4c43-9f4d-d19b5ca38623-istio-envoy\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:21.743958 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.743915    5570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6029825e-e953-4c43-9f4d-d19b5ca38623-istio-podinfo\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:21.743958 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.743927    5570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51c1be09-aa2e-47bd-a863-97f7daf94ddb-istio-token\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:21.743958 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.743939    5570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ee1819e-4649-4d3e-9cfc-0d2d72c1b481-istio-token\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:21.743958 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.743950    5570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51c1be09-aa2e-47bd-a863-97f7daf94ddb-istio-podinfo\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:21.743958 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.743964    5570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce7b1d98-c21f-4412-97ce-9dca59ce2649-istio-data\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:21.895215 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.895117    5570 generic.go:356] "Generic (PLEG): container finished" podID="6029825e-e953-4c43-9f4d-d19b5ca38623" containerID="61df14cbe40ac50825e634951cd7b5b5dbc121f9048dfffa32676dd32719d0b9" exitCode=0
Nov 03 20:27:21.895215 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.895189    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-5c82t"
Nov 03 20:27:21.895439 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.895203    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-5c82t" event={"ID":"6029825e-e953-4c43-9f4d-d19b5ca38623","Type":"ContainerDied","Data":"61df14cbe40ac50825e634951cd7b5b5dbc121f9048dfffa32676dd32719d0b9"}
Nov 03 20:27:21.895439 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.895287    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-5c82t" event={"ID":"6029825e-e953-4c43-9f4d-d19b5ca38623","Type":"ContainerDied","Data":"d039eb5c206efbf9b02b55250cc200257402319e89c62cbae7cb9aa8dd768afa"}
Nov 03 20:27:21.895439 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.895343    5570 scope.go:122] "RemoveContainer" containerID="61df14cbe40ac50825e634951cd7b5b5dbc121f9048dfffa32676dd32719d0b9"
Nov 03 20:27:21.897705 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.897679    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-wj4gp" event={"ID":"ce7b1d98-c21f-4412-97ce-9dca59ce2649","Type":"ContainerDied","Data":"8e857f2889f0ace42cbd1f2f2fb3080b52dd87372ddeb2e190b2666a84690a0e"}
Nov 03 20:27:21.898049 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.898009    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-wj4gp"
Nov 03 20:27:21.907661 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.907628    5570 generic.go:356] "Generic (PLEG): container finished" podID="51c1be09-aa2e-47bd-a863-97f7daf94ddb" containerID="8c8ff7ee0e824dae78ba0ae02863133dd5b8d0ddb46392cb8f4984679b96ed86" exitCode=0
Nov 03 20:27:21.908000 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.907973    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-fqjnh" event={"ID":"51c1be09-aa2e-47bd-a863-97f7daf94ddb","Type":"ContainerDied","Data":"8c8ff7ee0e824dae78ba0ae02863133dd5b8d0ddb46392cb8f4984679b96ed86"}
Nov 03 20:27:21.908165 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.908144    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-fqjnh" event={"ID":"51c1be09-aa2e-47bd-a863-97f7daf94ddb","Type":"ContainerDied","Data":"5fa4559cb10ba37ca774bae9818852783302bb23de4a2c2d4fd364e33441cbff"}
Nov 03 20:27:21.908369 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.908351    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-fqjnh"
Nov 03 20:27:21.922812 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.920336    5570 scope.go:122] "RemoveContainer" containerID="61df14cbe40ac50825e634951cd7b5b5dbc121f9048dfffa32676dd32719d0b9"
Nov 03 20:27:21.923839 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:27:21.923798    5570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61df14cbe40ac50825e634951cd7b5b5dbc121f9048dfffa32676dd32719d0b9\": container with ID starting with 61df14cbe40ac50825e634951cd7b5b5dbc121f9048dfffa32676dd32719d0b9 not found: ID does not exist" containerID="61df14cbe40ac50825e634951cd7b5b5dbc121f9048dfffa32676dd32719d0b9"
Nov 03 20:27:21.923977 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.923853    5570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61df14cbe40ac50825e634951cd7b5b5dbc121f9048dfffa32676dd32719d0b9"} err="failed to get container status \"61df14cbe40ac50825e634951cd7b5b5dbc121f9048dfffa32676dd32719d0b9\": rpc error: code = NotFound desc = could not find container \"61df14cbe40ac50825e634951cd7b5b5dbc121f9048dfffa32676dd32719d0b9\": container with ID starting with 61df14cbe40ac50825e634951cd7b5b5dbc121f9048dfffa32676dd32719d0b9 not found: ID does not exist"
Nov 03 20:27:21.923977 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.923881    5570 scope.go:122] "RemoveContainer" containerID="2065ee88578f7048629b049802360f906227df9654abe61d45dbb67e25165dbb"
Nov 03 20:27:21.929370 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.928581    5570 generic.go:356] "Generic (PLEG): container finished" podID="4ee1819e-4649-4d3e-9cfc-0d2d72c1b481" containerID="c70dfe54e707de3f4706134c4a875053c59da1192e0ce8ce4a82e0c984572532" exitCode=0
Nov 03 20:27:21.929370 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.928910    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-xszg4" event={"ID":"4ee1819e-4649-4d3e-9cfc-0d2d72c1b481","Type":"ContainerDied","Data":"c70dfe54e707de3f4706134c4a875053c59da1192e0ce8ce4a82e0c984572532"}
Nov 03 20:27:21.929370 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.929022    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-xszg4" event={"ID":"4ee1819e-4649-4d3e-9cfc-0d2d72c1b481","Type":"ContainerDied","Data":"3295b90f37b38c58c3dac7d240a875b80fc8cb0d376f52bad2d2dfc01a7093f1"}
Nov 03 20:27:21.929370 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.929102    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-xszg4"
Nov 03 20:27:21.935572 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.935546    5570 generic.go:356] "Generic (PLEG): container finished" podID="26831c71-2091-4090-843d-f7268d257114" containerID="50869a3fc874fdc6908769230eea560a48902e803a7a881735d87e7285089421" exitCode=0
Nov 03 20:27:21.935680 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.935609    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-5bn7t"
Nov 03 20:27:21.935680 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.935650    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-5bn7t" event={"ID":"26831c71-2091-4090-843d-f7268d257114","Type":"ContainerDied","Data":"50869a3fc874fdc6908769230eea560a48902e803a7a881735d87e7285089421"}
Nov 03 20:27:21.935835 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.935692    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-5bn7t" event={"ID":"26831c71-2091-4090-843d-f7268d257114","Type":"ContainerDied","Data":"6f18c125bb7b96723a2a1230ebd605a97e671aa34b786080797ff7650238c5f4"}
Nov 03 20:27:21.944049 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.943922    5570 scope.go:122] "RemoveContainer" containerID="8c8ff7ee0e824dae78ba0ae02863133dd5b8d0ddb46392cb8f4984679b96ed86"
Nov 03 20:27:21.951851 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.948561    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-9gvtf"
Nov 03 20:27:21.951851 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.948586    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-9gvtf" event={"ID":"5e7f17ab-0f1d-4a59-b415-fe844ae7c1e7","Type":"ContainerDied","Data":"81ea73746ad1a8b753bc3e054f74a6e5546fcc93e882a572e9488bed4c2fc2f6"}
Nov 03 20:27:21.953928 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.953900    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-wg864" event={"ID":"6a1b3d29-b647-46e1-9005-e540e937f85f","Type":"ContainerDied","Data":"10f3d7ff68442da2d65b5be8d4a9ecdf55afe10edfb109b2e2c967653062a859"}
Nov 03 20:27:21.953928 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.953914    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-wg864"
Nov 03 20:27:21.955572 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.955548    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-jbtqd" event={"ID":"baacee7f-b912-4964-9855-be2beb0094cb","Type":"ContainerDied","Data":"b48d415fd6952fb133ce8433e58075bd9c80126def429374aa8d77c457723a20"}
Nov 03 20:27:21.956353 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.956334    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-jbtqd"
Nov 03 20:27:21.959748 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.959729    5570 scope.go:122] "RemoveContainer" containerID="8c8ff7ee0e824dae78ba0ae02863133dd5b8d0ddb46392cb8f4984679b96ed86"
Nov 03 20:27:21.960097 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:27:21.960072    5570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c8ff7ee0e824dae78ba0ae02863133dd5b8d0ddb46392cb8f4984679b96ed86\": container with ID starting with 8c8ff7ee0e824dae78ba0ae02863133dd5b8d0ddb46392cb8f4984679b96ed86 not found: ID does not exist" containerID="8c8ff7ee0e824dae78ba0ae02863133dd5b8d0ddb46392cb8f4984679b96ed86"
Nov 03 20:27:21.960171 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.960107    5570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c8ff7ee0e824dae78ba0ae02863133dd5b8d0ddb46392cb8f4984679b96ed86"} err="failed to get container status \"8c8ff7ee0e824dae78ba0ae02863133dd5b8d0ddb46392cb8f4984679b96ed86\": rpc error: code = NotFound desc = could not find container \"8c8ff7ee0e824dae78ba0ae02863133dd5b8d0ddb46392cb8f4984679b96ed86\": container with ID starting with 8c8ff7ee0e824dae78ba0ae02863133dd5b8d0ddb46392cb8f4984679b96ed86 not found: ID does not exist"
Nov 03 20:27:21.960171 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.960127    5570 scope.go:122] "RemoveContainer" containerID="c70dfe54e707de3f4706134c4a875053c59da1192e0ce8ce4a82e0c984572532"
Nov 03 20:27:21.970598 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.970570    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55945440-d40e-4235-8b84-d1e2a013372d" path="/var/lib/kubelet/pods/55945440-d40e-4235-8b84-d1e2a013372d/volumes"
Nov 03 20:27:21.971281 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.971259    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63727774-9118-4b83-bede-04208e8d59b9" path="/var/lib/kubelet/pods/63727774-9118-4b83-bede-04208e8d59b9/volumes"
Nov 03 20:27:21.971879 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.971860    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1e2eadf-ede0-4ea2-9eea-f25a39779795" path="/var/lib/kubelet/pods/c1e2eadf-ede0-4ea2-9eea-f25a39779795/volumes"
Nov 03 20:27:21.975881 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.975861    5570 scope.go:122] "RemoveContainer" containerID="c70dfe54e707de3f4706134c4a875053c59da1192e0ce8ce4a82e0c984572532"
Nov 03 20:27:21.978146 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:27:21.978118    5570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c70dfe54e707de3f4706134c4a875053c59da1192e0ce8ce4a82e0c984572532\": container with ID starting with c70dfe54e707de3f4706134c4a875053c59da1192e0ce8ce4a82e0c984572532 not found: ID does not exist" containerID="c70dfe54e707de3f4706134c4a875053c59da1192e0ce8ce4a82e0c984572532"
Nov 03 20:27:21.978266 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.978157    5570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c70dfe54e707de3f4706134c4a875053c59da1192e0ce8ce4a82e0c984572532"} err="failed to get container status \"c70dfe54e707de3f4706134c4a875053c59da1192e0ce8ce4a82e0c984572532\": rpc error: code = NotFound desc = could not find container \"c70dfe54e707de3f4706134c4a875053c59da1192e0ce8ce4a82e0c984572532\": container with ID starting with c70dfe54e707de3f4706134c4a875053c59da1192e0ce8ce4a82e0c984572532 not found: ID does not exist"
Nov 03 20:27:21.978266 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:21.978180    5570 scope.go:122] "RemoveContainer" containerID="50869a3fc874fdc6908769230eea560a48902e803a7a881735d87e7285089421"
Nov 03 20:27:22.008000 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:22.007966    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-xszg4"]
Nov 03 20:27:22.013521 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:22.013492    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-xszg4"]
Nov 03 20:27:22.018169 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:22.018151    5570 scope.go:122] "RemoveContainer" containerID="50869a3fc874fdc6908769230eea560a48902e803a7a881735d87e7285089421"
Nov 03 20:27:22.018503 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:27:22.018481    5570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50869a3fc874fdc6908769230eea560a48902e803a7a881735d87e7285089421\": container with ID starting with 50869a3fc874fdc6908769230eea560a48902e803a7a881735d87e7285089421 not found: ID does not exist" containerID="50869a3fc874fdc6908769230eea560a48902e803a7a881735d87e7285089421"
Nov 03 20:27:22.018584 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:22.018510    5570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50869a3fc874fdc6908769230eea560a48902e803a7a881735d87e7285089421"} err="failed to get container status \"50869a3fc874fdc6908769230eea560a48902e803a7a881735d87e7285089421\": rpc error: code = NotFound desc = could not find container \"50869a3fc874fdc6908769230eea560a48902e803a7a881735d87e7285089421\": container with ID starting with 50869a3fc874fdc6908769230eea560a48902e803a7a881735d87e7285089421 not found: ID does not exist"
Nov 03 20:27:22.018584 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:22.018528    5570 scope.go:122] "RemoveContainer" containerID="3e7c2142d5118a937b9c4c1db78402b22ba93d6da618fb6dea1c3717ae2ab38b"
Nov 03 20:27:22.056875 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:22.056844    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-5bn7t"]
Nov 03 20:27:22.061391 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:22.061361    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-5bn7t"]
Nov 03 20:27:22.072688 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:22.072670    5570 scope.go:122] "RemoveContainer" containerID="c112fdc1a8a3d21c59453de2ff24628f1b064fd91b46b2f5e53085450bd216c0"
Nov 03 20:27:22.084986 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:22.084969    5570 scope.go:122] "RemoveContainer" containerID="f88e91ce1e2d9b540bdd3ec4f7167d68046dc7bc6e871d1e1f9efc08d87c6965"
Nov 03 20:27:22.118434 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:22.118395    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-5c82t"]
Nov 03 20:27:22.124750 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:22.124716    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-5c82t"]
Nov 03 20:27:22.162100 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:22.162005    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-fqjnh"]
Nov 03 20:27:22.168745 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:22.168713    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-fqjnh"]
Nov 03 20:27:22.199903 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:22.199875    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-wj4gp"]
Nov 03 20:27:22.203712 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:22.203690    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-wj4gp"]
Nov 03 20:27:22.239443 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:22.239412    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-9gvtf"]
Nov 03 20:27:22.246093 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:22.246064    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-9gvtf"]
Nov 03 20:27:22.281548 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:22.281489    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-wg864"]
Nov 03 20:27:22.286635 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:22.286604    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-wg864"]
Nov 03 20:27:22.323429 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:22.323395    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-jbtqd"]
Nov 03 20:27:22.331477 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:22.331443    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-jbtqd"]
Nov 03 20:27:22.564122 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:22.564093    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4cdbeb-7c468dc747-9kqqx"
Nov 03 20:27:22.988027 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:22.987762    5570 generic.go:356] "Generic (PLEG): container finished" podID="e314286e-4b2d-4fae-bc59-528809a8dbce" containerID="39dce94b7e35773069c096fb3e6dee58c6fc5bf5e34a8cbdbb93fa453abea1cc" exitCode=0
Nov 03 20:27:22.988027 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:22.987943    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-wkmg8" event={"ID":"e314286e-4b2d-4fae-bc59-528809a8dbce","Type":"ContainerDied","Data":"39dce94b7e35773069c096fb3e6dee58c6fc5bf5e34a8cbdbb93fa453abea1cc"}
Nov 03 20:27:22.996305 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:22.996259    5570 generic.go:356] "Generic (PLEG): container finished" podID="3c3d9f2d-dcbb-4fb3-a54f-ed9cdef4118e" containerID="5cfb10265f083a5dc0871a30cf040733f11efc2fb138b8ba9e9ecd8ad963fc77" exitCode=0
Nov 03 20:27:22.996527 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:22.996357    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-7458z" event={"ID":"3c3d9f2d-dcbb-4fb3-a54f-ed9cdef4118e","Type":"ContainerDied","Data":"5cfb10265f083a5dc0871a30cf040733f11efc2fb138b8ba9e9ecd8ad963fc77"}
Nov 03 20:27:23.001615 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:23.001568    5570 generic.go:356] "Generic (PLEG): container finished" podID="157f80a0-a704-4bed-b544-205c3f2a52b3" containerID="73be03f9649446acd3f30a87e3b60fbb7889d073564dcf2859c9ed6c410dcaf5" exitCode=0
Nov 03 20:27:23.001615 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:23.001619    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmkq7" event={"ID":"157f80a0-a704-4bed-b544-205c3f2a52b3","Type":"ContainerDied","Data":"73be03f9649446acd3f30a87e3b60fbb7889d073564dcf2859c9ed6c410dcaf5"}
Nov 03 20:27:23.123271 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:23.123247    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmkq7"
Nov 03 20:27:23.212064 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:23.212035    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-wkmg8"
Nov 03 20:27:23.222531 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:23.222504    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-7458z"
Nov 03 20:27:23.275758 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:23.275723    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/157f80a0-a704-4bed-b544-205c3f2a52b3-kube-api-access-z8npn\" (UniqueName: \"kubernetes.io/projected/157f80a0-a704-4bed-b544-205c3f2a52b3-kube-api-access-z8npn\") pod \"157f80a0-a704-4bed-b544-205c3f2a52b3\" (UID: \"157f80a0-a704-4bed-b544-205c3f2a52b3\") "
Nov 03 20:27:23.276093 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:23.276065    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/157f80a0-a704-4bed-b544-205c3f2a52b3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/157f80a0-a704-4bed-b544-205c3f2a52b3-workload-socket\") pod \"157f80a0-a704-4bed-b544-205c3f2a52b3\" (UID: \"157f80a0-a704-4bed-b544-205c3f2a52b3\") "
Nov 03 20:27:23.276307 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:23.276151    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/157f80a0-a704-4bed-b544-205c3f2a52b3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/157f80a0-a704-4bed-b544-205c3f2a52b3-credential-socket\") pod \"157f80a0-a704-4bed-b544-205c3f2a52b3\" (UID: \"157f80a0-a704-4bed-b544-205c3f2a52b3\") "
Nov 03 20:27:23.276307 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:23.276272    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e314286e-4b2d-4fae-bc59-528809a8dbce-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e314286e-4b2d-4fae-bc59-528809a8dbce-workload-certs\") pod \"e314286e-4b2d-4fae-bc59-528809a8dbce\" (UID: \"e314286e-4b2d-4fae-bc59-528809a8dbce\") "
Nov 03 20:27:23.276443 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:23.276322    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/157f80a0-a704-4bed-b544-205c3f2a52b3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/157f80a0-a704-4bed-b544-205c3f2a52b3-istiod-ca-cert\") pod \"157f80a0-a704-4bed-b544-205c3f2a52b3\" (UID: \"157f80a0-a704-4bed-b544-205c3f2a52b3\") "
Nov 03 20:27:23.276443 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:23.276349    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/157f80a0-a704-4bed-b544-205c3f2a52b3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/157f80a0-a704-4bed-b544-205c3f2a52b3-workload-certs\") pod \"157f80a0-a704-4bed-b544-205c3f2a52b3\" (UID: \"157f80a0-a704-4bed-b544-205c3f2a52b3\") "
Nov 03 20:27:23.276659 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:23.276611    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e314286e-4b2d-4fae-bc59-528809a8dbce-workload-certs" pod "e314286e-4b2d-4fae-bc59-528809a8dbce" (UID: "e314286e-4b2d-4fae-bc59-528809a8dbce"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:23.276810 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:23.276653    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/157f80a0-a704-4bed-b544-205c3f2a52b3-workload-socket" pod "157f80a0-a704-4bed-b544-205c3f2a52b3" (UID: "157f80a0-a704-4bed-b544-205c3f2a52b3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:23.276913 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:23.276887    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/157f80a0-a704-4bed-b544-205c3f2a52b3-istiod-ca-cert" pod "157f80a0-a704-4bed-b544-205c3f2a52b3" (UID: "157f80a0-a704-4bed-b544-205c3f2a52b3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:27:23.277012 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:23.276992    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/157f80a0-a704-4bed-b544-205c3f2a52b3-credential-socket" pod "157f80a0-a704-4bed-b544-205c3f2a52b3" (UID: "157f80a0-a704-4bed-b544-205c3f2a52b3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:23.277046 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:23.277012    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/157f80a0-a704-4bed-b544-205c3f2a52b3-workload-certs" pod "157f80a0-a704-4bed-b544-205c3f2a52b3" (UID: "157f80a0-a704-4bed-b544-205c3f2a52b3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:23.277289 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:23.277271    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e314286e-4b2d-4fae-bc59-528809a8dbce-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e314286e-4b2d-4fae-bc59-528809a8dbce-istio-envoy\") pod \"e314286e-4b2d-4fae-bc59-528809a8dbce\" (UID: \"e314286e-4b2d-4fae-bc59-528809a8dbce\") "
Nov 03 20:27:23.277393 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:23.277377    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e314286e-4b2d-4fae-bc59-528809a8dbce-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e314286e-4b2d-4fae-bc59-528809a8dbce-workload-socket\") pod \"e314286e-4b2d-4fae-bc59-528809a8dbce\" (UID: \"e314286e-4b2d-4fae-bc59-528809a8dbce\") "
Nov 03 20:27:23.277462 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:23.277447    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e314286e-4b2d-4fae-bc59-528809a8dbce-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e314286e-4b2d-4fae-bc59-528809a8dbce-credential-socket\") pod \"e314286e-4b2d-4fae-bc59-528809a8dbce\" (UID: \"e314286e-4b2d-4fae-bc59-528809a8dbce\") "
Nov 03 20:27:23.277528 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:23.277513    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/157f80a0-a704-4bed-b544-205c3f2a52b3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/157f80a0-a704-4bed-b544-205c3f2a52b3-istio-podinfo\") pod \"157f80a0-a704-4bed-b544-205c3f2a52b3\" (UID: \"157f80a0-a704-4bed-b544-205c3f2a52b3\") "
Nov 03 20:27:23.277579 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:23.277549    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/157f80a0-a704-4bed-b544-205c3f2a52b3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/157f80a0-a704-4bed-b544-205c3f2a52b3-istio-envoy\") pod \"157f80a0-a704-4bed-b544-205c3f2a52b3\" (UID: \"157f80a0-a704-4bed-b544-205c3f2a52b3\") "
Nov 03 20:27:23.277633 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:23.277595    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e314286e-4b2d-4fae-bc59-528809a8dbce-kube-api-access-2qpw8\" (UniqueName: \"kubernetes.io/projected/e314286e-4b2d-4fae-bc59-528809a8dbce-kube-api-access-2qpw8\") pod \"e314286e-4b2d-4fae-bc59-528809a8dbce\" (UID: \"e314286e-4b2d-4fae-bc59-528809a8dbce\") "
Nov 03 20:27:23.277687 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:23.277658    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/157f80a0-a704-4bed-b544-205c3f2a52b3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/157f80a0-a704-4bed-b544-205c3f2a52b3-istio-data\") pod \"157f80a0-a704-4bed-b544-205c3f2a52b3\" (UID: \"157f80a0-a704-4bed-b544-205c3f2a52b3\") "
Nov 03 20:27:23.277737 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:23.277720    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/157f80a0-a704-4bed-b544-205c3f2a52b3-istio-token\" (UniqueName: \"kubernetes.io/projected/157f80a0-a704-4bed-b544-205c3f2a52b3-istio-token\") pod \"157f80a0-a704-4bed-b544-205c3f2a52b3\" (UID: \"157f80a0-a704-4bed-b544-205c3f2a52b3\") "
Nov 03 20:27:23.277847 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:23.277830    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e314286e-4b2d-4fae-bc59-528809a8dbce-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e314286e-4b2d-4fae-bc59-528809a8dbce-istiod-ca-cert\") pod \"e314286e-4b2d-4fae-bc59-528809a8dbce\" (UID: \"e314286e-4b2d-4fae-bc59-528809a8dbce\") "
Nov 03 20:27:23.277917 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:23.277869    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e314286e-4b2d-4fae-bc59-528809a8dbce-istio-token\" (UniqueName: \"kubernetes.io/projected/e314286e-4b2d-4fae-bc59-528809a8dbce-istio-token\") pod \"e314286e-4b2d-4fae-bc59-528809a8dbce\" (UID: \"e314286e-4b2d-4fae-bc59-528809a8dbce\") "
Nov 03 20:27:23.277971 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:23.277939    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e314286e-4b2d-4fae-bc59-528809a8dbce-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e314286e-4b2d-4fae-bc59-528809a8dbce-istio-data\") pod \"e314286e-4b2d-4fae-bc59-528809a8dbce\" (UID: \"e314286e-4b2d-4fae-bc59-528809a8dbce\") "
Nov 03 20:27:23.277971 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:23.277951    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e314286e-4b2d-4fae-bc59-528809a8dbce-credential-socket" pod "e314286e-4b2d-4fae-bc59-528809a8dbce" (UID: "e314286e-4b2d-4fae-bc59-528809a8dbce"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:23.279000 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:23.277974    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e314286e-4b2d-4fae-bc59-528809a8dbce-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e314286e-4b2d-4fae-bc59-528809a8dbce-istio-podinfo\") pod \"e314286e-4b2d-4fae-bc59-528809a8dbce\" (UID: \"e314286e-4b2d-4fae-bc59-528809a8dbce\") "
Nov 03 20:27:23.279000 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:23.278065    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e314286e-4b2d-4fae-bc59-528809a8dbce-workload-socket" pod "e314286e-4b2d-4fae-bc59-528809a8dbce" (UID: "e314286e-4b2d-4fae-bc59-528809a8dbce"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:23.279000 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:23.278621    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e314286e-4b2d-4fae-bc59-528809a8dbce-istio-data" pod "e314286e-4b2d-4fae-bc59-528809a8dbce" (UID: "e314286e-4b2d-4fae-bc59-528809a8dbce"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:23.279000 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:23.278934    5570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/157f80a0-a704-4bed-b544-205c3f2a52b3-workload-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:23.279000 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:23.278956    5570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/157f80a0-a704-4bed-b544-205c3f2a52b3-credential-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:23.279000 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:23.278970    5570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e314286e-4b2d-4fae-bc59-528809a8dbce-workload-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:23.279000 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:23.278985    5570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/157f80a0-a704-4bed-b544-205c3f2a52b3-istiod-ca-cert\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:23.279000 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:23.278997    5570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/157f80a0-a704-4bed-b544-205c3f2a52b3-workload-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:23.279414 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:23.279010    5570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e314286e-4b2d-4fae-bc59-528809a8dbce-workload-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:23.279414 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:23.279023    5570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e314286e-4b2d-4fae-bc59-528809a8dbce-credential-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:23.279414 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:23.279036    5570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e314286e-4b2d-4fae-bc59-528809a8dbce-istio-data\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:23.286633 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:23.280027    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e314286e-4b2d-4fae-bc59-528809a8dbce-istiod-ca-cert" pod "e314286e-4b2d-4fae-bc59-528809a8dbce" (UID: "e314286e-4b2d-4fae-bc59-528809a8dbce"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:27:23.286633 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:23.280342    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/157f80a0-a704-4bed-b544-205c3f2a52b3-istio-data" pod "157f80a0-a704-4bed-b544-205c3f2a52b3" (UID: "157f80a0-a704-4bed-b544-205c3f2a52b3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:23.286633 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:23.284256    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e314286e-4b2d-4fae-bc59-528809a8dbce-istio-envoy" pod "e314286e-4b2d-4fae-bc59-528809a8dbce" (UID: "e314286e-4b2d-4fae-bc59-528809a8dbce"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:23.286633 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:23.284464    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/157f80a0-a704-4bed-b544-205c3f2a52b3-kube-api-access-z8npn" pod "157f80a0-a704-4bed-b544-205c3f2a52b3" (UID: "157f80a0-a704-4bed-b544-205c3f2a52b3"). InnerVolumeSpecName "kube-api-access-z8npn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:27:23.286633 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:23.286049    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e314286e-4b2d-4fae-bc59-528809a8dbce-istio-podinfo" pod "e314286e-4b2d-4fae-bc59-528809a8dbce" (UID: "e314286e-4b2d-4fae-bc59-528809a8dbce"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:27:23.286633 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:23.286250    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e314286e-4b2d-4fae-bc59-528809a8dbce-kube-api-access-2qpw8" pod "e314286e-4b2d-4fae-bc59-528809a8dbce" (UID: "e314286e-4b2d-4fae-bc59-528809a8dbce"). InnerVolumeSpecName "kube-api-access-2qpw8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:27:23.286633 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:23.286450    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/157f80a0-a704-4bed-b544-205c3f2a52b3-istio-envoy" pod "157f80a0-a704-4bed-b544-205c3f2a52b3" (UID: "157f80a0-a704-4bed-b544-205c3f2a52b3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:23.287652 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:23.286862    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/157f80a0-a704-4bed-b544-205c3f2a52b3-istio-podinfo" pod "157f80a0-a704-4bed-b544-205c3f2a52b3" (UID: "157f80a0-a704-4bed-b544-205c3f2a52b3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:27:23.287652 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:23.287008    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e314286e-4b2d-4fae-bc59-528809a8dbce-istio-token" pod "e314286e-4b2d-4fae-bc59-528809a8dbce" (UID: "e314286e-4b2d-4fae-bc59-528809a8dbce"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:27:23.287652 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:23.287095    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/157f80a0-a704-4bed-b544-205c3f2a52b3-istio-token" pod "157f80a0-a704-4bed-b544-205c3f2a52b3" (UID: "157f80a0-a704-4bed-b544-205c3f2a52b3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:27:23.380193 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:23.380070    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c3d9f2d-dcbb-4fb3-a54f-ed9cdef4118e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c3d9f2d-dcbb-4fb3-a54f-ed9cdef4118e-workload-socket\") pod \"3c3d9f2d-dcbb-4fb3-a54f-ed9cdef4118e\" (UID: \"3c3d9f2d-dcbb-4fb3-a54f-ed9cdef4118e\") "
Nov 03 20:27:23.380193 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:23.380137    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c3d9f2d-dcbb-4fb3-a54f-ed9cdef4118e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c3d9f2d-dcbb-4fb3-a54f-ed9cdef4118e-workload-certs\") pod \"3c3d9f2d-dcbb-4fb3-a54f-ed9cdef4118e\" (UID: \"3c3d9f2d-dcbb-4fb3-a54f-ed9cdef4118e\") "
Nov 03 20:27:23.380193 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:23.380185    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c3d9f2d-dcbb-4fb3-a54f-ed9cdef4118e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c3d9f2d-dcbb-4fb3-a54f-ed9cdef4118e-istiod-ca-cert\") pod \"3c3d9f2d-dcbb-4fb3-a54f-ed9cdef4118e\" (UID: \"3c3d9f2d-dcbb-4fb3-a54f-ed9cdef4118e\") "
Nov 03 20:27:23.380638 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:23.380215    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c3d9f2d-dcbb-4fb3-a54f-ed9cdef4118e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c3d9f2d-dcbb-4fb3-a54f-ed9cdef4118e-credential-socket\") pod \"3c3d9f2d-dcbb-4fb3-a54f-ed9cdef4118e\" (UID: \"3c3d9f2d-dcbb-4fb3-a54f-ed9cdef4118e\") "
Nov 03 20:27:23.380638 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:23.380250    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c3d9f2d-dcbb-4fb3-a54f-ed9cdef4118e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c3d9f2d-dcbb-4fb3-a54f-ed9cdef4118e-istio-envoy\") pod \"3c3d9f2d-dcbb-4fb3-a54f-ed9cdef4118e\" (UID: \"3c3d9f2d-dcbb-4fb3-a54f-ed9cdef4118e\") "
Nov 03 20:27:23.380638 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:23.380414    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c3d9f2d-dcbb-4fb3-a54f-ed9cdef4118e-workload-socket" pod "3c3d9f2d-dcbb-4fb3-a54f-ed9cdef4118e" (UID: "3c3d9f2d-dcbb-4fb3-a54f-ed9cdef4118e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:23.380857 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:23.380722    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c3d9f2d-dcbb-4fb3-a54f-ed9cdef4118e-istiod-ca-cert" pod "3c3d9f2d-dcbb-4fb3-a54f-ed9cdef4118e" (UID: "3c3d9f2d-dcbb-4fb3-a54f-ed9cdef4118e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:27:23.381014 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:23.380971    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c3d9f2d-dcbb-4fb3-a54f-ed9cdef4118e-credential-socket" pod "3c3d9f2d-dcbb-4fb3-a54f-ed9cdef4118e" (UID: "3c3d9f2d-dcbb-4fb3-a54f-ed9cdef4118e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:23.381014 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:23.380992    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c3d9f2d-dcbb-4fb3-a54f-ed9cdef4118e-workload-certs" pod "3c3d9f2d-dcbb-4fb3-a54f-ed9cdef4118e" (UID: "3c3d9f2d-dcbb-4fb3-a54f-ed9cdef4118e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:23.381271 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:23.381253    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c3d9f2d-dcbb-4fb3-a54f-ed9cdef4118e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c3d9f2d-dcbb-4fb3-a54f-ed9cdef4118e-istio-data\") pod \"3c3d9f2d-dcbb-4fb3-a54f-ed9cdef4118e\" (UID: \"3c3d9f2d-dcbb-4fb3-a54f-ed9cdef4118e\") "
Nov 03 20:27:23.381386 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:23.381371    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3c3d9f2d-dcbb-4fb3-a54f-ed9cdef4118e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c3d9f2d-dcbb-4fb3-a54f-ed9cdef4118e-istio-podinfo\") pod \"3c3d9f2d-dcbb-4fb3-a54f-ed9cdef4118e\" (UID: \"3c3d9f2d-dcbb-4fb3-a54f-ed9cdef4118e\") "
Nov 03 20:27:23.381528 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:23.381474    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c3d9f2d-dcbb-4fb3-a54f-ed9cdef4118e-kube-api-access-8wfct\" (UniqueName: \"kubernetes.io/projected/3c3d9f2d-dcbb-4fb3-a54f-ed9cdef4118e-kube-api-access-8wfct\") pod \"3c3d9f2d-dcbb-4fb3-a54f-ed9cdef4118e\" (UID: \"3c3d9f2d-dcbb-4fb3-a54f-ed9cdef4118e\") "
Nov 03 20:27:23.381749 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:23.381623    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c3d9f2d-dcbb-4fb3-a54f-ed9cdef4118e-istio-token\" (UniqueName: \"kubernetes.io/projected/3c3d9f2d-dcbb-4fb3-a54f-ed9cdef4118e-istio-token\") pod \"3c3d9f2d-dcbb-4fb3-a54f-ed9cdef4118e\" (UID: \"3c3d9f2d-dcbb-4fb3-a54f-ed9cdef4118e\") "
Nov 03 20:27:23.382473 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:23.381601    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c3d9f2d-dcbb-4fb3-a54f-ed9cdef4118e-istio-data" pod "3c3d9f2d-dcbb-4fb3-a54f-ed9cdef4118e" (UID: "3c3d9f2d-dcbb-4fb3-a54f-ed9cdef4118e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:23.382701 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:23.382672    5570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e314286e-4b2d-4fae-bc59-528809a8dbce-istio-podinfo\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:23.382787 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:23.382705    5570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c3d9f2d-dcbb-4fb3-a54f-ed9cdef4118e-workload-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:23.382787 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:23.382733    5570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c3d9f2d-dcbb-4fb3-a54f-ed9cdef4118e-workload-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:23.382787 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:23.382747    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8npn\" (UniqueName: \"kubernetes.io/projected/157f80a0-a704-4bed-b544-205c3f2a52b3-kube-api-access-z8npn\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:23.382936 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:23.382821    5570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c3d9f2d-dcbb-4fb3-a54f-ed9cdef4118e-istiod-ca-cert\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:23.382936 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:23.382873    5570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c3d9f2d-dcbb-4fb3-a54f-ed9cdef4118e-credential-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:23.382936 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:23.382888    5570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c3d9f2d-dcbb-4fb3-a54f-ed9cdef4118e-istio-data\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:23.382936 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:23.382902    5570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e314286e-4b2d-4fae-bc59-528809a8dbce-istio-envoy\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:23.382936 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:23.382915    5570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/157f80a0-a704-4bed-b544-205c3f2a52b3-istio-podinfo\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:23.382936 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:23.382927    5570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/157f80a0-a704-4bed-b544-205c3f2a52b3-istio-envoy\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:23.383262 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:23.382940    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qpw8\" (UniqueName: \"kubernetes.io/projected/e314286e-4b2d-4fae-bc59-528809a8dbce-kube-api-access-2qpw8\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:23.383262 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:23.382952    5570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/157f80a0-a704-4bed-b544-205c3f2a52b3-istio-data\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:23.383262 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:23.382960    5570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/157f80a0-a704-4bed-b544-205c3f2a52b3-istio-token\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:23.383262 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:23.382968    5570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e314286e-4b2d-4fae-bc59-528809a8dbce-istiod-ca-cert\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:23.383262 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:23.382976    5570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e314286e-4b2d-4fae-bc59-528809a8dbce-istio-token\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:23.383568 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:23.383381    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c3d9f2d-dcbb-4fb3-a54f-ed9cdef4118e-istio-envoy" pod "3c3d9f2d-dcbb-4fb3-a54f-ed9cdef4118e" (UID: "3c3d9f2d-dcbb-4fb3-a54f-ed9cdef4118e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:23.384999 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:23.384730    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c3d9f2d-dcbb-4fb3-a54f-ed9cdef4118e-kube-api-access-8wfct" pod "3c3d9f2d-dcbb-4fb3-a54f-ed9cdef4118e" (UID: "3c3d9f2d-dcbb-4fb3-a54f-ed9cdef4118e"). InnerVolumeSpecName "kube-api-access-8wfct". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:27:23.385413 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:23.385386    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3c3d9f2d-dcbb-4fb3-a54f-ed9cdef4118e-istio-podinfo" pod "3c3d9f2d-dcbb-4fb3-a54f-ed9cdef4118e" (UID: "3c3d9f2d-dcbb-4fb3-a54f-ed9cdef4118e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:27:23.385977 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:23.385943    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c3d9f2d-dcbb-4fb3-a54f-ed9cdef4118e-istio-token" pod "3c3d9f2d-dcbb-4fb3-a54f-ed9cdef4118e" (UID: "3c3d9f2d-dcbb-4fb3-a54f-ed9cdef4118e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:27:23.492573 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:23.492437    5570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c3d9f2d-dcbb-4fb3-a54f-ed9cdef4118e-istio-envoy\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:23.492573 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:23.492485    5570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c3d9f2d-dcbb-4fb3-a54f-ed9cdef4118e-istio-podinfo\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:23.492573 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:23.492500    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wfct\" (UniqueName: \"kubernetes.io/projected/3c3d9f2d-dcbb-4fb3-a54f-ed9cdef4118e-kube-api-access-8wfct\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:23.492573 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:23.492514    5570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c3d9f2d-dcbb-4fb3-a54f-ed9cdef4118e-istio-token\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:23.968517 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:23.968481    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26831c71-2091-4090-843d-f7268d257114" path="/var/lib/kubelet/pods/26831c71-2091-4090-843d-f7268d257114/volumes"
Nov 03 20:27:23.969202 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:23.969184    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ee1819e-4649-4d3e-9cfc-0d2d72c1b481" path="/var/lib/kubelet/pods/4ee1819e-4649-4d3e-9cfc-0d2d72c1b481/volumes"
Nov 03 20:27:23.969793 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:23.969763    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51c1be09-aa2e-47bd-a863-97f7daf94ddb" path="/var/lib/kubelet/pods/51c1be09-aa2e-47bd-a863-97f7daf94ddb/volumes"
Nov 03 20:27:23.970656 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:23.970632    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e7f17ab-0f1d-4a59-b415-fe844ae7c1e7" path="/var/lib/kubelet/pods/5e7f17ab-0f1d-4a59-b415-fe844ae7c1e7/volumes"
Nov 03 20:27:23.971179 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:23.971164    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6029825e-e953-4c43-9f4d-d19b5ca38623" path="/var/lib/kubelet/pods/6029825e-e953-4c43-9f4d-d19b5ca38623/volumes"
Nov 03 20:27:23.972187 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:23.972170    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a1b3d29-b647-46e1-9005-e540e937f85f" path="/var/lib/kubelet/pods/6a1b3d29-b647-46e1-9005-e540e937f85f/volumes"
Nov 03 20:27:23.973486 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:23.973469    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="baacee7f-b912-4964-9855-be2beb0094cb" path="/var/lib/kubelet/pods/baacee7f-b912-4964-9855-be2beb0094cb/volumes"
Nov 03 20:27:23.974004 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:23.973990    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce7b1d98-c21f-4412-97ce-9dca59ce2649" path="/var/lib/kubelet/pods/ce7b1d98-c21f-4412-97ce-9dca59ce2649/volumes"
Nov 03 20:27:24.012355 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:24.012323    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmkq7"
Nov 03 20:27:24.012556 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:24.012321    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmkq7" event={"ID":"157f80a0-a704-4bed-b544-205c3f2a52b3","Type":"ContainerDied","Data":"8be46425a20c8ac76a082b07be9e9ff588a7cb4c60dcb55d700db5efaca41da9"}
Nov 03 20:27:24.012556 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:24.012508    5570 scope.go:122] "RemoveContainer" containerID="73be03f9649446acd3f30a87e3b60fbb7889d073564dcf2859c9ed6c410dcaf5"
Nov 03 20:27:24.014134 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:24.014100    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-wkmg8"
Nov 03 20:27:24.014255 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:24.014126    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-wkmg8" event={"ID":"e314286e-4b2d-4fae-bc59-528809a8dbce","Type":"ContainerDied","Data":"1a75e80c761a8d9ef7d2023f9be3da61cca4e9173f219793eab19617ef57d8d7"}
Nov 03 20:27:24.016033 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:24.016004    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-7458z" event={"ID":"3c3d9f2d-dcbb-4fb3-a54f-ed9cdef4118e","Type":"ContainerDied","Data":"5b0f25bdb6d594fa90824eebc1bd7252a9ff48bf3b98c363cb7a49093003939f"}
Nov 03 20:27:24.016142 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:24.016051    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-7458z"
Nov 03 20:27:24.024612 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:24.024595    5570 scope.go:122] "RemoveContainer" containerID="39dce94b7e35773069c096fb3e6dee58c6fc5bf5e34a8cbdbb93fa453abea1cc"
Nov 03 20:27:24.036081 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:24.036060    5570 scope.go:122] "RemoveContainer" containerID="5cfb10265f083a5dc0871a30cf040733f11efc2fb138b8ba9e9ecd8ad963fc77"
Nov 03 20:27:24.071595 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:24.071558    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-wkmg8"]
Nov 03 20:27:24.076271 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:24.076238    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-wkmg8"]
Nov 03 20:27:24.121579 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:24.121542    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmkq7"]
Nov 03 20:27:24.125013 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:24.124992    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmkq7"]
Nov 03 20:27:24.163869 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:24.163823    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-7458z"]
Nov 03 20:27:24.167244 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:24.167223    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-7458z"]
Nov 03 20:27:25.967944 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:25.967914    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="157f80a0-a704-4bed-b544-205c3f2a52b3" path="/var/lib/kubelet/pods/157f80a0-a704-4bed-b544-205c3f2a52b3/volumes"
Nov 03 20:27:25.968474 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:25.968459    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c3d9f2d-dcbb-4fb3-a54f-ed9cdef4118e" path="/var/lib/kubelet/pods/3c3d9f2d-dcbb-4fb3-a54f-ed9cdef4118e/volumes"
Nov 03 20:27:25.969061 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:25.969046    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e314286e-4b2d-4fae-bc59-528809a8dbce" path="/var/lib/kubelet/pods/e314286e-4b2d-4fae-bc59-528809a8dbce/volumes"
Nov 03 20:27:27.238076 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:27:27.238040    5570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8bwhz" containerName="extractor"
Nov 03 20:27:29.915280 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:29.915240    5570 scope.go:122] "RemoveContainer" containerID="3fb0b1c4fb132283f1c575ea853e4c3e51f0937f24de0d2d5bb4eee799832e71"
Nov 03 20:27:29.928074 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:29.928050    5570 scope.go:122] "RemoveContainer" containerID="9aefaf40501ee5dc0f0ba88ef0e2a1c17232ed3858d61c58eca8b6d0b6f39b4b"
Nov 03 20:27:29.939196 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:29.939165    5570 scope.go:122] "RemoveContainer" containerID="886b41422b9f1b0fd3151bcd88be90244c135760b792a86285fd44871d58231a"
Nov 03 20:27:29.949566 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:29.949544    5570 scope.go:122] "RemoveContainer" containerID="126130e48b1ef8613d48e9bc21b7c9cf2466320cbd66e077de2689975c62af81"
Nov 03 20:27:29.959858 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:29.959835    5570 scope.go:122] "RemoveContainer" containerID="b1a97f151e6cfe91f285361e57a4d39552ecef5752a82463f243cd17408873bf"
Nov 03 20:27:29.971450 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:29.971430    5570 scope.go:122] "RemoveContainer" containerID="79eba0ca017a8bc3978468618c78158e97c41c4ffed7d08e1a92e80fadf784ba"
Nov 03 20:27:30.146095 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:30.146064    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-eb87f429-6b49f7957f-k2q4j"]
Nov 03 20:27:31.347190 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:31.347153    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4cdbeb-6b867f8fcd-rw72k"]
Nov 03 20:27:31.348167 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:31.348141    5570 cpu_manager.go:414] "containerMap: removing container" podUID="51c1be09-aa2e-47bd-a863-97f7daf94ddb" containerName="istio-proxy"
Nov 03 20:27:31.348167 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:31.348165    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51c1be09-aa2e-47bd-a863-97f7daf94ddb" containerName="istio-proxy"
Nov 03 20:27:31.348364 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:31.348185    5570 cpu_manager.go:414] "containerMap: removing container" podUID="6029825e-e953-4c43-9f4d-d19b5ca38623" containerName="istio-proxy"
Nov 03 20:27:31.348364 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:31.348193    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6029825e-e953-4c43-9f4d-d19b5ca38623" containerName="istio-proxy"
Nov 03 20:27:31.348364 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:31.348222    5570 cpu_manager.go:414] "containerMap: removing container" podUID="e314286e-4b2d-4fae-bc59-528809a8dbce" containerName="istio-proxy"
Nov 03 20:27:31.348364 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:31.348229    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e314286e-4b2d-4fae-bc59-528809a8dbce" containerName="istio-proxy"
Nov 03 20:27:31.348364 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:31.348240    5570 cpu_manager.go:414] "containerMap: removing container" podUID="157f80a0-a704-4bed-b544-205c3f2a52b3" containerName="istio-proxy"
Nov 03 20:27:31.348364 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:31.348246    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="157f80a0-a704-4bed-b544-205c3f2a52b3" containerName="istio-proxy"
Nov 03 20:27:31.348364 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:31.348257    5570 cpu_manager.go:414] "containerMap: removing container" podUID="baacee7f-b912-4964-9855-be2beb0094cb" containerName="istio-proxy"
Nov 03 20:27:31.348364 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:31.348263    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="baacee7f-b912-4964-9855-be2beb0094cb" containerName="istio-proxy"
Nov 03 20:27:31.348364 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:31.348277    5570 cpu_manager.go:414] "containerMap: removing container" podUID="5e7f17ab-0f1d-4a59-b415-fe844ae7c1e7" containerName="istio-proxy"
Nov 03 20:27:31.348364 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:31.348283    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e7f17ab-0f1d-4a59-b415-fe844ae7c1e7" containerName="istio-proxy"
Nov 03 20:27:31.348364 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:31.348306    5570 cpu_manager.go:414] "containerMap: removing container" podUID="ce7b1d98-c21f-4412-97ce-9dca59ce2649" containerName="istio-proxy"
Nov 03 20:27:31.348364 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:31.348312    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce7b1d98-c21f-4412-97ce-9dca59ce2649" containerName="istio-proxy"
Nov 03 20:27:31.348364 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:31.348323    5570 cpu_manager.go:414] "containerMap: removing container" podUID="6a1b3d29-b647-46e1-9005-e540e937f85f" containerName="istio-proxy"
Nov 03 20:27:31.348364 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:31.348330    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a1b3d29-b647-46e1-9005-e540e937f85f" containerName="istio-proxy"
Nov 03 20:27:31.348364 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:31.348348    5570 cpu_manager.go:414] "containerMap: removing container" podUID="63727774-9118-4b83-bede-04208e8d59b9" containerName="istio-proxy"
Nov 03 20:27:31.348364 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:31.348356    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63727774-9118-4b83-bede-04208e8d59b9" containerName="istio-proxy"
Nov 03 20:27:31.350523 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:31.348371    5570 cpu_manager.go:414] "containerMap: removing container" podUID="55945440-d40e-4235-8b84-d1e2a013372d" containerName="istio-proxy"
Nov 03 20:27:31.350523 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:31.348378    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55945440-d40e-4235-8b84-d1e2a013372d" containerName="istio-proxy"
Nov 03 20:27:31.350523 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:31.348398    5570 cpu_manager.go:414] "containerMap: removing container" podUID="c1e2eadf-ede0-4ea2-9eea-f25a39779795" containerName="istio-proxy"
Nov 03 20:27:31.350523 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:31.348405    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1e2eadf-ede0-4ea2-9eea-f25a39779795" containerName="istio-proxy"
Nov 03 20:27:31.350523 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:31.348417    5570 cpu_manager.go:414] "containerMap: removing container" podUID="26831c71-2091-4090-843d-f7268d257114" containerName="istio-proxy"
Nov 03 20:27:31.350523 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:31.348424    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26831c71-2091-4090-843d-f7268d257114" containerName="istio-proxy"
Nov 03 20:27:31.350523 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:31.348444    5570 cpu_manager.go:414] "containerMap: removing container" podUID="3c3d9f2d-dcbb-4fb3-a54f-ed9cdef4118e" containerName="istio-proxy"
Nov 03 20:27:31.350523 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:31.348452    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c3d9f2d-dcbb-4fb3-a54f-ed9cdef4118e" containerName="istio-proxy"
Nov 03 20:27:31.350523 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:31.348462    5570 cpu_manager.go:414] "containerMap: removing container" podUID="4ee1819e-4649-4d3e-9cfc-0d2d72c1b481" containerName="istio-proxy"
Nov 03 20:27:31.350523 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:31.348468    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ee1819e-4649-4d3e-9cfc-0d2d72c1b481" containerName="istio-proxy"
Nov 03 20:27:31.350523 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:31.348613    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="e314286e-4b2d-4fae-bc59-528809a8dbce" containerName="istio-proxy"
Nov 03 20:27:31.350523 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:31.348623    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="157f80a0-a704-4bed-b544-205c3f2a52b3" containerName="istio-proxy"
Nov 03 20:27:31.350523 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:31.348630    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="baacee7f-b912-4964-9855-be2beb0094cb" containerName="istio-proxy"
Nov 03 20:27:31.350523 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:31.348635    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e7f17ab-0f1d-4a59-b415-fe844ae7c1e7" containerName="istio-proxy"
Nov 03 20:27:31.350523 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:31.348644    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce7b1d98-c21f-4412-97ce-9dca59ce2649" containerName="istio-proxy"
Nov 03 20:27:31.350523 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:31.348650    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a1b3d29-b647-46e1-9005-e540e937f85f" containerName="istio-proxy"
Nov 03 20:27:31.350523 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:31.348655    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="63727774-9118-4b83-bede-04208e8d59b9" containerName="istio-proxy"
Nov 03 20:27:31.350523 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:31.348663    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="55945440-d40e-4235-8b84-d1e2a013372d" containerName="istio-proxy"
Nov 03 20:27:31.350523 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:31.348669    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1e2eadf-ede0-4ea2-9eea-f25a39779795" containerName="istio-proxy"
Nov 03 20:27:31.350523 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:31.348677    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="26831c71-2091-4090-843d-f7268d257114" containerName="istio-proxy"
Nov 03 20:27:31.350523 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:31.348683    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c3d9f2d-dcbb-4fb3-a54f-ed9cdef4118e" containerName="istio-proxy"
Nov 03 20:27:31.350523 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:31.348689    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ee1819e-4649-4d3e-9cfc-0d2d72c1b481" containerName="istio-proxy"
Nov 03 20:27:31.350523 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:31.348698    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="51c1be09-aa2e-47bd-a863-97f7daf94ddb" containerName="istio-proxy"
Nov 03 20:27:31.350523 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:31.348703    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="6029825e-e953-4c43-9f4d-d19b5ca38623" containerName="istio-proxy"
Nov 03 20:27:31.353811 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:31.353791    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4cdbeb-6b867f8fcd-rw72k"
Nov 03 20:27:31.367392 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:31.367365    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4cdbeb-6b867f8fcd-rw72k"]
Nov 03 20:27:31.372010 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:31.371983    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-4cdbeb-serving-cert\""
Nov 03 20:27:31.440320 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:31.440285    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72pxf\" (UniqueName: \"kubernetes.io/projected/2c6d9548-c1f6-4efb-969b-260308e4ac3a-kube-api-access-72pxf\") pod \"maas-api-e2e-api-b-4cdbeb-6b867f8fcd-rw72k\" (UID: \"2c6d9548-c1f6-4efb-969b-260308e4ac3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4cdbeb-6b867f8fcd-rw72k"
Nov 03 20:27:31.440484 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:31.440419    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c6d9548-c1f6-4efb-969b-260308e4ac3a-openshift-service-ca\") pod \"maas-api-e2e-api-b-4cdbeb-6b867f8fcd-rw72k\" (UID: \"2c6d9548-c1f6-4efb-969b-260308e4ac3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4cdbeb-6b867f8fcd-rw72k"
Nov 03 20:27:31.440525 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:31.440497    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c6d9548-c1f6-4efb-969b-260308e4ac3a-metrics-certs\") pod \"maas-api-e2e-api-b-4cdbeb-6b867f8fcd-rw72k\" (UID: \"2c6d9548-c1f6-4efb-969b-260308e4ac3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4cdbeb-6b867f8fcd-rw72k"
Nov 03 20:27:31.440640 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:31.440628    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c6d9548-c1f6-4efb-969b-260308e4ac3a-maas-api-tls\") pod \"maas-api-e2e-api-b-4cdbeb-6b867f8fcd-rw72k\" (UID: \"2c6d9548-c1f6-4efb-969b-260308e4ac3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4cdbeb-6b867f8fcd-rw72k"
Nov 03 20:27:31.541768 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:31.541729    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72pxf\" (UniqueName: \"kubernetes.io/projected/2c6d9548-c1f6-4efb-969b-260308e4ac3a-kube-api-access-72pxf\") pod \"maas-api-e2e-api-b-4cdbeb-6b867f8fcd-rw72k\" (UID: \"2c6d9548-c1f6-4efb-969b-260308e4ac3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4cdbeb-6b867f8fcd-rw72k"
Nov 03 20:27:31.541978 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:31.541849    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c6d9548-c1f6-4efb-969b-260308e4ac3a-openshift-service-ca\") pod \"maas-api-e2e-api-b-4cdbeb-6b867f8fcd-rw72k\" (UID: \"2c6d9548-c1f6-4efb-969b-260308e4ac3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4cdbeb-6b867f8fcd-rw72k"
Nov 03 20:27:31.541978 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:31.541905    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c6d9548-c1f6-4efb-969b-260308e4ac3a-metrics-certs\") pod \"maas-api-e2e-api-b-4cdbeb-6b867f8fcd-rw72k\" (UID: \"2c6d9548-c1f6-4efb-969b-260308e4ac3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4cdbeb-6b867f8fcd-rw72k"
Nov 03 20:27:31.542086 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:31.542062    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c6d9548-c1f6-4efb-969b-260308e4ac3a-maas-api-tls\") pod \"maas-api-e2e-api-b-4cdbeb-6b867f8fcd-rw72k\" (UID: \"2c6d9548-c1f6-4efb-969b-260308e4ac3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4cdbeb-6b867f8fcd-rw72k"
Nov 03 20:27:31.542604 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:31.542576    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c6d9548-c1f6-4efb-969b-260308e4ac3a-openshift-service-ca\") pod \"maas-api-e2e-api-b-4cdbeb-6b867f8fcd-rw72k\" (UID: \"2c6d9548-c1f6-4efb-969b-260308e4ac3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4cdbeb-6b867f8fcd-rw72k"
Nov 03 20:27:31.545570 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:31.545539    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c6d9548-c1f6-4efb-969b-260308e4ac3a-maas-api-tls\") pod \"maas-api-e2e-api-b-4cdbeb-6b867f8fcd-rw72k\" (UID: \"2c6d9548-c1f6-4efb-969b-260308e4ac3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4cdbeb-6b867f8fcd-rw72k"
Nov 03 20:27:31.545691 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:31.545670    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c6d9548-c1f6-4efb-969b-260308e4ac3a-metrics-certs\") pod \"maas-api-e2e-api-b-4cdbeb-6b867f8fcd-rw72k\" (UID: \"2c6d9548-c1f6-4efb-969b-260308e4ac3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4cdbeb-6b867f8fcd-rw72k"
Nov 03 20:27:31.563431 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:31.563405    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72pxf\" (UniqueName: \"kubernetes.io/projected/2c6d9548-c1f6-4efb-969b-260308e4ac3a-kube-api-access-72pxf\") pod \"maas-api-e2e-api-b-4cdbeb-6b867f8fcd-rw72k\" (UID: \"2c6d9548-c1f6-4efb-969b-260308e4ac3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4cdbeb-6b867f8fcd-rw72k"
Nov 03 20:27:31.680937 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:31.680691    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4cdbeb-6b867f8fcd-rw72k"
Nov 03 20:27:32.073035 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:32.073005    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4cdbeb-6b867f8fcd-rw72k"]
Nov 03 20:27:32.073975 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:27:32.073948    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c6d9548_c1f6_4efb_969b_260308e4ac3a.slice/crio-bfb84b25cc95ad52751e51bad85aef9c2e842dcdf06a9138b7040cbcfc4d45cf WatchSource:0}: Error finding container bfb84b25cc95ad52751e51bad85aef9c2e842dcdf06a9138b7040cbcfc4d45cf: Status 404 returned error can't find the container with id bfb84b25cc95ad52751e51bad85aef9c2e842dcdf06a9138b7040cbcfc4d45cf
Nov 03 20:27:32.188248 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:32.188209    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4cdbeb-6b867f8fcd-rw72k" event={"ID":"2c6d9548-c1f6-4efb-969b-260308e4ac3a","Type":"ContainerStarted","Data":"bfb84b25cc95ad52751e51bad85aef9c2e842dcdf06a9138b7040cbcfc4d45cf"}
Nov 03 20:27:33.231015 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:33.230529    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4cdbeb-6b867f8fcd-rw72k" event={"ID":"2c6d9548-c1f6-4efb-969b-260308e4ac3a","Type":"ContainerStarted","Data":"dd981c84133739e32d79464a1bfd7e6f5091642f5f731090459e003ef5dffcfc"}
Nov 03 20:27:33.236811 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:33.233739    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4cdbeb-6b867f8fcd-rw72k"
Nov 03 20:27:33.281669 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:33.281616    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4cdbeb-6b867f8fcd-rw72k" podStartSLOduration=1.66360332 podStartE2EDuration="2.281602114s" totalImagesPullingTime="617.998794ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:27:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:27:33.279543763 +0000 UTC m=+1387.432205652" watchObservedRunningTime="2026-11-03 20:27:33.281602114 +0000 UTC m=+1387.434264001"
Nov 03 20:27:35.146984 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:35.146943    5570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-eb87f429-6b49f7957f-k2q4j" podUID="4711b06e-dba0-4d7a-a217-defa6ab8f008" containerName="payload-pre-processing" containerID="cri-o://8352b9b1531ef87160ee04df65255c2b018105116c1b1b760f1becb449967e32" gracePeriod=25
Nov 03 20:27:35.337653 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:35.336255    5570 generic.go:356] "Generic (PLEG): container finished" podID="4711b06e-dba0-4d7a-a217-defa6ab8f008" containerID="8352b9b1531ef87160ee04df65255c2b018105116c1b1b760f1becb449967e32" exitCode=0
Nov 03 20:27:35.337653 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:35.336392    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-eb87f429-6b49f7957f-k2q4j" event={"ID":"4711b06e-dba0-4d7a-a217-defa6ab8f008","Type":"ContainerDied","Data":"8352b9b1531ef87160ee04df65255c2b018105116c1b1b760f1becb449967e32"}
Nov 03 20:27:35.461631 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:35.461596    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-eb87f429-6b49f7957f-k2q4j"
Nov 03 20:27:35.553292 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:35.553262    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4711b06e-dba0-4d7a-a217-defa6ab8f008-kube-api-access-g9zwv\" (UniqueName: \"kubernetes.io/projected/4711b06e-dba0-4d7a-a217-defa6ab8f008-kube-api-access-g9zwv\") pod \"4711b06e-dba0-4d7a-a217-defa6ab8f008\" (UID: \"4711b06e-dba0-4d7a-a217-defa6ab8f008\") "
Nov 03 20:27:35.553470 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:35.553436    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4711b06e-dba0-4d7a-a217-defa6ab8f008-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4711b06e-dba0-4d7a-a217-defa6ab8f008-plugins-config-volume\") pod \"4711b06e-dba0-4d7a-a217-defa6ab8f008\" (UID: \"4711b06e-dba0-4d7a-a217-defa6ab8f008\") "
Nov 03 20:27:35.554070 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:35.554035    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4711b06e-dba0-4d7a-a217-defa6ab8f008-plugins-config-volume" pod "4711b06e-dba0-4d7a-a217-defa6ab8f008" (UID: "4711b06e-dba0-4d7a-a217-defa6ab8f008"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:27:35.555502 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:35.555471    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4711b06e-dba0-4d7a-a217-defa6ab8f008-kube-api-access-g9zwv" pod "4711b06e-dba0-4d7a-a217-defa6ab8f008" (UID: "4711b06e-dba0-4d7a-a217-defa6ab8f008"). InnerVolumeSpecName "kube-api-access-g9zwv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:27:35.655964 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:35.655923    5570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4711b06e-dba0-4d7a-a217-defa6ab8f008-plugins-config-volume\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:35.655964 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:35.655956    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9zwv\" (UniqueName: \"kubernetes.io/projected/4711b06e-dba0-4d7a-a217-defa6ab8f008-kube-api-access-g9zwv\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:36.287012 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:36.286934    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d1bfb8-7b6cb98c-pkvgg"]
Nov 03 20:27:36.287649 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:36.287635    5570 cpu_manager.go:414] "containerMap: removing container" podUID="6a1b3d29-b647-46e1-9005-e540e937f85f" containerName="istio-proxy"
Nov 03 20:27:36.287697 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:36.287650    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a1b3d29-b647-46e1-9005-e540e937f85f" containerName="istio-proxy"
Nov 03 20:27:36.287697 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:36.287665    5570 cpu_manager.go:414] "containerMap: removing container" podUID="4711b06e-dba0-4d7a-a217-defa6ab8f008" containerName="payload-pre-processing"
Nov 03 20:27:36.287697 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:36.287670    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4711b06e-dba0-4d7a-a217-defa6ab8f008" containerName="payload-pre-processing"
Nov 03 20:27:36.287837 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:36.287826    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a1b3d29-b647-46e1-9005-e540e937f85f" containerName="istio-proxy"
Nov 03 20:27:36.287874 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:36.287841    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="4711b06e-dba0-4d7a-a217-defa6ab8f008" containerName="payload-pre-processing"
Nov 03 20:27:36.291582 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:36.291563    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d1bfb8-7b6cb98c-pkvgg"
Nov 03 20:27:36.309460 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:36.309430    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d1bfb8-7b6cb98c-pkvgg"]
Nov 03 20:27:36.317061 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:36.316131    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-d1bfb8-serving-cert\""
Nov 03 20:27:36.355185 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:36.355158    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-eb87f429-6b49f7957f-k2q4j"
Nov 03 20:27:36.355384 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:36.355156    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-eb87f429-6b49f7957f-k2q4j" event={"ID":"4711b06e-dba0-4d7a-a217-defa6ab8f008","Type":"ContainerDied","Data":"4ba5bba4799cf4c87494df665c2b48615e428452485794c642eb8881cfd62403"}
Nov 03 20:27:36.355384 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:36.355367    5570 scope.go:122] "RemoveContainer" containerID="8352b9b1531ef87160ee04df65255c2b018105116c1b1b760f1becb449967e32"
Nov 03 20:27:36.414338 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:36.414309    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-eb87f429-6b49f7957f-k2q4j"]
Nov 03 20:27:36.417764 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:36.417738    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-eb87f429-6b49f7957f-k2q4j"]
Nov 03 20:27:36.490798 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:36.486151    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7c7754a-0ea1-45c7-825e-d527b1ccc1eb-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-d1bfb8-7b6cb98c-pkvgg\" (UID: \"a7c7754a-0ea1-45c7-825e-d527b1ccc1eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d1bfb8-7b6cb98c-pkvgg"
Nov 03 20:27:36.490798 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:36.486244    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a7c7754a-0ea1-45c7-825e-d527b1ccc1eb-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-d1bfb8-7b6cb98c-pkvgg\" (UID: \"a7c7754a-0ea1-45c7-825e-d527b1ccc1eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d1bfb8-7b6cb98c-pkvgg"
Nov 03 20:27:36.490798 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:36.486464    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtbml\" (UniqueName: \"kubernetes.io/projected/a7c7754a-0ea1-45c7-825e-d527b1ccc1eb-kube-api-access-jtbml\") pod \"maas-api-e2e-shared-b-w3-d1bfb8-7b6cb98c-pkvgg\" (UID: \"a7c7754a-0ea1-45c7-825e-d527b1ccc1eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d1bfb8-7b6cb98c-pkvgg"
Nov 03 20:27:36.490798 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:36.486574    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a7c7754a-0ea1-45c7-825e-d527b1ccc1eb-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-d1bfb8-7b6cb98c-pkvgg\" (UID: \"a7c7754a-0ea1-45c7-825e-d527b1ccc1eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d1bfb8-7b6cb98c-pkvgg"
Nov 03 20:27:36.592284 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:36.592188    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtbml\" (UniqueName: \"kubernetes.io/projected/a7c7754a-0ea1-45c7-825e-d527b1ccc1eb-kube-api-access-jtbml\") pod \"maas-api-e2e-shared-b-w3-d1bfb8-7b6cb98c-pkvgg\" (UID: \"a7c7754a-0ea1-45c7-825e-d527b1ccc1eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d1bfb8-7b6cb98c-pkvgg"
Nov 03 20:27:36.592475 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:36.592363    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a7c7754a-0ea1-45c7-825e-d527b1ccc1eb-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-d1bfb8-7b6cb98c-pkvgg\" (UID: \"a7c7754a-0ea1-45c7-825e-d527b1ccc1eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d1bfb8-7b6cb98c-pkvgg"
Nov 03 20:27:36.592562 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:36.592538    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7c7754a-0ea1-45c7-825e-d527b1ccc1eb-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-d1bfb8-7b6cb98c-pkvgg\" (UID: \"a7c7754a-0ea1-45c7-825e-d527b1ccc1eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d1bfb8-7b6cb98c-pkvgg"
Nov 03 20:27:36.592619 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:36.592605    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a7c7754a-0ea1-45c7-825e-d527b1ccc1eb-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-d1bfb8-7b6cb98c-pkvgg\" (UID: \"a7c7754a-0ea1-45c7-825e-d527b1ccc1eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d1bfb8-7b6cb98c-pkvgg"
Nov 03 20:27:36.594174 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:27:36.594127    5570 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d1bfb8-serving-cert: secret "maas-api-e2e-shared-b-w3-d1bfb8-serving-cert" not found
Nov 03 20:27:36.594314 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:27:36.594296    5570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7c7754a-0ea1-45c7-825e-d527b1ccc1eb-maas-api-tls podName:a7c7754a-0ea1-45c7-825e-d527b1ccc1eb nodeName:}" failed. No retries permitted until 2026-11-03 20:27:37.094257657 +0000 UTC m=+1391.246919545 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a7c7754a-0ea1-45c7-825e-d527b1ccc1eb-maas-api-tls") pod "maas-api-e2e-shared-b-w3-d1bfb8-7b6cb98c-pkvgg" (UID: "a7c7754a-0ea1-45c7-825e-d527b1ccc1eb") : secret "maas-api-e2e-shared-b-w3-d1bfb8-serving-cert" not found
Nov 03 20:27:36.595527 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:36.595463    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7c7754a-0ea1-45c7-825e-d527b1ccc1eb-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-d1bfb8-7b6cb98c-pkvgg\" (UID: \"a7c7754a-0ea1-45c7-825e-d527b1ccc1eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d1bfb8-7b6cb98c-pkvgg"
Nov 03 20:27:36.616579 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:36.616541    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a7c7754a-0ea1-45c7-825e-d527b1ccc1eb-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-d1bfb8-7b6cb98c-pkvgg\" (UID: \"a7c7754a-0ea1-45c7-825e-d527b1ccc1eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d1bfb8-7b6cb98c-pkvgg"
Nov 03 20:27:36.626796 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:36.620214    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtbml\" (UniqueName: \"kubernetes.io/projected/a7c7754a-0ea1-45c7-825e-d527b1ccc1eb-kube-api-access-jtbml\") pod \"maas-api-e2e-shared-b-w3-d1bfb8-7b6cb98c-pkvgg\" (UID: \"a7c7754a-0ea1-45c7-825e-d527b1ccc1eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d1bfb8-7b6cb98c-pkvgg"
Nov 03 20:27:37.105534 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:37.104046    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a7c7754a-0ea1-45c7-825e-d527b1ccc1eb-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-d1bfb8-7b6cb98c-pkvgg\" (UID: \"a7c7754a-0ea1-45c7-825e-d527b1ccc1eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d1bfb8-7b6cb98c-pkvgg"
Nov 03 20:27:37.112991 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:37.110022    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a7c7754a-0ea1-45c7-825e-d527b1ccc1eb-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-d1bfb8-7b6cb98c-pkvgg\" (UID: \"a7c7754a-0ea1-45c7-825e-d527b1ccc1eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d1bfb8-7b6cb98c-pkvgg"
Nov 03 20:27:37.210788 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:37.210742    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d1bfb8-7b6cb98c-pkvgg"
Nov 03 20:27:37.747953 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:37.747922    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d1bfb8-7b6cb98c-pkvgg"]
Nov 03 20:27:37.748294 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:27:37.748269    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7c7754a_0ea1_45c7_825e_d527b1ccc1eb.slice/crio-9da2bd25638efa14c3011e4ac35191e0c7b3e9e8b1d6bdfbddb98bd0361312c6 WatchSource:0}: Error finding container 9da2bd25638efa14c3011e4ac35191e0c7b3e9e8b1d6bdfbddb98bd0361312c6: Status 404 returned error can't find the container with id 9da2bd25638efa14c3011e4ac35191e0c7b3e9e8b1d6bdfbddb98bd0361312c6
Nov 03 20:27:37.965697 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:37.965663    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4711b06e-dba0-4d7a-a217-defa6ab8f008" path="/var/lib/kubelet/pods/4711b06e-dba0-4d7a-a217-defa6ab8f008/volumes"
Nov 03 20:27:38.584607 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:38.566198    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d1bfb8-7b6cb98c-pkvgg" event={"ID":"a7c7754a-0ea1-45c7-825e-d527b1ccc1eb","Type":"ContainerStarted","Data":"c3e62fd6b5fa8b1021d309d18e1ce52272e7800a4cd8dd9b176ac711d481de7f"}
Nov 03 20:27:38.584607 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:38.566301    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d1bfb8-7b6cb98c-pkvgg" event={"ID":"a7c7754a-0ea1-45c7-825e-d527b1ccc1eb","Type":"ContainerStarted","Data":"9da2bd25638efa14c3011e4ac35191e0c7b3e9e8b1d6bdfbddb98bd0361312c6"}
Nov 03 20:27:38.584607 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:38.567794    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d1bfb8-7b6cb98c-pkvgg"
Nov 03 20:27:38.623245 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:38.623177    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d1bfb8-7b6cb98c-pkvgg" podStartSLOduration=2.204075115 podStartE2EDuration="2.623156341s" totalImagesPullingTime="419.081226ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:27:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:27:38.62062058 +0000 UTC m=+1392.773282478" watchObservedRunningTime="2026-11-03 20:27:38.623156341 +0000 UTC m=+1392.775818233"
Nov 03 20:27:39.332258 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:39.332225    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4cdbeb-6b867f8fcd-rw72k"
Nov 03 20:27:40.097394 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:40.097360    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g69f8"]
Nov 03 20:27:45.588255 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:45.588223    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d1bfb8-7b6cb98c-pkvgg"
Nov 03 20:27:45.627257 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:45.627222    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-c98nc"]
Nov 03 20:27:45.627478 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:45.627428    5570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-c98nc" podUID="755f6eab-5d03-4368-9670-b1402084474c" containerName="istio-proxy" containerID="cri-o://a414a503eff894ce1fa7aaee149e172cf442c1cd2ba6f8cd05579632d356f512" gracePeriod=30
Nov 03 20:27:45.631981 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:45.631289    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-kmqsw"]
Nov 03 20:27:45.631981 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:45.631537    5570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-kmqsw" podUID="28aada86-170c-4935-80ed-e8c5ee503c14" containerName="istio-proxy" containerID="cri-o://f8dd680f3a7881f299314139130aacee319f496bfe291e09b3d07a6d33658515" gracePeriod=30
Nov 03 20:27:45.646680 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:45.642154    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-qskf7"]
Nov 03 20:27:45.646680 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:45.642400    5570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-qskf7" podUID="469fc281-f7ee-4584-b3b6-85c6a669f806" containerName="istio-proxy" containerID="cri-o://04cc270bc0362b4abb3459a1a7771e91e258cc93bb68adfcdb883fbb3afe99bd" gracePeriod=30
Nov 03 20:27:45.692703 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:45.687786    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sf2lg"]
Nov 03 20:27:45.692703 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:45.688040    5570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sf2lg" podUID="07093a96-d60c-490f-910b-734f41752c17" containerName="istio-proxy" containerID="cri-o://52cc5ce0dff70b1613a235dd886d31d1c1e2960a2c37ef9b6eeff114ec29d26a" gracePeriod=30
Nov 03 20:27:45.694951 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:45.694001    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-b4zmq"]
Nov 03 20:27:45.694951 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:45.694236    5570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-b4zmq" podUID="8f7ff673-520b-4dd1-a76d-0e7aa1053c3f" containerName="istio-proxy" containerID="cri-o://c168feccf6d7ffe37ff51f11a79108fcc28d009b7994b2ad80a8a06cb49bd947" gracePeriod=30
Nov 03 20:27:45.700552 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:45.699598    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-vzfw9"]
Nov 03 20:27:45.700552 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:45.699881    5570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-vzfw9" podUID="a739707b-9c5a-4c9b-90df-501c5dd0c98d" containerName="istio-proxy" containerID="cri-o://c98ca04c2cb8bdbe656136e1cd8d0fcac1daa8fdb11f2f3858d8c7f463fa7537" gracePeriod=30
Nov 03 20:27:48.059387 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:48.059343    5570 patch_prober.go:29] interesting pod/e2e-worker-w2-9973d9-openshift-default-ffb6475-kmqsw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 20:27:48.059816 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:48.059417    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-kmqsw" podUID="28aada86-170c-4935-80ed-e8c5ee503c14" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 20:27:48.063250 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:48.063221    5570 patch_prober.go:29] interesting pod/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-b4zmq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 20:27:48.063381 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:48.063277    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-b4zmq" podUID="8f7ff673-520b-4dd1-a76d-0e7aa1053c3f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 20:27:48.827545 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:48.827498    5570 patch_prober.go:29] interesting pod/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sf2lg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 20:27:48.827737 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:48.827582    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sf2lg" podUID="07093a96-d60c-490f-910b-734f41752c17" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 20:27:50.312167 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:50.312121    5570 patch_prober.go:29] interesting pod/e2e-worker-w2-9973d9-openshift-default-ffb6475-qskf7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 20:27:50.312559 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:50.312189    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-qskf7" podUID="469fc281-f7ee-4584-b3b6-85c6a669f806" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 20:27:50.529981 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:50.529931    5570 patch_prober.go:29] interesting pod/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-vzfw9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 20:27:50.530163 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:50.530000    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-vzfw9" podUID="a739707b-9c5a-4c9b-90df-501c5dd0c98d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 20:27:51.047418 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.047387    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-kmqsw"
Nov 03 20:27:51.067034 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.067004    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-b4zmq"
Nov 03 20:27:51.141794 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.139663    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f7ff673-520b-4dd1-a76d-0e7aa1053c3f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f7ff673-520b-4dd1-a76d-0e7aa1053c3f-credential-socket\") pod \"8f7ff673-520b-4dd1-a76d-0e7aa1053c3f\" (UID: \"8f7ff673-520b-4dd1-a76d-0e7aa1053c3f\") "
Nov 03 20:27:51.141794 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.139714    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f7ff673-520b-4dd1-a76d-0e7aa1053c3f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f7ff673-520b-4dd1-a76d-0e7aa1053c3f-istio-data\") pod \"8f7ff673-520b-4dd1-a76d-0e7aa1053c3f\" (UID: \"8f7ff673-520b-4dd1-a76d-0e7aa1053c3f\") "
Nov 03 20:27:51.141794 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.140128    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f7ff673-520b-4dd1-a76d-0e7aa1053c3f-istio-data" pod "8f7ff673-520b-4dd1-a76d-0e7aa1053c3f" (UID: "8f7ff673-520b-4dd1-a76d-0e7aa1053c3f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:51.141794 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.140347    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f7ff673-520b-4dd1-a76d-0e7aa1053c3f-credential-socket" pod "8f7ff673-520b-4dd1-a76d-0e7aa1053c3f" (UID: "8f7ff673-520b-4dd1-a76d-0e7aa1053c3f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:51.141794 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.140454    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f7ff673-520b-4dd1-a76d-0e7aa1053c3f-istio-token\" (UniqueName: \"kubernetes.io/projected/8f7ff673-520b-4dd1-a76d-0e7aa1053c3f-istio-token\") pod \"8f7ff673-520b-4dd1-a76d-0e7aa1053c3f\" (UID: \"8f7ff673-520b-4dd1-a76d-0e7aa1053c3f\") "
Nov 03 20:27:51.141794 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.140537    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f7ff673-520b-4dd1-a76d-0e7aa1053c3f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f7ff673-520b-4dd1-a76d-0e7aa1053c3f-workload-certs\") pod \"8f7ff673-520b-4dd1-a76d-0e7aa1053c3f\" (UID: \"8f7ff673-520b-4dd1-a76d-0e7aa1053c3f\") "
Nov 03 20:27:51.141794 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.140593    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28aada86-170c-4935-80ed-e8c5ee503c14-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28aada86-170c-4935-80ed-e8c5ee503c14-istio-envoy\") pod \"28aada86-170c-4935-80ed-e8c5ee503c14\" (UID: \"28aada86-170c-4935-80ed-e8c5ee503c14\") "
Nov 03 20:27:51.141794 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.140659    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28aada86-170c-4935-80ed-e8c5ee503c14-kube-api-access-8n6x7\" (UniqueName: \"kubernetes.io/projected/28aada86-170c-4935-80ed-e8c5ee503c14-kube-api-access-8n6x7\") pod \"28aada86-170c-4935-80ed-e8c5ee503c14\" (UID: \"28aada86-170c-4935-80ed-e8c5ee503c14\") "
Nov 03 20:27:51.141794 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.140709    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28aada86-170c-4935-80ed-e8c5ee503c14-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28aada86-170c-4935-80ed-e8c5ee503c14-workload-socket\") pod \"28aada86-170c-4935-80ed-e8c5ee503c14\" (UID: \"28aada86-170c-4935-80ed-e8c5ee503c14\") "
Nov 03 20:27:51.141794 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.140743    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8f7ff673-520b-4dd1-a76d-0e7aa1053c3f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f7ff673-520b-4dd1-a76d-0e7aa1053c3f-istio-podinfo\") pod \"8f7ff673-520b-4dd1-a76d-0e7aa1053c3f\" (UID: \"8f7ff673-520b-4dd1-a76d-0e7aa1053c3f\") "
Nov 03 20:27:51.141794 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.140827    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/28aada86-170c-4935-80ed-e8c5ee503c14-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28aada86-170c-4935-80ed-e8c5ee503c14-istio-podinfo\") pod \"28aada86-170c-4935-80ed-e8c5ee503c14\" (UID: \"28aada86-170c-4935-80ed-e8c5ee503c14\") "
Nov 03 20:27:51.141794 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.140888    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28aada86-170c-4935-80ed-e8c5ee503c14-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28aada86-170c-4935-80ed-e8c5ee503c14-istio-data\") pod \"28aada86-170c-4935-80ed-e8c5ee503c14\" (UID: \"28aada86-170c-4935-80ed-e8c5ee503c14\") "
Nov 03 20:27:51.141794 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.140935    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f7ff673-520b-4dd1-a76d-0e7aa1053c3f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f7ff673-520b-4dd1-a76d-0e7aa1053c3f-istiod-ca-cert\") pod \"8f7ff673-520b-4dd1-a76d-0e7aa1053c3f\" (UID: \"8f7ff673-520b-4dd1-a76d-0e7aa1053c3f\") "
Nov 03 20:27:51.141794 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.141002    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28aada86-170c-4935-80ed-e8c5ee503c14-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28aada86-170c-4935-80ed-e8c5ee503c14-workload-certs\") pod \"28aada86-170c-4935-80ed-e8c5ee503c14\" (UID: \"28aada86-170c-4935-80ed-e8c5ee503c14\") "
Nov 03 20:27:51.141794 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.141062    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28aada86-170c-4935-80ed-e8c5ee503c14-istio-token\" (UniqueName: \"kubernetes.io/projected/28aada86-170c-4935-80ed-e8c5ee503c14-istio-token\") pod \"28aada86-170c-4935-80ed-e8c5ee503c14\" (UID: \"28aada86-170c-4935-80ed-e8c5ee503c14\") "
Nov 03 20:27:51.141794 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.141128    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f7ff673-520b-4dd1-a76d-0e7aa1053c3f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f7ff673-520b-4dd1-a76d-0e7aa1053c3f-istio-envoy\") pod \"8f7ff673-520b-4dd1-a76d-0e7aa1053c3f\" (UID: \"8f7ff673-520b-4dd1-a76d-0e7aa1053c3f\") "
Nov 03 20:27:51.141794 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.141567    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f7ff673-520b-4dd1-a76d-0e7aa1053c3f-istiod-ca-cert" pod "8f7ff673-520b-4dd1-a76d-0e7aa1053c3f" (UID: "8f7ff673-520b-4dd1-a76d-0e7aa1053c3f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:27:51.142830 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.141952    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28aada86-170c-4935-80ed-e8c5ee503c14-istio-data" pod "28aada86-170c-4935-80ed-e8c5ee503c14" (UID: "28aada86-170c-4935-80ed-e8c5ee503c14"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:51.142830 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.142635    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f7ff673-520b-4dd1-a76d-0e7aa1053c3f-workload-certs" pod "8f7ff673-520b-4dd1-a76d-0e7aa1053c3f" (UID: "8f7ff673-520b-4dd1-a76d-0e7aa1053c3f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:51.143156 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.142864    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28aada86-170c-4935-80ed-e8c5ee503c14-workload-certs" pod "28aada86-170c-4935-80ed-e8c5ee503c14" (UID: "28aada86-170c-4935-80ed-e8c5ee503c14"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:51.143222 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.143161    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28aada86-170c-4935-80ed-e8c5ee503c14-workload-socket" pod "28aada86-170c-4935-80ed-e8c5ee503c14" (UID: "28aada86-170c-4935-80ed-e8c5ee503c14"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:51.147304 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.143906    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f7ff673-520b-4dd1-a76d-0e7aa1053c3f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f7ff673-520b-4dd1-a76d-0e7aa1053c3f-workload-socket\") pod \"8f7ff673-520b-4dd1-a76d-0e7aa1053c3f\" (UID: \"8f7ff673-520b-4dd1-a76d-0e7aa1053c3f\") "
Nov 03 20:27:51.147304 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.143994    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28aada86-170c-4935-80ed-e8c5ee503c14-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28aada86-170c-4935-80ed-e8c5ee503c14-credential-socket\") pod \"28aada86-170c-4935-80ed-e8c5ee503c14\" (UID: \"28aada86-170c-4935-80ed-e8c5ee503c14\") "
Nov 03 20:27:51.147304 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.144067    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28aada86-170c-4935-80ed-e8c5ee503c14-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28aada86-170c-4935-80ed-e8c5ee503c14-istiod-ca-cert\") pod \"28aada86-170c-4935-80ed-e8c5ee503c14\" (UID: \"28aada86-170c-4935-80ed-e8c5ee503c14\") "
Nov 03 20:27:51.147304 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.144152    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f7ff673-520b-4dd1-a76d-0e7aa1053c3f-kube-api-access-x5kvd\" (UniqueName: \"kubernetes.io/projected/8f7ff673-520b-4dd1-a76d-0e7aa1053c3f-kube-api-access-x5kvd\") pod \"8f7ff673-520b-4dd1-a76d-0e7aa1053c3f\" (UID: \"8f7ff673-520b-4dd1-a76d-0e7aa1053c3f\") "
Nov 03 20:27:51.147304 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.145993    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f7ff673-520b-4dd1-a76d-0e7aa1053c3f-istio-envoy" pod "8f7ff673-520b-4dd1-a76d-0e7aa1053c3f" (UID: "8f7ff673-520b-4dd1-a76d-0e7aa1053c3f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:51.147304 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.146608    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f7ff673-520b-4dd1-a76d-0e7aa1053c3f-workload-socket" pod "8f7ff673-520b-4dd1-a76d-0e7aa1053c3f" (UID: "8f7ff673-520b-4dd1-a76d-0e7aa1053c3f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:51.147304 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.146910    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28aada86-170c-4935-80ed-e8c5ee503c14-credential-socket" pod "28aada86-170c-4935-80ed-e8c5ee503c14" (UID: "28aada86-170c-4935-80ed-e8c5ee503c14"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:51.147304 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.147215    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28aada86-170c-4935-80ed-e8c5ee503c14-istiod-ca-cert" pod "28aada86-170c-4935-80ed-e8c5ee503c14" (UID: "28aada86-170c-4935-80ed-e8c5ee503c14"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:27:51.148065 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.147949    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8f7ff673-520b-4dd1-a76d-0e7aa1053c3f-istio-podinfo" pod "8f7ff673-520b-4dd1-a76d-0e7aa1053c3f" (UID: "8f7ff673-520b-4dd1-a76d-0e7aa1053c3f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:27:51.152189 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.152161    5570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28aada86-170c-4935-80ed-e8c5ee503c14-workload-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:51.152309 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.152193    5570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f7ff673-520b-4dd1-a76d-0e7aa1053c3f-istio-podinfo\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:51.152309 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.152207    5570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28aada86-170c-4935-80ed-e8c5ee503c14-istio-data\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:51.152309 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.152218    5570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f7ff673-520b-4dd1-a76d-0e7aa1053c3f-istiod-ca-cert\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:51.152309 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.152230    5570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28aada86-170c-4935-80ed-e8c5ee503c14-workload-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:51.152309 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.152244    5570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f7ff673-520b-4dd1-a76d-0e7aa1053c3f-istio-envoy\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:51.152309 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.152256    5570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f7ff673-520b-4dd1-a76d-0e7aa1053c3f-workload-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:51.152309 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.152269    5570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28aada86-170c-4935-80ed-e8c5ee503c14-credential-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:51.152309 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.152282    5570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28aada86-170c-4935-80ed-e8c5ee503c14-istiod-ca-cert\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:51.152309 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.152296    5570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f7ff673-520b-4dd1-a76d-0e7aa1053c3f-credential-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:51.152309 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.152308    5570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f7ff673-520b-4dd1-a76d-0e7aa1053c3f-istio-data\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:51.152926 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.152320    5570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f7ff673-520b-4dd1-a76d-0e7aa1053c3f-workload-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:51.154158 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.154131    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f7ff673-520b-4dd1-a76d-0e7aa1053c3f-kube-api-access-x5kvd" pod "8f7ff673-520b-4dd1-a76d-0e7aa1053c3f" (UID: "8f7ff673-520b-4dd1-a76d-0e7aa1053c3f"). InnerVolumeSpecName "kube-api-access-x5kvd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:27:51.156136 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.155763    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28aada86-170c-4935-80ed-e8c5ee503c14-kube-api-access-8n6x7" pod "28aada86-170c-4935-80ed-e8c5ee503c14" (UID: "28aada86-170c-4935-80ed-e8c5ee503c14"). InnerVolumeSpecName "kube-api-access-8n6x7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:27:51.156532 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.156489    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/28aada86-170c-4935-80ed-e8c5ee503c14-istio-podinfo" pod "28aada86-170c-4935-80ed-e8c5ee503c14" (UID: "28aada86-170c-4935-80ed-e8c5ee503c14"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:27:51.164795 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.160155    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28aada86-170c-4935-80ed-e8c5ee503c14-istio-token" pod "28aada86-170c-4935-80ed-e8c5ee503c14" (UID: "28aada86-170c-4935-80ed-e8c5ee503c14"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:27:51.167846 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.165989    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f7ff673-520b-4dd1-a76d-0e7aa1053c3f-istio-token" pod "8f7ff673-520b-4dd1-a76d-0e7aa1053c3f" (UID: "8f7ff673-520b-4dd1-a76d-0e7aa1053c3f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:27:51.183670 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.182196    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28aada86-170c-4935-80ed-e8c5ee503c14-istio-envoy" pod "28aada86-170c-4935-80ed-e8c5ee503c14" (UID: "28aada86-170c-4935-80ed-e8c5ee503c14"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:51.203101 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.203074    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-vzfw9"
Nov 03 20:27:51.206315 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.206247    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sf2lg"
Nov 03 20:27:51.211996 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.211973    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-c98nc"
Nov 03 20:27:51.221053 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.221026    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-qskf7"
Nov 03 20:27:51.253831 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.253792    5570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28aada86-170c-4935-80ed-e8c5ee503c14-istio-podinfo\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:51.253831 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.253822    5570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28aada86-170c-4935-80ed-e8c5ee503c14-istio-token\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:51.253831 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.253837    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5kvd\" (UniqueName: \"kubernetes.io/projected/8f7ff673-520b-4dd1-a76d-0e7aa1053c3f-kube-api-access-x5kvd\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:51.254141 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.253852    5570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f7ff673-520b-4dd1-a76d-0e7aa1053c3f-istio-token\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:51.254141 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.253867    5570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28aada86-170c-4935-80ed-e8c5ee503c14-istio-envoy\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:51.254141 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.253879    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8n6x7\" (UniqueName: \"kubernetes.io/projected/28aada86-170c-4935-80ed-e8c5ee503c14-kube-api-access-8n6x7\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:51.354558 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.354523    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a739707b-9c5a-4c9b-90df-501c5dd0c98d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a739707b-9c5a-4c9b-90df-501c5dd0c98d-istio-podinfo\") pod \"a739707b-9c5a-4c9b-90df-501c5dd0c98d\" (UID: \"a739707b-9c5a-4c9b-90df-501c5dd0c98d\") "
Nov 03 20:27:51.355091 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.354620    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/755f6eab-5d03-4368-9670-b1402084474c-kube-api-access-bsbh2\" (UniqueName: \"kubernetes.io/projected/755f6eab-5d03-4368-9670-b1402084474c-kube-api-access-bsbh2\") pod \"755f6eab-5d03-4368-9670-b1402084474c\" (UID: \"755f6eab-5d03-4368-9670-b1402084474c\") "
Nov 03 20:27:51.355091 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.354669    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a739707b-9c5a-4c9b-90df-501c5dd0c98d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a739707b-9c5a-4c9b-90df-501c5dd0c98d-credential-socket\") pod \"a739707b-9c5a-4c9b-90df-501c5dd0c98d\" (UID: \"a739707b-9c5a-4c9b-90df-501c5dd0c98d\") "
Nov 03 20:27:51.355091 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.354698    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a739707b-9c5a-4c9b-90df-501c5dd0c98d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a739707b-9c5a-4c9b-90df-501c5dd0c98d-istio-envoy\") pod \"a739707b-9c5a-4c9b-90df-501c5dd0c98d\" (UID: \"a739707b-9c5a-4c9b-90df-501c5dd0c98d\") "
Nov 03 20:27:51.355091 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.354727    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/469fc281-f7ee-4584-b3b6-85c6a669f806-istio-token\" (UniqueName: \"kubernetes.io/projected/469fc281-f7ee-4584-b3b6-85c6a669f806-istio-token\") pod \"469fc281-f7ee-4584-b3b6-85c6a669f806\" (UID: \"469fc281-f7ee-4584-b3b6-85c6a669f806\") "
Nov 03 20:27:51.355091 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.354759    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/469fc281-f7ee-4584-b3b6-85c6a669f806-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/469fc281-f7ee-4584-b3b6-85c6a669f806-istio-podinfo\") pod \"469fc281-f7ee-4584-b3b6-85c6a669f806\" (UID: \"469fc281-f7ee-4584-b3b6-85c6a669f806\") "
Nov 03 20:27:51.355091 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.354817    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07093a96-d60c-490f-910b-734f41752c17-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07093a96-d60c-490f-910b-734f41752c17-workload-certs\") pod \"07093a96-d60c-490f-910b-734f41752c17\" (UID: \"07093a96-d60c-490f-910b-734f41752c17\") "
Nov 03 20:27:51.355091 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.354849    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07093a96-d60c-490f-910b-734f41752c17-istio-token\" (UniqueName: \"kubernetes.io/projected/07093a96-d60c-490f-910b-734f41752c17-istio-token\") pod \"07093a96-d60c-490f-910b-734f41752c17\" (UID: \"07093a96-d60c-490f-910b-734f41752c17\") "
Nov 03 20:27:51.355091 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.354878    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/755f6eab-5d03-4368-9670-b1402084474c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/755f6eab-5d03-4368-9670-b1402084474c-istio-envoy\") pod \"755f6eab-5d03-4368-9670-b1402084474c\" (UID: \"755f6eab-5d03-4368-9670-b1402084474c\") "
Nov 03 20:27:51.355091 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.354915    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/469fc281-f7ee-4584-b3b6-85c6a669f806-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/469fc281-f7ee-4584-b3b6-85c6a669f806-istio-envoy\") pod \"469fc281-f7ee-4584-b3b6-85c6a669f806\" (UID: \"469fc281-f7ee-4584-b3b6-85c6a669f806\") "
Nov 03 20:27:51.355091 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.354990    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07093a96-d60c-490f-910b-734f41752c17-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07093a96-d60c-490f-910b-734f41752c17-istiod-ca-cert\") pod \"07093a96-d60c-490f-910b-734f41752c17\" (UID: \"07093a96-d60c-490f-910b-734f41752c17\") "
Nov 03 20:27:51.355091 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.355048    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07093a96-d60c-490f-910b-734f41752c17-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07093a96-d60c-490f-910b-734f41752c17-istio-data\") pod \"07093a96-d60c-490f-910b-734f41752c17\" (UID: \"07093a96-d60c-490f-910b-734f41752c17\") "
Nov 03 20:27:51.355091 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.355092    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/07093a96-d60c-490f-910b-734f41752c17-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07093a96-d60c-490f-910b-734f41752c17-istio-podinfo\") pod \"07093a96-d60c-490f-910b-734f41752c17\" (UID: \"07093a96-d60c-490f-910b-734f41752c17\") "
Nov 03 20:27:51.355709 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.355132    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/469fc281-f7ee-4584-b3b6-85c6a669f806-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/469fc281-f7ee-4584-b3b6-85c6a669f806-credential-socket\") pod \"469fc281-f7ee-4584-b3b6-85c6a669f806\" (UID: \"469fc281-f7ee-4584-b3b6-85c6a669f806\") "
Nov 03 20:27:51.355709 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.355198    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/755f6eab-5d03-4368-9670-b1402084474c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/755f6eab-5d03-4368-9670-b1402084474c-workload-socket\") pod \"755f6eab-5d03-4368-9670-b1402084474c\" (UID: \"755f6eab-5d03-4368-9670-b1402084474c\") "
Nov 03 20:27:51.355709 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.355260    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a739707b-9c5a-4c9b-90df-501c5dd0c98d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a739707b-9c5a-4c9b-90df-501c5dd0c98d-istiod-ca-cert\") pod \"a739707b-9c5a-4c9b-90df-501c5dd0c98d\" (UID: \"a739707b-9c5a-4c9b-90df-501c5dd0c98d\") "
Nov 03 20:27:51.355709 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.355327    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/755f6eab-5d03-4368-9670-b1402084474c-istio-token\" (UniqueName: \"kubernetes.io/projected/755f6eab-5d03-4368-9670-b1402084474c-istio-token\") pod \"755f6eab-5d03-4368-9670-b1402084474c\" (UID: \"755f6eab-5d03-4368-9670-b1402084474c\") "
Nov 03 20:27:51.355709 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.355406    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/755f6eab-5d03-4368-9670-b1402084474c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/755f6eab-5d03-4368-9670-b1402084474c-workload-certs\") pod \"755f6eab-5d03-4368-9670-b1402084474c\" (UID: \"755f6eab-5d03-4368-9670-b1402084474c\") "
Nov 03 20:27:51.355709 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.355453    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07093a96-d60c-490f-910b-734f41752c17-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07093a96-d60c-490f-910b-734f41752c17-istio-envoy\") pod \"07093a96-d60c-490f-910b-734f41752c17\" (UID: \"07093a96-d60c-490f-910b-734f41752c17\") "
Nov 03 20:27:51.355709 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.355487    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/469fc281-f7ee-4584-b3b6-85c6a669f806-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/469fc281-f7ee-4584-b3b6-85c6a669f806-istio-data\") pod \"469fc281-f7ee-4584-b3b6-85c6a669f806\" (UID: \"469fc281-f7ee-4584-b3b6-85c6a669f806\") "
Nov 03 20:27:51.355709 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.355523    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07093a96-d60c-490f-910b-734f41752c17-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07093a96-d60c-490f-910b-734f41752c17-credential-socket\") pod \"07093a96-d60c-490f-910b-734f41752c17\" (UID: \"07093a96-d60c-490f-910b-734f41752c17\") "
Nov 03 20:27:51.355709 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.355567    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07093a96-d60c-490f-910b-734f41752c17-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07093a96-d60c-490f-910b-734f41752c17-workload-socket\") pod \"07093a96-d60c-490f-910b-734f41752c17\" (UID: \"07093a96-d60c-490f-910b-734f41752c17\") "
Nov 03 20:27:51.355709 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.355598    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/469fc281-f7ee-4584-b3b6-85c6a669f806-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/469fc281-f7ee-4584-b3b6-85c6a669f806-istiod-ca-cert\") pod \"469fc281-f7ee-4584-b3b6-85c6a669f806\" (UID: \"469fc281-f7ee-4584-b3b6-85c6a669f806\") "
Nov 03 20:27:51.355709 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.355645    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07093a96-d60c-490f-910b-734f41752c17-kube-api-access-jtwv6\" (UniqueName: \"kubernetes.io/projected/07093a96-d60c-490f-910b-734f41752c17-kube-api-access-jtwv6\") pod \"07093a96-d60c-490f-910b-734f41752c17\" (UID: \"07093a96-d60c-490f-910b-734f41752c17\") "
Nov 03 20:27:51.355709 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.355678    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/469fc281-f7ee-4584-b3b6-85c6a669f806-kube-api-access-6pwb2\" (UniqueName: \"kubernetes.io/projected/469fc281-f7ee-4584-b3b6-85c6a669f806-kube-api-access-6pwb2\") pod \"469fc281-f7ee-4584-b3b6-85c6a669f806\" (UID: \"469fc281-f7ee-4584-b3b6-85c6a669f806\") "
Nov 03 20:27:51.356352 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.355803    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a739707b-9c5a-4c9b-90df-501c5dd0c98d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a739707b-9c5a-4c9b-90df-501c5dd0c98d-istio-data\") pod \"a739707b-9c5a-4c9b-90df-501c5dd0c98d\" (UID: \"a739707b-9c5a-4c9b-90df-501c5dd0c98d\") "
Nov 03 20:27:51.356352 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.355836    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/755f6eab-5d03-4368-9670-b1402084474c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/755f6eab-5d03-4368-9670-b1402084474c-istiod-ca-cert\") pod \"755f6eab-5d03-4368-9670-b1402084474c\" (UID: \"755f6eab-5d03-4368-9670-b1402084474c\") "
Nov 03 20:27:51.356352 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.355895    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/469fc281-f7ee-4584-b3b6-85c6a669f806-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/469fc281-f7ee-4584-b3b6-85c6a669f806-workload-socket\") pod \"469fc281-f7ee-4584-b3b6-85c6a669f806\" (UID: \"469fc281-f7ee-4584-b3b6-85c6a669f806\") "
Nov 03 20:27:51.356352 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.355928    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a739707b-9c5a-4c9b-90df-501c5dd0c98d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a739707b-9c5a-4c9b-90df-501c5dd0c98d-workload-certs\") pod \"a739707b-9c5a-4c9b-90df-501c5dd0c98d\" (UID: \"a739707b-9c5a-4c9b-90df-501c5dd0c98d\") "
Nov 03 20:27:51.356352 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.355965    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/755f6eab-5d03-4368-9670-b1402084474c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/755f6eab-5d03-4368-9670-b1402084474c-istio-podinfo\") pod \"755f6eab-5d03-4368-9670-b1402084474c\" (UID: \"755f6eab-5d03-4368-9670-b1402084474c\") "
Nov 03 20:27:51.356352 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.356000    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a739707b-9c5a-4c9b-90df-501c5dd0c98d-istio-token\" (UniqueName: \"kubernetes.io/projected/a739707b-9c5a-4c9b-90df-501c5dd0c98d-istio-token\") pod \"a739707b-9c5a-4c9b-90df-501c5dd0c98d\" (UID: \"a739707b-9c5a-4c9b-90df-501c5dd0c98d\") "
Nov 03 20:27:51.356352 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.356030    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/755f6eab-5d03-4368-9670-b1402084474c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/755f6eab-5d03-4368-9670-b1402084474c-istio-data\") pod \"755f6eab-5d03-4368-9670-b1402084474c\" (UID: \"755f6eab-5d03-4368-9670-b1402084474c\") "
Nov 03 20:27:51.356352 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.356066    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/469fc281-f7ee-4584-b3b6-85c6a669f806-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/469fc281-f7ee-4584-b3b6-85c6a669f806-workload-certs\") pod \"469fc281-f7ee-4584-b3b6-85c6a669f806\" (UID: \"469fc281-f7ee-4584-b3b6-85c6a669f806\") "
Nov 03 20:27:51.356352 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.356097    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a739707b-9c5a-4c9b-90df-501c5dd0c98d-kube-api-access-879mv\" (UniqueName: \"kubernetes.io/projected/a739707b-9c5a-4c9b-90df-501c5dd0c98d-kube-api-access-879mv\") pod \"a739707b-9c5a-4c9b-90df-501c5dd0c98d\" (UID: \"a739707b-9c5a-4c9b-90df-501c5dd0c98d\") "
Nov 03 20:27:51.356352 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.356128    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a739707b-9c5a-4c9b-90df-501c5dd0c98d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a739707b-9c5a-4c9b-90df-501c5dd0c98d-workload-socket\") pod \"a739707b-9c5a-4c9b-90df-501c5dd0c98d\" (UID: \"a739707b-9c5a-4c9b-90df-501c5dd0c98d\") "
Nov 03 20:27:51.356352 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.356182    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/755f6eab-5d03-4368-9670-b1402084474c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/755f6eab-5d03-4368-9670-b1402084474c-credential-socket\") pod \"755f6eab-5d03-4368-9670-b1402084474c\" (UID: \"755f6eab-5d03-4368-9670-b1402084474c\") "
Nov 03 20:27:51.358002 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.357338    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07093a96-d60c-490f-910b-734f41752c17-workload-certs" pod "07093a96-d60c-490f-910b-734f41752c17" (UID: "07093a96-d60c-490f-910b-734f41752c17"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:51.358002 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.357539    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/755f6eab-5d03-4368-9670-b1402084474c-credential-socket" pod "755f6eab-5d03-4368-9670-b1402084474c" (UID: "755f6eab-5d03-4368-9670-b1402084474c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:51.359416 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.359391    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/755f6eab-5d03-4368-9670-b1402084474c-kube-api-access-bsbh2" pod "755f6eab-5d03-4368-9670-b1402084474c" (UID: "755f6eab-5d03-4368-9670-b1402084474c"). InnerVolumeSpecName "kube-api-access-bsbh2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:27:51.359584 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.359068    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/755f6eab-5d03-4368-9670-b1402084474c-istio-envoy" pod "755f6eab-5d03-4368-9670-b1402084474c" (UID: "755f6eab-5d03-4368-9670-b1402084474c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:51.360876 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.360851    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a739707b-9c5a-4c9b-90df-501c5dd0c98d-credential-socket" pod "a739707b-9c5a-4c9b-90df-501c5dd0c98d" (UID: "a739707b-9c5a-4c9b-90df-501c5dd0c98d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:51.361249 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.361221    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a739707b-9c5a-4c9b-90df-501c5dd0c98d-workload-certs" pod "a739707b-9c5a-4c9b-90df-501c5dd0c98d" (UID: "a739707b-9c5a-4c9b-90df-501c5dd0c98d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:51.362224 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.362174    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a739707b-9c5a-4c9b-90df-501c5dd0c98d-istio-podinfo" pod "a739707b-9c5a-4c9b-90df-501c5dd0c98d" (UID: "a739707b-9c5a-4c9b-90df-501c5dd0c98d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:27:51.362224 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.362212    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/469fc281-f7ee-4584-b3b6-85c6a669f806-istio-envoy" pod "469fc281-f7ee-4584-b3b6-85c6a669f806" (UID: "469fc281-f7ee-4584-b3b6-85c6a669f806"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:51.362394 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.362308    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/469fc281-f7ee-4584-b3b6-85c6a669f806-istiod-ca-cert" pod "469fc281-f7ee-4584-b3b6-85c6a669f806" (UID: "469fc281-f7ee-4584-b3b6-85c6a669f806"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:27:51.362742 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.362716    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07093a96-d60c-490f-910b-734f41752c17-istiod-ca-cert" pod "07093a96-d60c-490f-910b-734f41752c17" (UID: "07093a96-d60c-490f-910b-734f41752c17"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:27:51.363044 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.363021    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07093a96-d60c-490f-910b-734f41752c17-istio-data" pod "07093a96-d60c-490f-910b-734f41752c17" (UID: "07093a96-d60c-490f-910b-734f41752c17"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:51.364911 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.364536    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/755f6eab-5d03-4368-9670-b1402084474c-istio-podinfo" pod "755f6eab-5d03-4368-9670-b1402084474c" (UID: "755f6eab-5d03-4368-9670-b1402084474c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:27:51.364911 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.364563    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07093a96-d60c-490f-910b-734f41752c17-kube-api-access-jtwv6" pod "07093a96-d60c-490f-910b-734f41752c17" (UID: "07093a96-d60c-490f-910b-734f41752c17"). InnerVolumeSpecName "kube-api-access-jtwv6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:27:51.365570 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.365534    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a739707b-9c5a-4c9b-90df-501c5dd0c98d-istio-envoy" pod "a739707b-9c5a-4c9b-90df-501c5dd0c98d" (UID: "a739707b-9c5a-4c9b-90df-501c5dd0c98d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:51.365570 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.365550    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/07093a96-d60c-490f-910b-734f41752c17-istio-podinfo" pod "07093a96-d60c-490f-910b-734f41752c17" (UID: "07093a96-d60c-490f-910b-734f41752c17"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:27:51.365932 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.365907    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/469fc281-f7ee-4584-b3b6-85c6a669f806-credential-socket" pod "469fc281-f7ee-4584-b3b6-85c6a669f806" (UID: "469fc281-f7ee-4584-b3b6-85c6a669f806"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:51.366040 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.366014    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/755f6eab-5d03-4368-9670-b1402084474c-istiod-ca-cert" pod "755f6eab-5d03-4368-9670-b1402084474c" (UID: "755f6eab-5d03-4368-9670-b1402084474c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:27:51.366164 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.366142    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/755f6eab-5d03-4368-9670-b1402084474c-workload-socket" pod "755f6eab-5d03-4368-9670-b1402084474c" (UID: "755f6eab-5d03-4368-9670-b1402084474c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:51.366388 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.366320    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a739707b-9c5a-4c9b-90df-501c5dd0c98d-istio-data" pod "a739707b-9c5a-4c9b-90df-501c5dd0c98d" (UID: "a739707b-9c5a-4c9b-90df-501c5dd0c98d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:51.366487 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.366386    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/469fc281-f7ee-4584-b3b6-85c6a669f806-istio-token" pod "469fc281-f7ee-4584-b3b6-85c6a669f806" (UID: "469fc281-f7ee-4584-b3b6-85c6a669f806"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:27:51.366550 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.366504    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a739707b-9c5a-4c9b-90df-501c5dd0c98d-istiod-ca-cert" pod "a739707b-9c5a-4c9b-90df-501c5dd0c98d" (UID: "a739707b-9c5a-4c9b-90df-501c5dd0c98d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:27:51.366796 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.366683    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/469fc281-f7ee-4584-b3b6-85c6a669f806-istio-podinfo" pod "469fc281-f7ee-4584-b3b6-85c6a669f806" (UID: "469fc281-f7ee-4584-b3b6-85c6a669f806"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:27:51.368320 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.367218    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a739707b-9c5a-4c9b-90df-501c5dd0c98d-istio-token" pod "a739707b-9c5a-4c9b-90df-501c5dd0c98d" (UID: "a739707b-9c5a-4c9b-90df-501c5dd0c98d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:27:51.368320 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.367446    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/755f6eab-5d03-4368-9670-b1402084474c-workload-certs" pod "755f6eab-5d03-4368-9670-b1402084474c" (UID: "755f6eab-5d03-4368-9670-b1402084474c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:51.368320 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.367591    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/755f6eab-5d03-4368-9670-b1402084474c-istio-data" pod "755f6eab-5d03-4368-9670-b1402084474c" (UID: "755f6eab-5d03-4368-9670-b1402084474c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:51.368320 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.367725    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07093a96-d60c-490f-910b-734f41752c17-credential-socket" pod "07093a96-d60c-490f-910b-734f41752c17" (UID: "07093a96-d60c-490f-910b-734f41752c17"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:51.368320 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.367843    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/469fc281-f7ee-4584-b3b6-85c6a669f806-workload-socket" pod "469fc281-f7ee-4584-b3b6-85c6a669f806" (UID: "469fc281-f7ee-4584-b3b6-85c6a669f806"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:51.368320 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.367934    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/469fc281-f7ee-4584-b3b6-85c6a669f806-istio-data" pod "469fc281-f7ee-4584-b3b6-85c6a669f806" (UID: "469fc281-f7ee-4584-b3b6-85c6a669f806"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:51.368320 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.367971    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07093a96-d60c-490f-910b-734f41752c17-workload-socket" pod "07093a96-d60c-490f-910b-734f41752c17" (UID: "07093a96-d60c-490f-910b-734f41752c17"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:51.368320 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.368277    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/469fc281-f7ee-4584-b3b6-85c6a669f806-workload-certs" pod "469fc281-f7ee-4584-b3b6-85c6a669f806" (UID: "469fc281-f7ee-4584-b3b6-85c6a669f806"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:51.368874 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.368643    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/469fc281-f7ee-4584-b3b6-85c6a669f806-kube-api-access-6pwb2" pod "469fc281-f7ee-4584-b3b6-85c6a669f806" (UID: "469fc281-f7ee-4584-b3b6-85c6a669f806"). InnerVolumeSpecName "kube-api-access-6pwb2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:27:51.368874 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.368658    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a739707b-9c5a-4c9b-90df-501c5dd0c98d-workload-socket" pod "a739707b-9c5a-4c9b-90df-501c5dd0c98d" (UID: "a739707b-9c5a-4c9b-90df-501c5dd0c98d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:51.368874 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.368849    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07093a96-d60c-490f-910b-734f41752c17-istio-token" pod "07093a96-d60c-490f-910b-734f41752c17" (UID: "07093a96-d60c-490f-910b-734f41752c17"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:27:51.369040 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.369020    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/755f6eab-5d03-4368-9670-b1402084474c-istio-token" pod "755f6eab-5d03-4368-9670-b1402084474c" (UID: "755f6eab-5d03-4368-9670-b1402084474c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:27:51.370560 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.370535    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07093a96-d60c-490f-910b-734f41752c17-istio-envoy" pod "07093a96-d60c-490f-910b-734f41752c17" (UID: "07093a96-d60c-490f-910b-734f41752c17"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:51.408937 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.408895    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a739707b-9c5a-4c9b-90df-501c5dd0c98d-kube-api-access-879mv" pod "a739707b-9c5a-4c9b-90df-501c5dd0c98d" (UID: "a739707b-9c5a-4c9b-90df-501c5dd0c98d"). InnerVolumeSpecName "kube-api-access-879mv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:27:51.458709 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.458667    5570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/469fc281-f7ee-4584-b3b6-85c6a669f806-workload-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:51.458709 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.458702    5570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a739707b-9c5a-4c9b-90df-501c5dd0c98d-workload-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:51.458709 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.458714    5570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/755f6eab-5d03-4368-9670-b1402084474c-istio-podinfo\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:51.459072 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.458726    5570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a739707b-9c5a-4c9b-90df-501c5dd0c98d-istio-token\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:51.459072 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.458737    5570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/755f6eab-5d03-4368-9670-b1402084474c-istio-data\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:51.459072 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.458748    5570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/469fc281-f7ee-4584-b3b6-85c6a669f806-workload-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:51.459072 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.458760    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-879mv\" (UniqueName: \"kubernetes.io/projected/a739707b-9c5a-4c9b-90df-501c5dd0c98d-kube-api-access-879mv\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:51.459072 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.458787    5570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a739707b-9c5a-4c9b-90df-501c5dd0c98d-workload-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:51.459072 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.458801    5570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/755f6eab-5d03-4368-9670-b1402084474c-credential-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:51.459072 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.458814    5570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a739707b-9c5a-4c9b-90df-501c5dd0c98d-istio-podinfo\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:51.459072 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.458827    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bsbh2\" (UniqueName: \"kubernetes.io/projected/755f6eab-5d03-4368-9670-b1402084474c-kube-api-access-bsbh2\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:51.459072 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.458848    5570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a739707b-9c5a-4c9b-90df-501c5dd0c98d-credential-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:51.459072 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.458862    5570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a739707b-9c5a-4c9b-90df-501c5dd0c98d-istio-envoy\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:51.459072 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.458873    5570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/469fc281-f7ee-4584-b3b6-85c6a669f806-istio-token\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:51.459072 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.458884    5570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/469fc281-f7ee-4584-b3b6-85c6a669f806-istio-podinfo\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:51.459072 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.458896    5570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07093a96-d60c-490f-910b-734f41752c17-workload-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:51.459072 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.458907    5570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07093a96-d60c-490f-910b-734f41752c17-istio-token\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:51.459072 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.458921    5570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/755f6eab-5d03-4368-9670-b1402084474c-istio-envoy\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:51.459072 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.458932    5570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/469fc281-f7ee-4584-b3b6-85c6a669f806-istio-envoy\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:51.459072 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.458943    5570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07093a96-d60c-490f-910b-734f41752c17-istiod-ca-cert\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:51.459072 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.458954    5570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07093a96-d60c-490f-910b-734f41752c17-istio-data\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:51.459072 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.458967    5570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07093a96-d60c-490f-910b-734f41752c17-istio-podinfo\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:51.459072 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.458980    5570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/469fc281-f7ee-4584-b3b6-85c6a669f806-credential-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:51.459072 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.458993    5570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/755f6eab-5d03-4368-9670-b1402084474c-workload-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:51.459072 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.459007    5570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a739707b-9c5a-4c9b-90df-501c5dd0c98d-istiod-ca-cert\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:51.459072 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.459019    5570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/755f6eab-5d03-4368-9670-b1402084474c-istio-token\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:51.459072 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.459032    5570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/755f6eab-5d03-4368-9670-b1402084474c-workload-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:51.459072 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.459044    5570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07093a96-d60c-490f-910b-734f41752c17-istio-envoy\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:51.459072 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.459057    5570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/469fc281-f7ee-4584-b3b6-85c6a669f806-istio-data\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:51.459901 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.459069    5570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07093a96-d60c-490f-910b-734f41752c17-credential-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:51.459901 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.459101    5570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07093a96-d60c-490f-910b-734f41752c17-workload-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:51.459901 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.459116    5570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/469fc281-f7ee-4584-b3b6-85c6a669f806-istiod-ca-cert\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:51.459901 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.459129    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jtwv6\" (UniqueName: \"kubernetes.io/projected/07093a96-d60c-490f-910b-734f41752c17-kube-api-access-jtwv6\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:51.459901 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.459144    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6pwb2\" (UniqueName: \"kubernetes.io/projected/469fc281-f7ee-4584-b3b6-85c6a669f806-kube-api-access-6pwb2\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:51.459901 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.459159    5570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a739707b-9c5a-4c9b-90df-501c5dd0c98d-istio-data\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:51.459901 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.459173    5570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/755f6eab-5d03-4368-9670-b1402084474c-istiod-ca-cert\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:27:51.684503 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.684466    5570 generic.go:356] "Generic (PLEG): container finished" podID="8f7ff673-520b-4dd1-a76d-0e7aa1053c3f" containerID="c168feccf6d7ffe37ff51f11a79108fcc28d009b7994b2ad80a8a06cb49bd947" exitCode=0
Nov 03 20:27:51.685041 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.684594    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-b4zmq" event={"ID":"8f7ff673-520b-4dd1-a76d-0e7aa1053c3f","Type":"ContainerDied","Data":"c168feccf6d7ffe37ff51f11a79108fcc28d009b7994b2ad80a8a06cb49bd947"}
Nov 03 20:27:51.685041 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.684647    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-b4zmq" event={"ID":"8f7ff673-520b-4dd1-a76d-0e7aa1053c3f","Type":"ContainerDied","Data":"743236b596fb07582b4fc24ccf3b8e10f871c0fb0f63f7ce1bb5a5b47e890382"}
Nov 03 20:27:51.685041 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.684687    5570 scope.go:122] "RemoveContainer" containerID="c168feccf6d7ffe37ff51f11a79108fcc28d009b7994b2ad80a8a06cb49bd947"
Nov 03 20:27:51.685041 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.684940    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-b4zmq"
Nov 03 20:27:51.688305 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.688142    5570 generic.go:356] "Generic (PLEG): container finished" podID="a739707b-9c5a-4c9b-90df-501c5dd0c98d" containerID="c98ca04c2cb8bdbe656136e1cd8d0fcac1daa8fdb11f2f3858d8c7f463fa7537" exitCode=0
Nov 03 20:27:51.688305 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.688212    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-vzfw9"
Nov 03 20:27:51.688305 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.688226    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-vzfw9" event={"ID":"a739707b-9c5a-4c9b-90df-501c5dd0c98d","Type":"ContainerDied","Data":"c98ca04c2cb8bdbe656136e1cd8d0fcac1daa8fdb11f2f3858d8c7f463fa7537"}
Nov 03 20:27:51.688305 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.688271    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-vzfw9" event={"ID":"a739707b-9c5a-4c9b-90df-501c5dd0c98d","Type":"ContainerDied","Data":"89b406441abf91875aceffa1f69a33f268137f5fcf043e76144e49fe17fb3ac1"}
Nov 03 20:27:51.690586 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.690561    5570 generic.go:356] "Generic (PLEG): container finished" podID="469fc281-f7ee-4584-b3b6-85c6a669f806" containerID="04cc270bc0362b4abb3459a1a7771e91e258cc93bb68adfcdb883fbb3afe99bd" exitCode=0
Nov 03 20:27:51.691081 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.690639    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-qskf7" event={"ID":"469fc281-f7ee-4584-b3b6-85c6a669f806","Type":"ContainerDied","Data":"04cc270bc0362b4abb3459a1a7771e91e258cc93bb68adfcdb883fbb3afe99bd"}
Nov 03 20:27:51.691081 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.690812    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-qskf7" event={"ID":"469fc281-f7ee-4584-b3b6-85c6a669f806","Type":"ContainerDied","Data":"cdb58f004d531827d29b96dde040e3906021c454874c789ca6b67e00663dbcfa"}
Nov 03 20:27:51.691411 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.691388    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-qskf7"
Nov 03 20:27:51.692575 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.692357    5570 generic.go:356] "Generic (PLEG): container finished" podID="28aada86-170c-4935-80ed-e8c5ee503c14" containerID="f8dd680f3a7881f299314139130aacee319f496bfe291e09b3d07a6d33658515" exitCode=0
Nov 03 20:27:51.692575 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.692408    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-kmqsw"
Nov 03 20:27:51.692575 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.692496    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-kmqsw" event={"ID":"28aada86-170c-4935-80ed-e8c5ee503c14","Type":"ContainerDied","Data":"f8dd680f3a7881f299314139130aacee319f496bfe291e09b3d07a6d33658515"}
Nov 03 20:27:51.692575 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.692537    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-kmqsw" event={"ID":"28aada86-170c-4935-80ed-e8c5ee503c14","Type":"ContainerDied","Data":"5969452b5e187cb57129ed9a6482c4a56325d5ef9259c66fbae67df3976f53e5"}
Nov 03 20:27:51.695236 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.694754    5570 generic.go:356] "Generic (PLEG): container finished" podID="755f6eab-5d03-4368-9670-b1402084474c" containerID="a414a503eff894ce1fa7aaee149e172cf442c1cd2ba6f8cd05579632d356f512" exitCode=0
Nov 03 20:27:51.695236 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.694923    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-c98nc" event={"ID":"755f6eab-5d03-4368-9670-b1402084474c","Type":"ContainerDied","Data":"a414a503eff894ce1fa7aaee149e172cf442c1cd2ba6f8cd05579632d356f512"}
Nov 03 20:27:51.695236 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.694967    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-c98nc" event={"ID":"755f6eab-5d03-4368-9670-b1402084474c","Type":"ContainerDied","Data":"e3767356ae7f055f3f8ffb0b6a7d0d90d09e33d2e9e54712809d90f637d27725"}
Nov 03 20:27:51.695236 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.695044    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-c98nc"
Nov 03 20:27:51.706693 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.706658    5570 generic.go:356] "Generic (PLEG): container finished" podID="07093a96-d60c-490f-910b-734f41752c17" containerID="52cc5ce0dff70b1613a235dd886d31d1c1e2960a2c37ef9b6eeff114ec29d26a" exitCode=0
Nov 03 20:27:51.706858 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.706690    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sf2lg" event={"ID":"07093a96-d60c-490f-910b-734f41752c17","Type":"ContainerDied","Data":"52cc5ce0dff70b1613a235dd886d31d1c1e2960a2c37ef9b6eeff114ec29d26a"}
Nov 03 20:27:51.706858 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.706756    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sf2lg" event={"ID":"07093a96-d60c-490f-910b-734f41752c17","Type":"ContainerDied","Data":"2f0a020b0fbcc4d7ff6676b2bbb635d5f5f89249504ac614149acab58ca6525a"}
Nov 03 20:27:51.706858 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.706710    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sf2lg"
Nov 03 20:27:51.736192 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.736149    5570 scope.go:122] "RemoveContainer" containerID="c168feccf6d7ffe37ff51f11a79108fcc28d009b7994b2ad80a8a06cb49bd947"
Nov 03 20:27:51.736682 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:27:51.736658    5570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c168feccf6d7ffe37ff51f11a79108fcc28d009b7994b2ad80a8a06cb49bd947\": container with ID starting with c168feccf6d7ffe37ff51f11a79108fcc28d009b7994b2ad80a8a06cb49bd947 not found: ID does not exist" containerID="c168feccf6d7ffe37ff51f11a79108fcc28d009b7994b2ad80a8a06cb49bd947"
Nov 03 20:27:51.737870 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.736696    5570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c168feccf6d7ffe37ff51f11a79108fcc28d009b7994b2ad80a8a06cb49bd947"} err="failed to get container status \"c168feccf6d7ffe37ff51f11a79108fcc28d009b7994b2ad80a8a06cb49bd947\": rpc error: code = NotFound desc = could not find container \"c168feccf6d7ffe37ff51f11a79108fcc28d009b7994b2ad80a8a06cb49bd947\": container with ID starting with c168feccf6d7ffe37ff51f11a79108fcc28d009b7994b2ad80a8a06cb49bd947 not found: ID does not exist"
Nov 03 20:27:51.737870 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.736748    5570 scope.go:122] "RemoveContainer" containerID="c98ca04c2cb8bdbe656136e1cd8d0fcac1daa8fdb11f2f3858d8c7f463fa7537"
Nov 03 20:27:51.750276 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.750253    5570 scope.go:122] "RemoveContainer" containerID="016c483e8a6ffb6d372c0a550482b3dc7344bd52f597fb28b1466c5c5565141a"
Nov 03 20:27:51.763569 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.763546    5570 scope.go:122] "RemoveContainer" containerID="c98ca04c2cb8bdbe656136e1cd8d0fcac1daa8fdb11f2f3858d8c7f463fa7537"
Nov 03 20:27:51.763944 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:27:51.763912    5570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c98ca04c2cb8bdbe656136e1cd8d0fcac1daa8fdb11f2f3858d8c7f463fa7537\": container with ID starting with c98ca04c2cb8bdbe656136e1cd8d0fcac1daa8fdb11f2f3858d8c7f463fa7537 not found: ID does not exist" containerID="c98ca04c2cb8bdbe656136e1cd8d0fcac1daa8fdb11f2f3858d8c7f463fa7537"
Nov 03 20:27:51.764047 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.763951    5570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c98ca04c2cb8bdbe656136e1cd8d0fcac1daa8fdb11f2f3858d8c7f463fa7537"} err="failed to get container status \"c98ca04c2cb8bdbe656136e1cd8d0fcac1daa8fdb11f2f3858d8c7f463fa7537\": rpc error: code = NotFound desc = could not find container \"c98ca04c2cb8bdbe656136e1cd8d0fcac1daa8fdb11f2f3858d8c7f463fa7537\": container with ID starting with c98ca04c2cb8bdbe656136e1cd8d0fcac1daa8fdb11f2f3858d8c7f463fa7537 not found: ID does not exist"
Nov 03 20:27:51.764047 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.763977    5570 scope.go:122] "RemoveContainer" containerID="016c483e8a6ffb6d372c0a550482b3dc7344bd52f597fb28b1466c5c5565141a"
Nov 03 20:27:51.764368 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:27:51.764345    5570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"016c483e8a6ffb6d372c0a550482b3dc7344bd52f597fb28b1466c5c5565141a\": container with ID starting with 016c483e8a6ffb6d372c0a550482b3dc7344bd52f597fb28b1466c5c5565141a not found: ID does not exist" containerID="016c483e8a6ffb6d372c0a550482b3dc7344bd52f597fb28b1466c5c5565141a"
Nov 03 20:27:51.764423 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.764379    5570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"016c483e8a6ffb6d372c0a550482b3dc7344bd52f597fb28b1466c5c5565141a"} err="failed to get container status \"016c483e8a6ffb6d372c0a550482b3dc7344bd52f597fb28b1466c5c5565141a\": rpc error: code = NotFound desc = could not find container \"016c483e8a6ffb6d372c0a550482b3dc7344bd52f597fb28b1466c5c5565141a\": container with ID starting with 016c483e8a6ffb6d372c0a550482b3dc7344bd52f597fb28b1466c5c5565141a not found: ID does not exist"
Nov 03 20:27:51.764423 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.764402    5570 scope.go:122] "RemoveContainer" containerID="04cc270bc0362b4abb3459a1a7771e91e258cc93bb68adfcdb883fbb3afe99bd"
Nov 03 20:27:51.776890 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.776865    5570 scope.go:122] "RemoveContainer" containerID="04cc270bc0362b4abb3459a1a7771e91e258cc93bb68adfcdb883fbb3afe99bd"
Nov 03 20:27:51.777211 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:27:51.777190    5570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04cc270bc0362b4abb3459a1a7771e91e258cc93bb68adfcdb883fbb3afe99bd\": container with ID starting with 04cc270bc0362b4abb3459a1a7771e91e258cc93bb68adfcdb883fbb3afe99bd not found: ID does not exist" containerID="04cc270bc0362b4abb3459a1a7771e91e258cc93bb68adfcdb883fbb3afe99bd"
Nov 03 20:27:51.777291 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.777219    5570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04cc270bc0362b4abb3459a1a7771e91e258cc93bb68adfcdb883fbb3afe99bd"} err="failed to get container status \"04cc270bc0362b4abb3459a1a7771e91e258cc93bb68adfcdb883fbb3afe99bd\": rpc error: code = NotFound desc = could not find container \"04cc270bc0362b4abb3459a1a7771e91e258cc93bb68adfcdb883fbb3afe99bd\": container with ID starting with 04cc270bc0362b4abb3459a1a7771e91e258cc93bb68adfcdb883fbb3afe99bd not found: ID does not exist"
Nov 03 20:27:51.777291 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.777236    5570 scope.go:122] "RemoveContainer" containerID="f8dd680f3a7881f299314139130aacee319f496bfe291e09b3d07a6d33658515"
Nov 03 20:27:51.788169 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.788152    5570 scope.go:122] "RemoveContainer" containerID="f8dd680f3a7881f299314139130aacee319f496bfe291e09b3d07a6d33658515"
Nov 03 20:27:51.788452 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:27:51.788431    5570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8dd680f3a7881f299314139130aacee319f496bfe291e09b3d07a6d33658515\": container with ID starting with f8dd680f3a7881f299314139130aacee319f496bfe291e09b3d07a6d33658515 not found: ID does not exist" containerID="f8dd680f3a7881f299314139130aacee319f496bfe291e09b3d07a6d33658515"
Nov 03 20:27:51.788544 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.788458    5570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8dd680f3a7881f299314139130aacee319f496bfe291e09b3d07a6d33658515"} err="failed to get container status \"f8dd680f3a7881f299314139130aacee319f496bfe291e09b3d07a6d33658515\": rpc error: code = NotFound desc = could not find container \"f8dd680f3a7881f299314139130aacee319f496bfe291e09b3d07a6d33658515\": container with ID starting with f8dd680f3a7881f299314139130aacee319f496bfe291e09b3d07a6d33658515 not found: ID does not exist"
Nov 03 20:27:51.788544 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.788476    5570 scope.go:122] "RemoveContainer" containerID="a414a503eff894ce1fa7aaee149e172cf442c1cd2ba6f8cd05579632d356f512"
Nov 03 20:27:51.798289 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.798266    5570 scope.go:122] "RemoveContainer" containerID="a414a503eff894ce1fa7aaee149e172cf442c1cd2ba6f8cd05579632d356f512"
Nov 03 20:27:51.798564 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:27:51.798542    5570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a414a503eff894ce1fa7aaee149e172cf442c1cd2ba6f8cd05579632d356f512\": container with ID starting with a414a503eff894ce1fa7aaee149e172cf442c1cd2ba6f8cd05579632d356f512 not found: ID does not exist" containerID="a414a503eff894ce1fa7aaee149e172cf442c1cd2ba6f8cd05579632d356f512"
Nov 03 20:27:51.798641 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.798576    5570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a414a503eff894ce1fa7aaee149e172cf442c1cd2ba6f8cd05579632d356f512"} err="failed to get container status \"a414a503eff894ce1fa7aaee149e172cf442c1cd2ba6f8cd05579632d356f512\": rpc error: code = NotFound desc = could not find container \"a414a503eff894ce1fa7aaee149e172cf442c1cd2ba6f8cd05579632d356f512\": container with ID starting with a414a503eff894ce1fa7aaee149e172cf442c1cd2ba6f8cd05579632d356f512 not found: ID does not exist"
Nov 03 20:27:51.798641 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.798598    5570 scope.go:122] "RemoveContainer" containerID="52cc5ce0dff70b1613a235dd886d31d1c1e2960a2c37ef9b6eeff114ec29d26a"
Nov 03 20:27:51.809705 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.809542    5570 scope.go:122] "RemoveContainer" containerID="52cc5ce0dff70b1613a235dd886d31d1c1e2960a2c37ef9b6eeff114ec29d26a"
Nov 03 20:27:51.809878 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:27:51.809851    5570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52cc5ce0dff70b1613a235dd886d31d1c1e2960a2c37ef9b6eeff114ec29d26a\": container with ID starting with 52cc5ce0dff70b1613a235dd886d31d1c1e2960a2c37ef9b6eeff114ec29d26a not found: ID does not exist" containerID="52cc5ce0dff70b1613a235dd886d31d1c1e2960a2c37ef9b6eeff114ec29d26a"
Nov 03 20:27:51.809968 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.809886    5570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52cc5ce0dff70b1613a235dd886d31d1c1e2960a2c37ef9b6eeff114ec29d26a"} err="failed to get container status \"52cc5ce0dff70b1613a235dd886d31d1c1e2960a2c37ef9b6eeff114ec29d26a\": rpc error: code = NotFound desc = could not find container \"52cc5ce0dff70b1613a235dd886d31d1c1e2960a2c37ef9b6eeff114ec29d26a\": container with ID starting with 52cc5ce0dff70b1613a235dd886d31d1c1e2960a2c37ef9b6eeff114ec29d26a not found: ID does not exist"
Nov 03 20:27:51.814274 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.814253    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-vzfw9"]
Nov 03 20:27:51.819783 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.819747    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-vzfw9"]
Nov 03 20:27:51.856191 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.856161    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-c98nc"]
Nov 03 20:27:51.859517 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.859488    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-c98nc"]
Nov 03 20:27:51.894959 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.894925    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-qskf7"]
Nov 03 20:27:51.900993 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.900962    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-qskf7"]
Nov 03 20:27:51.936488 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.936456    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-b4zmq"]
Nov 03 20:27:51.941906 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.941877    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-b4zmq"]
Nov 03 20:27:51.977210 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.977173    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="469fc281-f7ee-4584-b3b6-85c6a669f806" path="/var/lib/kubelet/pods/469fc281-f7ee-4584-b3b6-85c6a669f806/volumes"
Nov 03 20:27:51.977730 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.977713    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="755f6eab-5d03-4368-9670-b1402084474c" path="/var/lib/kubelet/pods/755f6eab-5d03-4368-9670-b1402084474c/volumes"
Nov 03 20:27:51.978245 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.978231    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f7ff673-520b-4dd1-a76d-0e7aa1053c3f" path="/var/lib/kubelet/pods/8f7ff673-520b-4dd1-a76d-0e7aa1053c3f/volumes"
Nov 03 20:27:51.979104 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.979087    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a739707b-9c5a-4c9b-90df-501c5dd0c98d" path="/var/lib/kubelet/pods/a739707b-9c5a-4c9b-90df-501c5dd0c98d/volumes"
Nov 03 20:27:51.979482 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.979467    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sf2lg"]
Nov 03 20:27:51.984546 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:51.984487    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-sf2lg"]
Nov 03 20:27:52.018973 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:52.018941    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-kmqsw"]
Nov 03 20:27:52.028690 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:52.028648    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-kmqsw"]
Nov 03 20:27:53.966591 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:53.966555    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07093a96-d60c-490f-910b-734f41752c17" path="/var/lib/kubelet/pods/07093a96-d60c-490f-910b-734f41752c17/volumes"
Nov 03 20:27:53.967205 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:27:53.967187    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28aada86-170c-4935-80ed-e8c5ee503c14" path="/var/lib/kubelet/pods/28aada86-170c-4935-80ed-e8c5ee503c14/volumes"
Nov 03 20:27:57.229519 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:27:57.229478    5570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8bwhz" containerName="extractor"
Nov 03 20:28:14.039220 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:14.039182    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-addf1a-openshift-default-69ffb7968c-pgb22"]
Nov 03 20:28:14.040153 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:14.040116    5570 cpu_manager.go:414] "containerMap: removing container" podUID="469fc281-f7ee-4584-b3b6-85c6a669f806" containerName="istio-proxy"
Nov 03 20:28:14.040153 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:14.040139    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="469fc281-f7ee-4584-b3b6-85c6a669f806" containerName="istio-proxy"
Nov 03 20:28:14.040153 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:14.040153    5570 cpu_manager.go:414] "containerMap: removing container" podUID="8f7ff673-520b-4dd1-a76d-0e7aa1053c3f" containerName="istio-proxy"
Nov 03 20:28:14.040153 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:14.040159    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f7ff673-520b-4dd1-a76d-0e7aa1053c3f" containerName="istio-proxy"
Nov 03 20:28:14.040461 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:14.040172    5570 cpu_manager.go:414] "containerMap: removing container" podUID="28aada86-170c-4935-80ed-e8c5ee503c14" containerName="istio-proxy"
Nov 03 20:28:14.040461 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:14.040178    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28aada86-170c-4935-80ed-e8c5ee503c14" containerName="istio-proxy"
Nov 03 20:28:14.040461 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:14.040188    5570 cpu_manager.go:414] "containerMap: removing container" podUID="a739707b-9c5a-4c9b-90df-501c5dd0c98d" containerName="istio-proxy"
Nov 03 20:28:14.040461 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:14.040192    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a739707b-9c5a-4c9b-90df-501c5dd0c98d" containerName="istio-proxy"
Nov 03 20:28:14.040461 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:14.040218    5570 cpu_manager.go:414] "containerMap: removing container" podUID="755f6eab-5d03-4368-9670-b1402084474c" containerName="istio-proxy"
Nov 03 20:28:14.040461 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:14.040223    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="755f6eab-5d03-4368-9670-b1402084474c" containerName="istio-proxy"
Nov 03 20:28:14.040461 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:14.040237    5570 cpu_manager.go:414] "containerMap: removing container" podUID="a739707b-9c5a-4c9b-90df-501c5dd0c98d" containerName="istio-proxy"
Nov 03 20:28:14.040461 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:14.040241    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a739707b-9c5a-4c9b-90df-501c5dd0c98d" containerName="istio-proxy"
Nov 03 20:28:14.040461 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:14.040247    5570 cpu_manager.go:414] "containerMap: removing container" podUID="07093a96-d60c-490f-910b-734f41752c17" containerName="istio-proxy"
Nov 03 20:28:14.040461 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:14.040251    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07093a96-d60c-490f-910b-734f41752c17" containerName="istio-proxy"
Nov 03 20:28:14.040461 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:14.040362    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="a739707b-9c5a-4c9b-90df-501c5dd0c98d" containerName="istio-proxy"
Nov 03 20:28:14.040461 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:14.040371    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="07093a96-d60c-490f-910b-734f41752c17" containerName="istio-proxy"
Nov 03 20:28:14.040461 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:14.040383    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="469fc281-f7ee-4584-b3b6-85c6a669f806" containerName="istio-proxy"
Nov 03 20:28:14.040461 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:14.040390    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f7ff673-520b-4dd1-a76d-0e7aa1053c3f" containerName="istio-proxy"
Nov 03 20:28:14.040461 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:14.040397    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="28aada86-170c-4935-80ed-e8c5ee503c14" containerName="istio-proxy"
Nov 03 20:28:14.040461 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:14.040408    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="755f6eab-5d03-4368-9670-b1402084474c" containerName="istio-proxy"
Nov 03 20:28:14.058374 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:14.058343    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-addf1a-openshift-default-69ffb7968c-pgb22"
Nov 03 20:28:14.076273 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:14.076231    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5d67e4b-c834-426f-8df5-8d6dc4ae77ab-istio-podinfo\") pod \"e2e-api-a-addf1a-openshift-default-69ffb7968c-pgb22\" (UID: \"e5d67e4b-c834-426f-8df5-8d6dc4ae77ab\") " pod="openshift-ingress/e2e-api-a-addf1a-openshift-default-69ffb7968c-pgb22"
Nov 03 20:28:14.076466 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:14.076313    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5d67e4b-c834-426f-8df5-8d6dc4ae77ab-istio-data\") pod \"e2e-api-a-addf1a-openshift-default-69ffb7968c-pgb22\" (UID: \"e5d67e4b-c834-426f-8df5-8d6dc4ae77ab\") " pod="openshift-ingress/e2e-api-a-addf1a-openshift-default-69ffb7968c-pgb22"
Nov 03 20:28:14.076466 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:14.076388    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5d67e4b-c834-426f-8df5-8d6dc4ae77ab-workload-certs\") pod \"e2e-api-a-addf1a-openshift-default-69ffb7968c-pgb22\" (UID: \"e5d67e4b-c834-426f-8df5-8d6dc4ae77ab\") " pod="openshift-ingress/e2e-api-a-addf1a-openshift-default-69ffb7968c-pgb22"
Nov 03 20:28:14.076583 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:14.076429    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5d67e4b-c834-426f-8df5-8d6dc4ae77ab-istio-token\") pod \"e2e-api-a-addf1a-openshift-default-69ffb7968c-pgb22\" (UID: \"e5d67e4b-c834-426f-8df5-8d6dc4ae77ab\") " pod="openshift-ingress/e2e-api-a-addf1a-openshift-default-69ffb7968c-pgb22"
Nov 03 20:28:14.076583 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:14.076575    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5d67e4b-c834-426f-8df5-8d6dc4ae77ab-workload-socket\") pod \"e2e-api-a-addf1a-openshift-default-69ffb7968c-pgb22\" (UID: \"e5d67e4b-c834-426f-8df5-8d6dc4ae77ab\") " pod="openshift-ingress/e2e-api-a-addf1a-openshift-default-69ffb7968c-pgb22"
Nov 03 20:28:14.076681 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:14.076643    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48wkg\" (UniqueName: \"kubernetes.io/projected/e5d67e4b-c834-426f-8df5-8d6dc4ae77ab-kube-api-access-48wkg\") pod \"e2e-api-a-addf1a-openshift-default-69ffb7968c-pgb22\" (UID: \"e5d67e4b-c834-426f-8df5-8d6dc4ae77ab\") " pod="openshift-ingress/e2e-api-a-addf1a-openshift-default-69ffb7968c-pgb22"
Nov 03 20:28:14.076731 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:14.076689    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5d67e4b-c834-426f-8df5-8d6dc4ae77ab-istio-envoy\") pod \"e2e-api-a-addf1a-openshift-default-69ffb7968c-pgb22\" (UID: \"e5d67e4b-c834-426f-8df5-8d6dc4ae77ab\") " pod="openshift-ingress/e2e-api-a-addf1a-openshift-default-69ffb7968c-pgb22"
Nov 03 20:28:14.076824 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:14.076806    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5d67e4b-c834-426f-8df5-8d6dc4ae77ab-credential-socket\") pod \"e2e-api-a-addf1a-openshift-default-69ffb7968c-pgb22\" (UID: \"e5d67e4b-c834-426f-8df5-8d6dc4ae77ab\") " pod="openshift-ingress/e2e-api-a-addf1a-openshift-default-69ffb7968c-pgb22"
Nov 03 20:28:14.076880 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:14.076866    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-addf1a-openshift-default-dockercfg-pvlsv\""
Nov 03 20:28:14.077155 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:14.077134    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5d67e4b-c834-426f-8df5-8d6dc4ae77ab-istiod-ca-cert\") pod \"e2e-api-a-addf1a-openshift-default-69ffb7968c-pgb22\" (UID: \"e5d67e4b-c834-426f-8df5-8d6dc4ae77ab\") " pod="openshift-ingress/e2e-api-a-addf1a-openshift-default-69ffb7968c-pgb22"
Nov 03 20:28:14.086558 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:14.086527    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-addf1a-openshift-default-69ffb7968c-pgb22"]
Nov 03 20:28:14.178995 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:14.178960    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5d67e4b-c834-426f-8df5-8d6dc4ae77ab-istio-envoy\") pod \"e2e-api-a-addf1a-openshift-default-69ffb7968c-pgb22\" (UID: \"e5d67e4b-c834-426f-8df5-8d6dc4ae77ab\") " pod="openshift-ingress/e2e-api-a-addf1a-openshift-default-69ffb7968c-pgb22"
Nov 03 20:28:14.179177 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:14.179039    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5d67e4b-c834-426f-8df5-8d6dc4ae77ab-credential-socket\") pod \"e2e-api-a-addf1a-openshift-default-69ffb7968c-pgb22\" (UID: \"e5d67e4b-c834-426f-8df5-8d6dc4ae77ab\") " pod="openshift-ingress/e2e-api-a-addf1a-openshift-default-69ffb7968c-pgb22"
Nov 03 20:28:14.179177 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:14.179139    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5d67e4b-c834-426f-8df5-8d6dc4ae77ab-istiod-ca-cert\") pod \"e2e-api-a-addf1a-openshift-default-69ffb7968c-pgb22\" (UID: \"e5d67e4b-c834-426f-8df5-8d6dc4ae77ab\") " pod="openshift-ingress/e2e-api-a-addf1a-openshift-default-69ffb7968c-pgb22"
Nov 03 20:28:14.179264 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:14.179214    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5d67e4b-c834-426f-8df5-8d6dc4ae77ab-istio-podinfo\") pod \"e2e-api-a-addf1a-openshift-default-69ffb7968c-pgb22\" (UID: \"e5d67e4b-c834-426f-8df5-8d6dc4ae77ab\") " pod="openshift-ingress/e2e-api-a-addf1a-openshift-default-69ffb7968c-pgb22"
Nov 03 20:28:14.179264 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:14.179252    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5d67e4b-c834-426f-8df5-8d6dc4ae77ab-istio-data\") pod \"e2e-api-a-addf1a-openshift-default-69ffb7968c-pgb22\" (UID: \"e5d67e4b-c834-426f-8df5-8d6dc4ae77ab\") " pod="openshift-ingress/e2e-api-a-addf1a-openshift-default-69ffb7968c-pgb22"
Nov 03 20:28:14.179373 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:14.179308    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5d67e4b-c834-426f-8df5-8d6dc4ae77ab-workload-certs\") pod \"e2e-api-a-addf1a-openshift-default-69ffb7968c-pgb22\" (UID: \"e5d67e4b-c834-426f-8df5-8d6dc4ae77ab\") " pod="openshift-ingress/e2e-api-a-addf1a-openshift-default-69ffb7968c-pgb22"
Nov 03 20:28:14.179488 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:14.179464    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5d67e4b-c834-426f-8df5-8d6dc4ae77ab-istio-token\") pod \"e2e-api-a-addf1a-openshift-default-69ffb7968c-pgb22\" (UID: \"e5d67e4b-c834-426f-8df5-8d6dc4ae77ab\") " pod="openshift-ingress/e2e-api-a-addf1a-openshift-default-69ffb7968c-pgb22"
Nov 03 20:28:14.179564 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:14.179510    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5d67e4b-c834-426f-8df5-8d6dc4ae77ab-credential-socket\") pod \"e2e-api-a-addf1a-openshift-default-69ffb7968c-pgb22\" (UID: \"e5d67e4b-c834-426f-8df5-8d6dc4ae77ab\") " pod="openshift-ingress/e2e-api-a-addf1a-openshift-default-69ffb7968c-pgb22"
Nov 03 20:28:14.179564 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:14.179516    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5d67e4b-c834-426f-8df5-8d6dc4ae77ab-workload-socket\") pod \"e2e-api-a-addf1a-openshift-default-69ffb7968c-pgb22\" (UID: \"e5d67e4b-c834-426f-8df5-8d6dc4ae77ab\") " pod="openshift-ingress/e2e-api-a-addf1a-openshift-default-69ffb7968c-pgb22"
Nov 03 20:28:14.179685 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:14.179588    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5d67e4b-c834-426f-8df5-8d6dc4ae77ab-workload-certs\") pod \"e2e-api-a-addf1a-openshift-default-69ffb7968c-pgb22\" (UID: \"e5d67e4b-c834-426f-8df5-8d6dc4ae77ab\") " pod="openshift-ingress/e2e-api-a-addf1a-openshift-default-69ffb7968c-pgb22"
Nov 03 20:28:14.179685 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:14.179606    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48wkg\" (UniqueName: \"kubernetes.io/projected/e5d67e4b-c834-426f-8df5-8d6dc4ae77ab-kube-api-access-48wkg\") pod \"e2e-api-a-addf1a-openshift-default-69ffb7968c-pgb22\" (UID: \"e5d67e4b-c834-426f-8df5-8d6dc4ae77ab\") " pod="openshift-ingress/e2e-api-a-addf1a-openshift-default-69ffb7968c-pgb22"
Nov 03 20:28:14.180037 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:14.180015    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5d67e4b-c834-426f-8df5-8d6dc4ae77ab-istiod-ca-cert\") pod \"e2e-api-a-addf1a-openshift-default-69ffb7968c-pgb22\" (UID: \"e5d67e4b-c834-426f-8df5-8d6dc4ae77ab\") " pod="openshift-ingress/e2e-api-a-addf1a-openshift-default-69ffb7968c-pgb22"
Nov 03 20:28:14.180135 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:14.180045    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5d67e4b-c834-426f-8df5-8d6dc4ae77ab-istio-data\") pod \"e2e-api-a-addf1a-openshift-default-69ffb7968c-pgb22\" (UID: \"e5d67e4b-c834-426f-8df5-8d6dc4ae77ab\") " pod="openshift-ingress/e2e-api-a-addf1a-openshift-default-69ffb7968c-pgb22"
Nov 03 20:28:14.180249 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:14.180228    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5d67e4b-c834-426f-8df5-8d6dc4ae77ab-workload-socket\") pod \"e2e-api-a-addf1a-openshift-default-69ffb7968c-pgb22\" (UID: \"e5d67e4b-c834-426f-8df5-8d6dc4ae77ab\") " pod="openshift-ingress/e2e-api-a-addf1a-openshift-default-69ffb7968c-pgb22"
Nov 03 20:28:14.181522 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:14.181499    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5d67e4b-c834-426f-8df5-8d6dc4ae77ab-istio-envoy\") pod \"e2e-api-a-addf1a-openshift-default-69ffb7968c-pgb22\" (UID: \"e5d67e4b-c834-426f-8df5-8d6dc4ae77ab\") " pod="openshift-ingress/e2e-api-a-addf1a-openshift-default-69ffb7968c-pgb22"
Nov 03 20:28:14.182086 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:14.182060    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5d67e4b-c834-426f-8df5-8d6dc4ae77ab-istio-podinfo\") pod \"e2e-api-a-addf1a-openshift-default-69ffb7968c-pgb22\" (UID: \"e5d67e4b-c834-426f-8df5-8d6dc4ae77ab\") " pod="openshift-ingress/e2e-api-a-addf1a-openshift-default-69ffb7968c-pgb22"
Nov 03 20:28:14.200151 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:14.200118    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5d67e4b-c834-426f-8df5-8d6dc4ae77ab-istio-token\") pod \"e2e-api-a-addf1a-openshift-default-69ffb7968c-pgb22\" (UID: \"e5d67e4b-c834-426f-8df5-8d6dc4ae77ab\") " pod="openshift-ingress/e2e-api-a-addf1a-openshift-default-69ffb7968c-pgb22"
Nov 03 20:28:14.200450 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:14.200425    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48wkg\" (UniqueName: \"kubernetes.io/projected/e5d67e4b-c834-426f-8df5-8d6dc4ae77ab-kube-api-access-48wkg\") pod \"e2e-api-a-addf1a-openshift-default-69ffb7968c-pgb22\" (UID: \"e5d67e4b-c834-426f-8df5-8d6dc4ae77ab\") " pod="openshift-ingress/e2e-api-a-addf1a-openshift-default-69ffb7968c-pgb22"
Nov 03 20:28:14.410852 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:14.409724    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-addf1a-openshift-default-69ffb7968c-pgb22"
Nov 03 20:28:14.922780 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:14.922744    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-addf1a-openshift-default-69ffb7968c-pgb22"]
Nov 03 20:28:14.923750 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:28:14.923720    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5d67e4b_c834_426f_8df5_8d6dc4ae77ab.slice/crio-08d353ae76158d0f9cd74eab287bde363c8bddab6fde6e2150fb1df959ccdd30 WatchSource:0}: Error finding container 08d353ae76158d0f9cd74eab287bde363c8bddab6fde6e2150fb1df959ccdd30: Status 404 returned error can't find the container with id 08d353ae76158d0f9cd74eab287bde363c8bddab6fde6e2150fb1df959ccdd30
Nov 03 20:28:14.926643 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:14.926605    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:28:14.926751 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:14.926696    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:28:14.926992 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:14.926746    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:28:15.021381 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:15.021332    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-addf1a-openshift-default-69ffb7968c-pgb22" event={"ID":"e5d67e4b-c834-426f-8df5-8d6dc4ae77ab","Type":"ContainerStarted","Data":"5ede59604e62a22821fb828b83994fcbd9b4102722a967137e0cf46fa579cc50"}
Nov 03 20:28:15.021613 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:15.021405    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-addf1a-openshift-default-69ffb7968c-pgb22" event={"ID":"e5d67e4b-c834-426f-8df5-8d6dc4ae77ab","Type":"ContainerStarted","Data":"08d353ae76158d0f9cd74eab287bde363c8bddab6fde6e2150fb1df959ccdd30"}
Nov 03 20:28:15.068155 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:15.068091    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-addf1a-openshift-default-69ffb7968c-pgb22" podStartSLOduration=1.068070302 podStartE2EDuration="1.068070302s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:28:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:28:15.064930945 +0000 UTC m=+1429.217592834" watchObservedRunningTime="2026-11-03 20:28:15.068070302 +0000 UTC m=+1429.220732191"
Nov 03 20:28:15.411487 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:15.411454    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-addf1a-openshift-default-69ffb7968c-pgb22"
Nov 03 20:28:15.413243 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:15.413216    5570 patch_prober.go:29] interesting pod/e2e-api-a-addf1a-openshift-default-69ffb7968c-pgb22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body=
Nov 03 20:28:15.413373 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:15.413279    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-addf1a-openshift-default-69ffb7968c-pgb22" podUID="e5d67e4b-c834-426f-8df5-8d6dc4ae77ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused"
Nov 03 20:28:16.279834 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:16.279787    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-dc7b94-777ff98d5f-4l8x2"]
Nov 03 20:28:16.416188 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:16.416162    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-addf1a-openshift-default-69ffb7968c-pgb22"
Nov 03 20:28:17.103819 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:17.102851    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-addf1a-openshift-default-69ffb7968c-pgb22"
Nov 03 20:28:17.118211 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:17.118183    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-addf1a-openshift-default-69ffb7968c-pgb22"
Nov 03 20:28:19.168936 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:19.168895    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-addf1a-5584bbb5cb-tv2db"]
Nov 03 20:28:19.169829 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:19.169810    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="a739707b-9c5a-4c9b-90df-501c5dd0c98d" containerName="istio-proxy"
Nov 03 20:28:19.173458 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:19.173441    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-addf1a-5584bbb5cb-tv2db"
Nov 03 20:28:19.189704 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:19.189679    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-addf1a-5584bbb5cb-tv2db"]
Nov 03 20:28:19.189954 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:19.189927    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16b1d5c2-f657-4041-b4e2-582d074725bd-plugins-config-volume\") pod \"payload-processing-e2e-api-a-addf1a-5584bbb5cb-tv2db\" (UID: \"16b1d5c2-f657-4041-b4e2-582d074725bd\") " pod="openshift-ingress/payload-processing-e2e-api-a-addf1a-5584bbb5cb-tv2db"
Nov 03 20:28:19.190233 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:19.190215    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbqbl\" (UniqueName: \"kubernetes.io/projected/16b1d5c2-f657-4041-b4e2-582d074725bd-kube-api-access-nbqbl\") pod \"payload-processing-e2e-api-a-addf1a-5584bbb5cb-tv2db\" (UID: \"16b1d5c2-f657-4041-b4e2-582d074725bd\") " pod="openshift-ingress/payload-processing-e2e-api-a-addf1a-5584bbb5cb-tv2db"
Nov 03 20:28:19.199135 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:19.199109    5570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-addf1a\""
Nov 03 20:28:19.238097 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:19.238068    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-addf1a-857995db79-q4sn9"]
Nov 03 20:28:19.242102 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:19.242073    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-addf1a-dockercfg-t5btm\""
Nov 03 20:28:19.242436 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:19.242419    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-addf1a-857995db79-q4sn9"
Nov 03 20:28:19.258649 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:19.258627    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-addf1a-857995db79-q4sn9"]
Nov 03 20:28:19.349655 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:19.345522    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16b1d5c2-f657-4041-b4e2-582d074725bd-plugins-config-volume\") pod \"payload-processing-e2e-api-a-addf1a-5584bbb5cb-tv2db\" (UID: \"16b1d5c2-f657-4041-b4e2-582d074725bd\") " pod="openshift-ingress/payload-processing-e2e-api-a-addf1a-5584bbb5cb-tv2db"
Nov 03 20:28:19.349655 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:19.345727    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbqbl\" (UniqueName: \"kubernetes.io/projected/16b1d5c2-f657-4041-b4e2-582d074725bd-kube-api-access-nbqbl\") pod \"payload-processing-e2e-api-a-addf1a-5584bbb5cb-tv2db\" (UID: \"16b1d5c2-f657-4041-b4e2-582d074725bd\") " pod="openshift-ingress/payload-processing-e2e-api-a-addf1a-5584bbb5cb-tv2db"
Nov 03 20:28:19.352045 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:19.345761    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nx6lz\" (UniqueName: \"kubernetes.io/projected/87cae337-afa6-4e03-99b6-2f0ff53c985a-kube-api-access-nx6lz\") pod \"payload-pre-processing-e2e-api-a-addf1a-857995db79-q4sn9\" (UID: \"87cae337-afa6-4e03-99b6-2f0ff53c985a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-addf1a-857995db79-q4sn9"
Nov 03 20:28:19.352045 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:19.350013    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87cae337-afa6-4e03-99b6-2f0ff53c985a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-addf1a-857995db79-q4sn9\" (UID: \"87cae337-afa6-4e03-99b6-2f0ff53c985a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-addf1a-857995db79-q4sn9"
Nov 03 20:28:19.352045 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:19.351104    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16b1d5c2-f657-4041-b4e2-582d074725bd-plugins-config-volume\") pod \"payload-processing-e2e-api-a-addf1a-5584bbb5cb-tv2db\" (UID: \"16b1d5c2-f657-4041-b4e2-582d074725bd\") " pod="openshift-ingress/payload-processing-e2e-api-a-addf1a-5584bbb5cb-tv2db"
Nov 03 20:28:19.452921 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:19.452878    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nx6lz\" (UniqueName: \"kubernetes.io/projected/87cae337-afa6-4e03-99b6-2f0ff53c985a-kube-api-access-nx6lz\") pod \"payload-pre-processing-e2e-api-a-addf1a-857995db79-q4sn9\" (UID: \"87cae337-afa6-4e03-99b6-2f0ff53c985a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-addf1a-857995db79-q4sn9"
Nov 03 20:28:19.453107 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:19.452933    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87cae337-afa6-4e03-99b6-2f0ff53c985a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-addf1a-857995db79-q4sn9\" (UID: \"87cae337-afa6-4e03-99b6-2f0ff53c985a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-addf1a-857995db79-q4sn9"
Nov 03 20:28:19.475814 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:19.467650    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87cae337-afa6-4e03-99b6-2f0ff53c985a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-addf1a-857995db79-q4sn9\" (UID: \"87cae337-afa6-4e03-99b6-2f0ff53c985a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-addf1a-857995db79-q4sn9"
Nov 03 20:28:19.540816 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:19.533358    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbqbl\" (UniqueName: \"kubernetes.io/projected/16b1d5c2-f657-4041-b4e2-582d074725bd-kube-api-access-nbqbl\") pod \"payload-processing-e2e-api-a-addf1a-5584bbb5cb-tv2db\" (UID: \"16b1d5c2-f657-4041-b4e2-582d074725bd\") " pod="openshift-ingress/payload-processing-e2e-api-a-addf1a-5584bbb5cb-tv2db"
Nov 03 20:28:19.641797 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:19.619894    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nx6lz\" (UniqueName: \"kubernetes.io/projected/87cae337-afa6-4e03-99b6-2f0ff53c985a-kube-api-access-nx6lz\") pod \"payload-pre-processing-e2e-api-a-addf1a-857995db79-q4sn9\" (UID: \"87cae337-afa6-4e03-99b6-2f0ff53c985a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-addf1a-857995db79-q4sn9"
Nov 03 20:28:19.791662 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:19.791571    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-addf1a-5584bbb5cb-tv2db"
Nov 03 20:28:19.881817 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:19.871128    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-addf1a-857995db79-q4sn9"
Nov 03 20:28:20.214051 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:20.214018    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-addf1a-5584bbb5cb-tv2db"]
Nov 03 20:28:20.216042 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:28:20.216011    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16b1d5c2_f657_4041_b4e2_582d074725bd.slice/crio-98c4d23a5c7a6918078c7ae1b09f8cfc8e02181235903c9df710f3037bf0fc51 WatchSource:0}: Error finding container 98c4d23a5c7a6918078c7ae1b09f8cfc8e02181235903c9df710f3037bf0fc51: Status 404 returned error can't find the container with id 98c4d23a5c7a6918078c7ae1b09f8cfc8e02181235903c9df710f3037bf0fc51
Nov 03 20:28:20.253681 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:20.253643    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-addf1a-857995db79-q4sn9"]
Nov 03 20:28:20.286345 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:28:20.286312    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87cae337_afa6_4e03_99b6_2f0ff53c985a.slice/crio-75af2602a7363e8868dd4c52eae28a369b313c49f2348a848f2a0fa8c6bd87c4 WatchSource:0}: Error finding container 75af2602a7363e8868dd4c52eae28a369b313c49f2348a848f2a0fa8c6bd87c4: Status 404 returned error can't find the container with id 75af2602a7363e8868dd4c52eae28a369b313c49f2348a848f2a0fa8c6bd87c4
Nov 03 20:28:21.129641 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:21.129602    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-addf1a-5584bbb5cb-tv2db" event={"ID":"16b1d5c2-f657-4041-b4e2-582d074725bd","Type":"ContainerStarted","Data":"1ee01d63ad2321edbf20229db31c8f2d0858af09c650d151f1adaa61b5494ce0"}
Nov 03 20:28:21.129876 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:21.129656    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-addf1a-5584bbb5cb-tv2db" event={"ID":"16b1d5c2-f657-4041-b4e2-582d074725bd","Type":"ContainerStarted","Data":"98c4d23a5c7a6918078c7ae1b09f8cfc8e02181235903c9df710f3037bf0fc51"}
Nov 03 20:28:21.129876 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:21.129830    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-addf1a-5584bbb5cb-tv2db"
Nov 03 20:28:21.131416 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:21.131392    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-addf1a-857995db79-q4sn9" event={"ID":"87cae337-afa6-4e03-99b6-2f0ff53c985a","Type":"ContainerStarted","Data":"aac616e6892e874ea3c74677d75ab45da1b4eedf93284aa5bd0fdcf17c3d996c"}
Nov 03 20:28:21.131548 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:21.131430    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-addf1a-857995db79-q4sn9" event={"ID":"87cae337-afa6-4e03-99b6-2f0ff53c985a","Type":"ContainerStarted","Data":"75af2602a7363e8868dd4c52eae28a369b313c49f2348a848f2a0fa8c6bd87c4"}
Nov 03 20:28:21.131548 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:21.131531    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-addf1a-857995db79-q4sn9"
Nov 03 20:28:21.171084 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:21.171029    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-addf1a-5584bbb5cb-tv2db" podStartSLOduration=2.171013259 podStartE2EDuration="2.171013259s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:28:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:28:21.168828472 +0000 UTC m=+1435.321490364" watchObservedRunningTime="2026-11-03 20:28:21.171013259 +0000 UTC m=+1435.323675124"
Nov 03 20:28:21.210171 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:21.210118    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-addf1a-857995db79-q4sn9" podStartSLOduration=2.210104024 podStartE2EDuration="2.210104024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:28:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:28:21.208838731 +0000 UTC m=+1435.361500619" watchObservedRunningTime="2026-11-03 20:28:21.210104024 +0000 UTC m=+1435.362765921"
Nov 03 20:28:21.281101 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:21.281060    5570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-dc7b94-777ff98d5f-4l8x2" podUID="edc8b223-be7f-441c-aaa6-d22974f93443" containerName="payload-pre-processing" containerID="cri-o://8ec3dd7ee1d9b56d9c6952e0000ca593092b590016ce9c7d7c857c04aeb26adf" gracePeriod=25
Nov 03 20:28:21.725910 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:21.725882    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-dc7b94-777ff98d5f-4l8x2"
Nov 03 20:28:21.811028 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:21.810990    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/edc8b223-be7f-441c-aaa6-d22974f93443-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/edc8b223-be7f-441c-aaa6-d22974f93443-plugins-config-volume\") pod \"edc8b223-be7f-441c-aaa6-d22974f93443\" (UID: \"edc8b223-be7f-441c-aaa6-d22974f93443\") "
Nov 03 20:28:21.811283 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:21.811265    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/edc8b223-be7f-441c-aaa6-d22974f93443-kube-api-access-x6t7b\" (UniqueName: \"kubernetes.io/projected/edc8b223-be7f-441c-aaa6-d22974f93443-kube-api-access-x6t7b\") pod \"edc8b223-be7f-441c-aaa6-d22974f93443\" (UID: \"edc8b223-be7f-441c-aaa6-d22974f93443\") "
Nov 03 20:28:21.811444 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:21.811415    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/edc8b223-be7f-441c-aaa6-d22974f93443-plugins-config-volume" pod "edc8b223-be7f-441c-aaa6-d22974f93443" (UID: "edc8b223-be7f-441c-aaa6-d22974f93443"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:28:21.811932 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:21.811910    5570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/edc8b223-be7f-441c-aaa6-d22974f93443-plugins-config-volume\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:28:21.814157 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:21.814130    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edc8b223-be7f-441c-aaa6-d22974f93443-kube-api-access-x6t7b" pod "edc8b223-be7f-441c-aaa6-d22974f93443" (UID: "edc8b223-be7f-441c-aaa6-d22974f93443"). InnerVolumeSpecName "kube-api-access-x6t7b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:28:21.913420 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:21.913387    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6t7b\" (UniqueName: \"kubernetes.io/projected/edc8b223-be7f-441c-aaa6-d22974f93443-kube-api-access-x6t7b\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:28:22.140344 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:22.140304    5570 generic.go:356] "Generic (PLEG): container finished" podID="edc8b223-be7f-441c-aaa6-d22974f93443" containerID="8ec3dd7ee1d9b56d9c6952e0000ca593092b590016ce9c7d7c857c04aeb26adf" exitCode=0
Nov 03 20:28:22.140520 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:22.140348    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-dc7b94-777ff98d5f-4l8x2"
Nov 03 20:28:22.140520 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:22.140388    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-dc7b94-777ff98d5f-4l8x2" event={"ID":"edc8b223-be7f-441c-aaa6-d22974f93443","Type":"ContainerDied","Data":"8ec3dd7ee1d9b56d9c6952e0000ca593092b590016ce9c7d7c857c04aeb26adf"}
Nov 03 20:28:22.140520 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:22.140440    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-dc7b94-777ff98d5f-4l8x2" event={"ID":"edc8b223-be7f-441c-aaa6-d22974f93443","Type":"ContainerDied","Data":"8a608bbd6d689f30d739676e402a3b638b6a820d4c0cbaffabffa7db08fce9ae"}
Nov 03 20:28:22.140520 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:22.140466    5570 scope.go:122] "RemoveContainer" containerID="8ec3dd7ee1d9b56d9c6952e0000ca593092b590016ce9c7d7c857c04aeb26adf"
Nov 03 20:28:22.151464 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:22.151440    5570 scope.go:122] "RemoveContainer" containerID="8ec3dd7ee1d9b56d9c6952e0000ca593092b590016ce9c7d7c857c04aeb26adf"
Nov 03 20:28:22.151723 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:28:22.151698    5570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ec3dd7ee1d9b56d9c6952e0000ca593092b590016ce9c7d7c857c04aeb26adf\": container with ID starting with 8ec3dd7ee1d9b56d9c6952e0000ca593092b590016ce9c7d7c857c04aeb26adf not found: ID does not exist" containerID="8ec3dd7ee1d9b56d9c6952e0000ca593092b590016ce9c7d7c857c04aeb26adf"
Nov 03 20:28:22.151856 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:22.151732    5570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ec3dd7ee1d9b56d9c6952e0000ca593092b590016ce9c7d7c857c04aeb26adf"} err="failed to get container status \"8ec3dd7ee1d9b56d9c6952e0000ca593092b590016ce9c7d7c857c04aeb26adf\": rpc error: code = NotFound desc = could not find container \"8ec3dd7ee1d9b56d9c6952e0000ca593092b590016ce9c7d7c857c04aeb26adf\": container with ID starting with 8ec3dd7ee1d9b56d9c6952e0000ca593092b590016ce9c7d7c857c04aeb26adf not found: ID does not exist"
Nov 03 20:28:22.195766 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:22.195725    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-dc7b94-777ff98d5f-4l8x2"]
Nov 03 20:28:22.208904 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:22.208872    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-dc7b94-777ff98d5f-4l8x2"]
Nov 03 20:28:23.966978 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:23.966942    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="edc8b223-be7f-441c-aaa6-d22974f93443" path="/var/lib/kubelet/pods/edc8b223-be7f-441c-aaa6-d22974f93443/volumes"
Nov 03 20:28:25.791373 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:25.791332    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g69f8"]
Nov 03 20:28:27.274756 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:28:27.274710    5570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8bwhz" containerName="extractor"
Nov 03 20:28:30.120105 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:30.120074    5570 scope.go:122] "RemoveContainer" containerID="aa7c85a41a5902ad05b98dc85d3ca3f9fea44ffd1a1413cd179f7c3edbece58e"
Nov 03 20:28:30.130798 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:30.130753    5570 scope.go:122] "RemoveContainer" containerID="170be724fae6bc1478f780b1bf297ccd3787593ede151862c62b912871839dca"
Nov 03 20:28:30.140495 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:30.140474    5570 scope.go:122] "RemoveContainer" containerID="218686cf266f775f69ae8515a4e96ff410b158739543489796ea28e215b5d176"
Nov 03 20:28:32.143893 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:32.143860    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-addf1a-857995db79-q4sn9"
Nov 03 20:28:32.144339 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:32.143955    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-addf1a-5584bbb5cb-tv2db"
Nov 03 20:28:32.917850 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:32.917812    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-6fc748/premium-w1-6fc748-kserve-7474db9cd7-nf2xk"]
Nov 03 20:28:33.574522 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:33.573200    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-bx2dr"]
Nov 03 20:28:33.585381 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:33.585337    5570 cpu_manager.go:414] "containerMap: removing container" podUID="edc8b223-be7f-441c-aaa6-d22974f93443" containerName="payload-pre-processing"
Nov 03 20:28:33.585620 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:33.585602    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="edc8b223-be7f-441c-aaa6-d22974f93443" containerName="payload-pre-processing"
Nov 03 20:28:33.586049 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:33.586030    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="edc8b223-be7f-441c-aaa6-d22974f93443" containerName="payload-pre-processing"
Nov 03 20:28:33.610799 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:33.606763    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-bx2dr"]
Nov 03 20:28:33.610799 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:33.606973    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-bx2dr"
Nov 03 20:28:33.845335 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:33.845087    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-c4a71d-openshift-default-dockercfg-gr5wd\""
Nov 03 20:28:33.977248 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:33.977205    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca63212a-77c0-4b97-8a23-5d0a518a55ef-istio-token\") pod \"e2e-api-a-c4a71d-openshift-default-7647f5cdbf-bx2dr\" (UID: \"ca63212a-77c0-4b97-8a23-5d0a518a55ef\") " pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-bx2dr"
Nov 03 20:28:33.977449 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:33.977320    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca63212a-77c0-4b97-8a23-5d0a518a55ef-istio-podinfo\") pod \"e2e-api-a-c4a71d-openshift-default-7647f5cdbf-bx2dr\" (UID: \"ca63212a-77c0-4b97-8a23-5d0a518a55ef\") " pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-bx2dr"
Nov 03 20:28:33.977531 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:33.977457    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca63212a-77c0-4b97-8a23-5d0a518a55ef-credential-socket\") pod \"e2e-api-a-c4a71d-openshift-default-7647f5cdbf-bx2dr\" (UID: \"ca63212a-77c0-4b97-8a23-5d0a518a55ef\") " pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-bx2dr"
Nov 03 20:28:33.977531 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:33.977489    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9w6sx\" (UniqueName: \"kubernetes.io/projected/ca63212a-77c0-4b97-8a23-5d0a518a55ef-kube-api-access-9w6sx\") pod \"e2e-api-a-c4a71d-openshift-default-7647f5cdbf-bx2dr\" (UID: \"ca63212a-77c0-4b97-8a23-5d0a518a55ef\") " pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-bx2dr"
Nov 03 20:28:33.977653 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:33.977620    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca63212a-77c0-4b97-8a23-5d0a518a55ef-istio-data\") pod \"e2e-api-a-c4a71d-openshift-default-7647f5cdbf-bx2dr\" (UID: \"ca63212a-77c0-4b97-8a23-5d0a518a55ef\") " pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-bx2dr"
Nov 03 20:28:33.977727 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:33.977686    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca63212a-77c0-4b97-8a23-5d0a518a55ef-workload-certs\") pod \"e2e-api-a-c4a71d-openshift-default-7647f5cdbf-bx2dr\" (UID: \"ca63212a-77c0-4b97-8a23-5d0a518a55ef\") " pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-bx2dr"
Nov 03 20:28:33.977863 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:33.977789    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca63212a-77c0-4b97-8a23-5d0a518a55ef-istio-envoy\") pod \"e2e-api-a-c4a71d-openshift-default-7647f5cdbf-bx2dr\" (UID: \"ca63212a-77c0-4b97-8a23-5d0a518a55ef\") " pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-bx2dr"
Nov 03 20:28:33.977863 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:33.977823    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca63212a-77c0-4b97-8a23-5d0a518a55ef-istiod-ca-cert\") pod \"e2e-api-a-c4a71d-openshift-default-7647f5cdbf-bx2dr\" (UID: \"ca63212a-77c0-4b97-8a23-5d0a518a55ef\") " pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-bx2dr"
Nov 03 20:28:33.978043 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:33.977883    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca63212a-77c0-4b97-8a23-5d0a518a55ef-workload-socket\") pod \"e2e-api-a-c4a71d-openshift-default-7647f5cdbf-bx2dr\" (UID: \"ca63212a-77c0-4b97-8a23-5d0a518a55ef\") " pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-bx2dr"
Nov 03 20:28:34.083689 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:34.083603    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca63212a-77c0-4b97-8a23-5d0a518a55ef-workload-certs\") pod \"e2e-api-a-c4a71d-openshift-default-7647f5cdbf-bx2dr\" (UID: \"ca63212a-77c0-4b97-8a23-5d0a518a55ef\") " pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-bx2dr"
Nov 03 20:28:34.083934 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:34.083750    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca63212a-77c0-4b97-8a23-5d0a518a55ef-istio-envoy\") pod \"e2e-api-a-c4a71d-openshift-default-7647f5cdbf-bx2dr\" (UID: \"ca63212a-77c0-4b97-8a23-5d0a518a55ef\") " pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-bx2dr"
Nov 03 20:28:34.083934 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:34.083812    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca63212a-77c0-4b97-8a23-5d0a518a55ef-istiod-ca-cert\") pod \"e2e-api-a-c4a71d-openshift-default-7647f5cdbf-bx2dr\" (UID: \"ca63212a-77c0-4b97-8a23-5d0a518a55ef\") " pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-bx2dr"
Nov 03 20:28:34.083934 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:34.083873    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca63212a-77c0-4b97-8a23-5d0a518a55ef-workload-socket\") pod \"e2e-api-a-c4a71d-openshift-default-7647f5cdbf-bx2dr\" (UID: \"ca63212a-77c0-4b97-8a23-5d0a518a55ef\") " pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-bx2dr"
Nov 03 20:28:34.083934 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:34.083922    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca63212a-77c0-4b97-8a23-5d0a518a55ef-istio-token\") pod \"e2e-api-a-c4a71d-openshift-default-7647f5cdbf-bx2dr\" (UID: \"ca63212a-77c0-4b97-8a23-5d0a518a55ef\") " pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-bx2dr"
Nov 03 20:28:34.084164 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:34.084012    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca63212a-77c0-4b97-8a23-5d0a518a55ef-istio-podinfo\") pod \"e2e-api-a-c4a71d-openshift-default-7647f5cdbf-bx2dr\" (UID: \"ca63212a-77c0-4b97-8a23-5d0a518a55ef\") " pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-bx2dr"
Nov 03 20:28:34.084164 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:34.084056    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca63212a-77c0-4b97-8a23-5d0a518a55ef-workload-certs\") pod \"e2e-api-a-c4a71d-openshift-default-7647f5cdbf-bx2dr\" (UID: \"ca63212a-77c0-4b97-8a23-5d0a518a55ef\") " pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-bx2dr"
Nov 03 20:28:34.084274 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:34.084155    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca63212a-77c0-4b97-8a23-5d0a518a55ef-credential-socket\") pod \"e2e-api-a-c4a71d-openshift-default-7647f5cdbf-bx2dr\" (UID: \"ca63212a-77c0-4b97-8a23-5d0a518a55ef\") " pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-bx2dr"
Nov 03 20:28:34.084274 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:34.084199    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9w6sx\" (UniqueName: \"kubernetes.io/projected/ca63212a-77c0-4b97-8a23-5d0a518a55ef-kube-api-access-9w6sx\") pod \"e2e-api-a-c4a71d-openshift-default-7647f5cdbf-bx2dr\" (UID: \"ca63212a-77c0-4b97-8a23-5d0a518a55ef\") " pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-bx2dr"
Nov 03 20:28:34.084382 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:34.084279    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca63212a-77c0-4b97-8a23-5d0a518a55ef-istio-data\") pod \"e2e-api-a-c4a71d-openshift-default-7647f5cdbf-bx2dr\" (UID: \"ca63212a-77c0-4b97-8a23-5d0a518a55ef\") " pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-bx2dr"
Nov 03 20:28:34.084729 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:34.084709    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca63212a-77c0-4b97-8a23-5d0a518a55ef-credential-socket\") pod \"e2e-api-a-c4a71d-openshift-default-7647f5cdbf-bx2dr\" (UID: \"ca63212a-77c0-4b97-8a23-5d0a518a55ef\") " pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-bx2dr"
Nov 03 20:28:34.096932 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:34.095305    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca63212a-77c0-4b97-8a23-5d0a518a55ef-workload-socket\") pod \"e2e-api-a-c4a71d-openshift-default-7647f5cdbf-bx2dr\" (UID: \"ca63212a-77c0-4b97-8a23-5d0a518a55ef\") " pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-bx2dr"
Nov 03 20:28:34.105655 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:34.103507    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca63212a-77c0-4b97-8a23-5d0a518a55ef-istio-envoy\") pod \"e2e-api-a-c4a71d-openshift-default-7647f5cdbf-bx2dr\" (UID: \"ca63212a-77c0-4b97-8a23-5d0a518a55ef\") " pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-bx2dr"
Nov 03 20:28:34.105655 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:34.104835    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca63212a-77c0-4b97-8a23-5d0a518a55ef-istio-podinfo\") pod \"e2e-api-a-c4a71d-openshift-default-7647f5cdbf-bx2dr\" (UID: \"ca63212a-77c0-4b97-8a23-5d0a518a55ef\") " pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-bx2dr"
Nov 03 20:28:34.114989 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:34.107741    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca63212a-77c0-4b97-8a23-5d0a518a55ef-istiod-ca-cert\") pod \"e2e-api-a-c4a71d-openshift-default-7647f5cdbf-bx2dr\" (UID: \"ca63212a-77c0-4b97-8a23-5d0a518a55ef\") " pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-bx2dr"
Nov 03 20:28:34.114989 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:34.108408    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca63212a-77c0-4b97-8a23-5d0a518a55ef-istio-data\") pod \"e2e-api-a-c4a71d-openshift-default-7647f5cdbf-bx2dr\" (UID: \"ca63212a-77c0-4b97-8a23-5d0a518a55ef\") " pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-bx2dr"
Nov 03 20:28:34.114989 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:34.114274    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9w6sx\" (UniqueName: \"kubernetes.io/projected/ca63212a-77c0-4b97-8a23-5d0a518a55ef-kube-api-access-9w6sx\") pod \"e2e-api-a-c4a71d-openshift-default-7647f5cdbf-bx2dr\" (UID: \"ca63212a-77c0-4b97-8a23-5d0a518a55ef\") " pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-bx2dr"
Nov 03 20:28:34.122056 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:34.122029    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca63212a-77c0-4b97-8a23-5d0a518a55ef-istio-token\") pod \"e2e-api-a-c4a71d-openshift-default-7647f5cdbf-bx2dr\" (UID: \"ca63212a-77c0-4b97-8a23-5d0a518a55ef\") " pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-bx2dr"
Nov 03 20:28:34.176912 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:34.176884    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-bx2dr"
Nov 03 20:28:34.492516 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:34.492481    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-bx2dr"]
Nov 03 20:28:34.493460 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:28:34.493433    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca63212a_77c0_4b97_8a23_5d0a518a55ef.slice/crio-89018a843dc01fafbf9305a4f86a25539c93c57ae4b668a011c9899136f7671c WatchSource:0}: Error finding container 89018a843dc01fafbf9305a4f86a25539c93c57ae4b668a011c9899136f7671c: Status 404 returned error can't find the container with id 89018a843dc01fafbf9305a4f86a25539c93c57ae4b668a011c9899136f7671c
Nov 03 20:28:34.496315 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:34.496279    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:28:34.496388 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:34.496351    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:28:34.496388 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:34.496379    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:28:34.564035 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:34.560752    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-eb87f429-59cb4f857f-d25tg"]
Nov 03 20:28:34.564035 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:34.561001    5570 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-eb87f429-59cb4f857f-d25tg" podUID="257fbf5f-f3b8-4ac2-a7f2-8032a68526db" containerName="maas-api" containerID="cri-o://adc2adfa52bf7342d17e4b9f29b8b24917b5d3a4f45729f13c94f35aa82ea52e" gracePeriod=30
Nov 03 20:28:35.339805 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:35.337694    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-eb87f429-59cb4f857f-d25tg"
Nov 03 20:28:35.347812 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:35.344033    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-bx2dr" event={"ID":"ca63212a-77c0-4b97-8a23-5d0a518a55ef","Type":"ContainerStarted","Data":"a4842be231f4b13a0568dacd707e44fd989a4a1f6d171db686a6aac8a246eb0f"}
Nov 03 20:28:35.347812 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:35.344098    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-bx2dr" event={"ID":"ca63212a-77c0-4b97-8a23-5d0a518a55ef","Type":"ContainerStarted","Data":"89018a843dc01fafbf9305a4f86a25539c93c57ae4b668a011c9899136f7671c"}
Nov 03 20:28:35.351851 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:35.348242    5570 generic.go:356] "Generic (PLEG): container finished" podID="257fbf5f-f3b8-4ac2-a7f2-8032a68526db" containerID="adc2adfa52bf7342d17e4b9f29b8b24917b5d3a4f45729f13c94f35aa82ea52e" exitCode=0
Nov 03 20:28:35.351851 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:35.348391    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-eb87f429-59cb4f857f-d25tg" event={"ID":"257fbf5f-f3b8-4ac2-a7f2-8032a68526db","Type":"ContainerDied","Data":"adc2adfa52bf7342d17e4b9f29b8b24917b5d3a4f45729f13c94f35aa82ea52e"}
Nov 03 20:28:35.351851 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:35.348444    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-eb87f429-59cb4f857f-d25tg" event={"ID":"257fbf5f-f3b8-4ac2-a7f2-8032a68526db","Type":"ContainerDied","Data":"f717ed30d3f0812c7cc52ce26f5f375eb8a2dd9457f328b89832ec7fe169639f"}
Nov 03 20:28:35.351851 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:35.348478    5570 scope.go:122] "RemoveContainer" containerID="adc2adfa52bf7342d17e4b9f29b8b24917b5d3a4f45729f13c94f35aa82ea52e"
Nov 03 20:28:35.351851 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:35.348637    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-eb87f429-59cb4f857f-d25tg"
Nov 03 20:28:35.440673 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:35.428791    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/257fbf5f-f3b8-4ac2-a7f2-8032a68526db-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/257fbf5f-f3b8-4ac2-a7f2-8032a68526db-openshift-service-ca\") pod \"257fbf5f-f3b8-4ac2-a7f2-8032a68526db\" (UID: \"257fbf5f-f3b8-4ac2-a7f2-8032a68526db\") "
Nov 03 20:28:35.440673 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:35.428968    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/257fbf5f-f3b8-4ac2-a7f2-8032a68526db-metrics-certs\" (UniqueName: \"kubernetes.io/secret/257fbf5f-f3b8-4ac2-a7f2-8032a68526db-metrics-certs\") pod \"257fbf5f-f3b8-4ac2-a7f2-8032a68526db\" (UID: \"257fbf5f-f3b8-4ac2-a7f2-8032a68526db\") "
Nov 03 20:28:35.440673 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:35.429047    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/257fbf5f-f3b8-4ac2-a7f2-8032a68526db-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/257fbf5f-f3b8-4ac2-a7f2-8032a68526db-maas-api-tls\") pod \"257fbf5f-f3b8-4ac2-a7f2-8032a68526db\" (UID: \"257fbf5f-f3b8-4ac2-a7f2-8032a68526db\") "
Nov 03 20:28:35.440673 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:35.429096    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/257fbf5f-f3b8-4ac2-a7f2-8032a68526db-kube-api-access-tmxn4\" (UniqueName: \"kubernetes.io/projected/257fbf5f-f3b8-4ac2-a7f2-8032a68526db-kube-api-access-tmxn4\") pod \"257fbf5f-f3b8-4ac2-a7f2-8032a68526db\" (UID: \"257fbf5f-f3b8-4ac2-a7f2-8032a68526db\") "
Nov 03 20:28:35.453012 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:35.452975    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/257fbf5f-f3b8-4ac2-a7f2-8032a68526db-openshift-service-ca" pod "257fbf5f-f3b8-4ac2-a7f2-8032a68526db" (UID: "257fbf5f-f3b8-4ac2-a7f2-8032a68526db"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:28:35.590823 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:35.570456    5570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/257fbf5f-f3b8-4ac2-a7f2-8032a68526db-openshift-service-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:28:35.590823 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:35.572519    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/257fbf5f-f3b8-4ac2-a7f2-8032a68526db-maas-api-tls" pod "257fbf5f-f3b8-4ac2-a7f2-8032a68526db" (UID: "257fbf5f-f3b8-4ac2-a7f2-8032a68526db"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:28:35.590823 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:35.577792    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/257fbf5f-f3b8-4ac2-a7f2-8032a68526db-metrics-certs" pod "257fbf5f-f3b8-4ac2-a7f2-8032a68526db" (UID: "257fbf5f-f3b8-4ac2-a7f2-8032a68526db"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:28:35.600437 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:35.600360    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/257fbf5f-f3b8-4ac2-a7f2-8032a68526db-kube-api-access-tmxn4" pod "257fbf5f-f3b8-4ac2-a7f2-8032a68526db" (UID: "257fbf5f-f3b8-4ac2-a7f2-8032a68526db"). InnerVolumeSpecName "kube-api-access-tmxn4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:28:35.655981 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:35.653757    5570 scope.go:122] "RemoveContainer" containerID="adc2adfa52bf7342d17e4b9f29b8b24917b5d3a4f45729f13c94f35aa82ea52e"
Nov 03 20:28:35.656397 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:28:35.656287    5570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"adc2adfa52bf7342d17e4b9f29b8b24917b5d3a4f45729f13c94f35aa82ea52e\": container with ID starting with adc2adfa52bf7342d17e4b9f29b8b24917b5d3a4f45729f13c94f35aa82ea52e not found: ID does not exist" containerID="adc2adfa52bf7342d17e4b9f29b8b24917b5d3a4f45729f13c94f35aa82ea52e"
Nov 03 20:28:35.656397 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:35.656332    5570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"adc2adfa52bf7342d17e4b9f29b8b24917b5d3a4f45729f13c94f35aa82ea52e"} err="failed to get container status \"adc2adfa52bf7342d17e4b9f29b8b24917b5d3a4f45729f13c94f35aa82ea52e\": rpc error: code = NotFound desc = could not find container \"adc2adfa52bf7342d17e4b9f29b8b24917b5d3a4f45729f13c94f35aa82ea52e\": container with ID starting with adc2adfa52bf7342d17e4b9f29b8b24917b5d3a4f45729f13c94f35aa82ea52e not found: ID does not exist"
Nov 03 20:28:35.672831 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:35.671824    5570 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/257fbf5f-f3b8-4ac2-a7f2-8032a68526db-metrics-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:28:35.672831 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:35.671850    5570 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/257fbf5f-f3b8-4ac2-a7f2-8032a68526db-maas-api-tls\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:28:35.672831 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:35.671861    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmxn4\" (UniqueName: \"kubernetes.io/projected/257fbf5f-f3b8-4ac2-a7f2-8032a68526db-kube-api-access-tmxn4\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:28:35.703718 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:35.703674    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-bx2dr" podStartSLOduration=2.703660968 podStartE2EDuration="2.703660968s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:28:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:28:35.628811669 +0000 UTC m=+1449.781473556" watchObservedRunningTime="2026-11-03 20:28:35.703660968 +0000 UTC m=+1449.856322855"
Nov 03 20:28:35.708951 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:35.708920    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-eb87f429-59cb4f857f-d25tg"]
Nov 03 20:28:35.712037 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:35.712013    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-eb87f429-59cb4f857f-d25tg"]
Nov 03 20:28:35.966422 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:35.966387    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="257fbf5f-f3b8-4ac2-a7f2-8032a68526db" path="/var/lib/kubelet/pods/257fbf5f-f3b8-4ac2-a7f2-8032a68526db/volumes"
Nov 03 20:28:36.177047 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:36.177016    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-bx2dr"
Nov 03 20:28:36.184114 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:36.184088    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-bx2dr"
Nov 03 20:28:36.358965 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:36.358879    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-bx2dr"
Nov 03 20:28:36.359865 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:36.359844    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-bx2dr"
Nov 03 20:28:38.680795 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:38.680692    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-6fc748-6b8d84dfb-4tvlr"]
Nov 03 20:28:40.270124 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:40.270088    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c4a71d-665484ddc9-csmw4"]
Nov 03 20:28:40.270825 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:40.270798    5570 cpu_manager.go:414] "containerMap: removing container" podUID="257fbf5f-f3b8-4ac2-a7f2-8032a68526db" containerName="maas-api"
Nov 03 20:28:40.270825 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:40.270818    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="257fbf5f-f3b8-4ac2-a7f2-8032a68526db" containerName="maas-api"
Nov 03 20:28:40.270992 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:40.270964    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="257fbf5f-f3b8-4ac2-a7f2-8032a68526db" containerName="maas-api"
Nov 03 20:28:40.276373 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:40.276355    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c4a71d-665484ddc9-csmw4"
Nov 03 20:28:40.290347 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:40.290321    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c4a71d-665484ddc9-csmw4"]
Nov 03 20:28:40.305839 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:40.300133    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-c4a71d-dockercfg-52wqk\""
Nov 03 20:28:40.305839 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:40.300268    5570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-c4a71d\""
Nov 03 20:28:40.442795 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:40.442143    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6d4sk\" (UniqueName: \"kubernetes.io/projected/7b9f0474-e24d-4b53-bf9b-d7292207971b-kube-api-access-6d4sk\") pod \"payload-pre-processing-e2e-api-a-c4a71d-665484ddc9-csmw4\" (UID: \"7b9f0474-e24d-4b53-bf9b-d7292207971b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c4a71d-665484ddc9-csmw4"
Nov 03 20:28:40.442795 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:40.442215    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b9f0474-e24d-4b53-bf9b-d7292207971b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-c4a71d-665484ddc9-csmw4\" (UID: \"7b9f0474-e24d-4b53-bf9b-d7292207971b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c4a71d-665484ddc9-csmw4"
Nov 03 20:28:40.585868 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:40.543336    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c4a71d-5c8bb448f4-jsmhw"]
Nov 03 20:28:40.598790 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:40.591064    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6d4sk\" (UniqueName: \"kubernetes.io/projected/7b9f0474-e24d-4b53-bf9b-d7292207971b-kube-api-access-6d4sk\") pod \"payload-pre-processing-e2e-api-a-c4a71d-665484ddc9-csmw4\" (UID: \"7b9f0474-e24d-4b53-bf9b-d7292207971b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c4a71d-665484ddc9-csmw4"
Nov 03 20:28:40.598790 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:40.591120    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b9f0474-e24d-4b53-bf9b-d7292207971b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-c4a71d-665484ddc9-csmw4\" (UID: \"7b9f0474-e24d-4b53-bf9b-d7292207971b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c4a71d-665484ddc9-csmw4"
Nov 03 20:28:40.618430 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:40.617824    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b9f0474-e24d-4b53-bf9b-d7292207971b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-c4a71d-665484ddc9-csmw4\" (UID: \"7b9f0474-e24d-4b53-bf9b-d7292207971b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c4a71d-665484ddc9-csmw4"
Nov 03 20:28:40.669789 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:40.669736    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6d4sk\" (UniqueName: \"kubernetes.io/projected/7b9f0474-e24d-4b53-bf9b-d7292207971b-kube-api-access-6d4sk\") pod \"payload-pre-processing-e2e-api-a-c4a71d-665484ddc9-csmw4\" (UID: \"7b9f0474-e24d-4b53-bf9b-d7292207971b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c4a71d-665484ddc9-csmw4"
Nov 03 20:28:40.723955 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:40.687440    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-c4a71d-5c8bb448f4-jsmhw"
Nov 03 20:28:40.724241 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:40.704347    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be6cbff2-19be-4730-b7b4-ab513d8c6859-plugins-config-volume\") pod \"payload-processing-e2e-api-a-c4a71d-5c8bb448f4-jsmhw\" (UID: \"be6cbff2-19be-4730-b7b4-ab513d8c6859\") " pod="openshift-ingress/payload-processing-e2e-api-a-c4a71d-5c8bb448f4-jsmhw"
Nov 03 20:28:40.724402 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:40.724375    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vw4mt\" (UniqueName: \"kubernetes.io/projected/be6cbff2-19be-4730-b7b4-ab513d8c6859-kube-api-access-vw4mt\") pod \"payload-processing-e2e-api-a-c4a71d-5c8bb448f4-jsmhw\" (UID: \"be6cbff2-19be-4730-b7b4-ab513d8c6859\") " pod="openshift-ingress/payload-processing-e2e-api-a-c4a71d-5c8bb448f4-jsmhw"
Nov 03 20:28:40.729406 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:40.729376    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c4a71d-5c8bb448f4-jsmhw"]
Nov 03 20:28:40.826820 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:40.826751    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be6cbff2-19be-4730-b7b4-ab513d8c6859-plugins-config-volume\") pod \"payload-processing-e2e-api-a-c4a71d-5c8bb448f4-jsmhw\" (UID: \"be6cbff2-19be-4730-b7b4-ab513d8c6859\") " pod="openshift-ingress/payload-processing-e2e-api-a-c4a71d-5c8bb448f4-jsmhw"
Nov 03 20:28:40.826992 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:40.826885    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vw4mt\" (UniqueName: \"kubernetes.io/projected/be6cbff2-19be-4730-b7b4-ab513d8c6859-kube-api-access-vw4mt\") pod \"payload-processing-e2e-api-a-c4a71d-5c8bb448f4-jsmhw\" (UID: \"be6cbff2-19be-4730-b7b4-ab513d8c6859\") " pod="openshift-ingress/payload-processing-e2e-api-a-c4a71d-5c8bb448f4-jsmhw"
Nov 03 20:28:40.827484 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:40.827462    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be6cbff2-19be-4730-b7b4-ab513d8c6859-plugins-config-volume\") pod \"payload-processing-e2e-api-a-c4a71d-5c8bb448f4-jsmhw\" (UID: \"be6cbff2-19be-4730-b7b4-ab513d8c6859\") " pod="openshift-ingress/payload-processing-e2e-api-a-c4a71d-5c8bb448f4-jsmhw"
Nov 03 20:28:40.846877 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:40.846790    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vw4mt\" (UniqueName: \"kubernetes.io/projected/be6cbff2-19be-4730-b7b4-ab513d8c6859-kube-api-access-vw4mt\") pod \"payload-processing-e2e-api-a-c4a71d-5c8bb448f4-jsmhw\" (UID: \"be6cbff2-19be-4730-b7b4-ab513d8c6859\") " pod="openshift-ingress/payload-processing-e2e-api-a-c4a71d-5c8bb448f4-jsmhw"
Nov 03 20:28:40.893705 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:40.893672    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c4a71d-665484ddc9-csmw4"
Nov 03 20:28:41.077505 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:41.077468    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-c4a71d-5c8bb448f4-jsmhw"
Nov 03 20:28:41.204555 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:41.204513    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c4a71d-665484ddc9-csmw4"]
Nov 03 20:28:41.207226 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:28:41.207189    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b9f0474_e24d_4b53_bf9b_d7292207971b.slice/crio-1baf64282060eecf2b9993e066ea62ae80d86908f81e455c0b425689b2f0856f WatchSource:0}: Error finding container 1baf64282060eecf2b9993e066ea62ae80d86908f81e455c0b425689b2f0856f: Status 404 returned error can't find the container with id 1baf64282060eecf2b9993e066ea62ae80d86908f81e455c0b425689b2f0856f
Nov 03 20:28:41.422817 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:28:41.416140    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe6cbff2_19be_4730_b7b4_ab513d8c6859.slice/crio-5b1bee1ad41a04fef77a6e7bf5bc9941c608ddb0c687d1109ab40297e192fec5 WatchSource:0}: Error finding container 5b1bee1ad41a04fef77a6e7bf5bc9941c608ddb0c687d1109ab40297e192fec5: Status 404 returned error can't find the container with id 5b1bee1ad41a04fef77a6e7bf5bc9941c608ddb0c687d1109ab40297e192fec5
Nov 03 20:28:41.422817 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:41.419000    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c4a71d-5c8bb448f4-jsmhw"]
Nov 03 20:28:41.660967 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:41.660921    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-c4a71d-5c8bb448f4-jsmhw" event={"ID":"be6cbff2-19be-4730-b7b4-ab513d8c6859","Type":"ContainerStarted","Data":"b59566cf486a15588b4da19a9b55f09732218dc808043b8b397bb652190b1c29"}
Nov 03 20:28:41.661136 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:41.660993    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-c4a71d-5c8bb448f4-jsmhw"
Nov 03 20:28:41.661136 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:41.661015    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-c4a71d-5c8bb448f4-jsmhw" event={"ID":"be6cbff2-19be-4730-b7b4-ab513d8c6859","Type":"ContainerStarted","Data":"5b1bee1ad41a04fef77a6e7bf5bc9941c608ddb0c687d1109ab40297e192fec5"}
Nov 03 20:28:41.662689 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:41.662617    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c4a71d-665484ddc9-csmw4" event={"ID":"7b9f0474-e24d-4b53-bf9b-d7292207971b","Type":"ContainerStarted","Data":"f9818b0562a1e45412512a0dd2060af8663585c5c08b69a8840938ed5dc0e65d"}
Nov 03 20:28:41.662689 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:41.662672    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c4a71d-665484ddc9-csmw4" event={"ID":"7b9f0474-e24d-4b53-bf9b-d7292207971b","Type":"ContainerStarted","Data":"1baf64282060eecf2b9993e066ea62ae80d86908f81e455c0b425689b2f0856f"}
Nov 03 20:28:41.662891 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:41.662694    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c4a71d-665484ddc9-csmw4"
Nov 03 20:28:41.702332 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:41.702270    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-c4a71d-5c8bb448f4-jsmhw" podStartSLOduration=1.702254492 podStartE2EDuration="1.702254492s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:28:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:28:41.701108365 +0000 UTC m=+1455.853770253" watchObservedRunningTime="2026-11-03 20:28:41.702254492 +0000 UTC m=+1455.854916381"
Nov 03 20:28:41.745610 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:41.745550    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c4a71d-665484ddc9-csmw4" podStartSLOduration=1.745534782 podStartE2EDuration="1.745534782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:28:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:28:41.742835888 +0000 UTC m=+1455.895497778" watchObservedRunningTime="2026-11-03 20:28:41.745534782 +0000 UTC m=+1455.898196733"
Nov 03 20:28:43.170539 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:43.170506    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-eb87f429-7fbff5576b-89srx"]
Nov 03 20:28:43.175538 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:43.175515    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-eb87f429-7fbff5576b-89srx"
Nov 03 20:28:43.192399 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:43.192370    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-eb87f429-7fbff5576b-89srx"]
Nov 03 20:28:43.200327 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:43.200301    5570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-eb87f429\""
Nov 03 20:28:43.219483 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:43.219451    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-eb87f429-dockercfg-s7z4n\""
Nov 03 20:28:43.232381 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:43.232345    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-eb87f429-6b49f7957f-bc4ph"]
Nov 03 20:28:43.236821 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:43.236795    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-eb87f429-6b49f7957f-bc4ph"
Nov 03 20:28:43.252993 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:43.252962    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-eb87f429-6b49f7957f-bc4ph"]
Nov 03 20:28:43.329798 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:43.326700    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frj95\" (UniqueName: \"kubernetes.io/projected/fc90ef38-bcf6-4df4-a07c-cb9726b731af-kube-api-access-frj95\") pod \"payload-processing-e2e-mt-eb87f429-7fbff5576b-89srx\" (UID: \"fc90ef38-bcf6-4df4-a07c-cb9726b731af\") " pod="openshift-ingress/payload-processing-e2e-mt-eb87f429-7fbff5576b-89srx"
Nov 03 20:28:43.329798 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:43.327148    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc90ef38-bcf6-4df4-a07c-cb9726b731af-plugins-config-volume\") pod \"payload-processing-e2e-mt-eb87f429-7fbff5576b-89srx\" (UID: \"fc90ef38-bcf6-4df4-a07c-cb9726b731af\") " pod="openshift-ingress/payload-processing-e2e-mt-eb87f429-7fbff5576b-89srx"
Nov 03 20:28:43.494807 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:43.470092    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcc6z\" (UniqueName: \"kubernetes.io/projected/f578561f-2a3b-49cf-b7fa-3a22da0240b3-kube-api-access-bcc6z\") pod \"payload-pre-processing-e2e-mt-eb87f429-6b49f7957f-bc4ph\" (UID: \"f578561f-2a3b-49cf-b7fa-3a22da0240b3\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-eb87f429-6b49f7957f-bc4ph"
Nov 03 20:28:43.494807 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:43.470213    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc90ef38-bcf6-4df4-a07c-cb9726b731af-plugins-config-volume\") pod \"payload-processing-e2e-mt-eb87f429-7fbff5576b-89srx\" (UID: \"fc90ef38-bcf6-4df4-a07c-cb9726b731af\") " pod="openshift-ingress/payload-processing-e2e-mt-eb87f429-7fbff5576b-89srx"
Nov 03 20:28:43.494807 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:43.470250    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f578561f-2a3b-49cf-b7fa-3a22da0240b3-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-eb87f429-6b49f7957f-bc4ph\" (UID: \"f578561f-2a3b-49cf-b7fa-3a22da0240b3\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-eb87f429-6b49f7957f-bc4ph"
Nov 03 20:28:43.494807 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:43.470489    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frj95\" (UniqueName: \"kubernetes.io/projected/fc90ef38-bcf6-4df4-a07c-cb9726b731af-kube-api-access-frj95\") pod \"payload-processing-e2e-mt-eb87f429-7fbff5576b-89srx\" (UID: \"fc90ef38-bcf6-4df4-a07c-cb9726b731af\") " pod="openshift-ingress/payload-processing-e2e-mt-eb87f429-7fbff5576b-89srx"
Nov 03 20:28:43.494807 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:43.483813    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc90ef38-bcf6-4df4-a07c-cb9726b731af-plugins-config-volume\") pod \"payload-processing-e2e-mt-eb87f429-7fbff5576b-89srx\" (UID: \"fc90ef38-bcf6-4df4-a07c-cb9726b731af\") " pod="openshift-ingress/payload-processing-e2e-mt-eb87f429-7fbff5576b-89srx"
Nov 03 20:28:43.582810 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:43.581897    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f578561f-2a3b-49cf-b7fa-3a22da0240b3-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-eb87f429-6b49f7957f-bc4ph\" (UID: \"f578561f-2a3b-49cf-b7fa-3a22da0240b3\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-eb87f429-6b49f7957f-bc4ph"
Nov 03 20:28:43.582810 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:43.582395    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcc6z\" (UniqueName: \"kubernetes.io/projected/f578561f-2a3b-49cf-b7fa-3a22da0240b3-kube-api-access-bcc6z\") pod \"payload-pre-processing-e2e-mt-eb87f429-6b49f7957f-bc4ph\" (UID: \"f578561f-2a3b-49cf-b7fa-3a22da0240b3\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-eb87f429-6b49f7957f-bc4ph"
Nov 03 20:28:43.635807 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:43.584353    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f578561f-2a3b-49cf-b7fa-3a22da0240b3-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-eb87f429-6b49f7957f-bc4ph\" (UID: \"f578561f-2a3b-49cf-b7fa-3a22da0240b3\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-eb87f429-6b49f7957f-bc4ph"
Nov 03 20:28:43.671804 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:43.667709    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frj95\" (UniqueName: \"kubernetes.io/projected/fc90ef38-bcf6-4df4-a07c-cb9726b731af-kube-api-access-frj95\") pod \"payload-processing-e2e-mt-eb87f429-7fbff5576b-89srx\" (UID: \"fc90ef38-bcf6-4df4-a07c-cb9726b731af\") " pod="openshift-ingress/payload-processing-e2e-mt-eb87f429-7fbff5576b-89srx"
Nov 03 20:28:43.684793 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:43.681747    5570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-6fc748-6b8d84dfb-4tvlr" podUID="75db83cf-ba54-414a-9041-094aa630a60a" containerName="payload-processing" containerID="cri-o://a9afc22fe0d84c757e6fa956bb7652781c88cdd767dc124612294b26216b2019" gracePeriod=25
Nov 03 20:28:43.684793 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:43.682474    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcc6z\" (UniqueName: \"kubernetes.io/projected/f578561f-2a3b-49cf-b7fa-3a22da0240b3-kube-api-access-bcc6z\") pod \"payload-pre-processing-e2e-mt-eb87f429-6b49f7957f-bc4ph\" (UID: \"f578561f-2a3b-49cf-b7fa-3a22da0240b3\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-eb87f429-6b49f7957f-bc4ph"
Nov 03 20:28:43.792177 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:43.792111    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-eb87f429-7fbff5576b-89srx"
Nov 03 20:28:43.853570 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:43.853114    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-eb87f429-6b49f7957f-bc4ph"
Nov 03 20:28:44.040108 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:44.040073    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-6fc748-6b8d84dfb-4tvlr"
Nov 03 20:28:44.100934 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:44.100876    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75db83cf-ba54-414a-9041-094aa630a60a-kube-api-access-7lg2h\" (UniqueName: \"kubernetes.io/projected/75db83cf-ba54-414a-9041-094aa630a60a-kube-api-access-7lg2h\") pod \"75db83cf-ba54-414a-9041-094aa630a60a\" (UID: \"75db83cf-ba54-414a-9041-094aa630a60a\") "
Nov 03 20:28:44.101369 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:44.101114    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75db83cf-ba54-414a-9041-094aa630a60a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75db83cf-ba54-414a-9041-094aa630a60a-plugins-config-volume\") pod \"75db83cf-ba54-414a-9041-094aa630a60a\" (UID: \"75db83cf-ba54-414a-9041-094aa630a60a\") "
Nov 03 20:28:44.102221 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:44.101569    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75db83cf-ba54-414a-9041-094aa630a60a-plugins-config-volume" pod "75db83cf-ba54-414a-9041-094aa630a60a" (UID: "75db83cf-ba54-414a-9041-094aa630a60a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:28:44.103761 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:44.103619    5570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75db83cf-ba54-414a-9041-094aa630a60a-plugins-config-volume\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:28:44.104089 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:44.104065    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-eb87f429-7fbff5576b-89srx"]
Nov 03 20:28:44.104186 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:28:44.104101    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc90ef38_bcf6_4df4_a07c_cb9726b731af.slice/crio-aa599f8f8e6611c1ad2f72df2a56964b147d3247d4e7088b7df25375a83d3cd0 WatchSource:0}: Error finding container aa599f8f8e6611c1ad2f72df2a56964b147d3247d4e7088b7df25375a83d3cd0: Status 404 returned error can't find the container with id aa599f8f8e6611c1ad2f72df2a56964b147d3247d4e7088b7df25375a83d3cd0
Nov 03 20:28:44.105904 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:44.105877    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75db83cf-ba54-414a-9041-094aa630a60a-kube-api-access-7lg2h" pod "75db83cf-ba54-414a-9041-094aa630a60a" (UID: "75db83cf-ba54-414a-9041-094aa630a60a"). InnerVolumeSpecName "kube-api-access-7lg2h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:28:44.174536 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:44.174498    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-eb87f429-6b49f7957f-bc4ph"]
Nov 03 20:28:44.175302 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:28:44.175278    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf578561f_2a3b_49cf_b7fa_3a22da0240b3.slice/crio-c7ad3a2e50b57f264536ccd3ee7d9c23b238355fadca444ed300b55439c0f95f WatchSource:0}: Error finding container c7ad3a2e50b57f264536ccd3ee7d9c23b238355fadca444ed300b55439c0f95f: Status 404 returned error can't find the container with id c7ad3a2e50b57f264536ccd3ee7d9c23b238355fadca444ed300b55439c0f95f
Nov 03 20:28:44.205722 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:44.205693    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7lg2h\" (UniqueName: \"kubernetes.io/projected/75db83cf-ba54-414a-9041-094aa630a60a-kube-api-access-7lg2h\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:28:44.698757 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:44.698723    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-eb87f429-6b49f7957f-bc4ph" event={"ID":"f578561f-2a3b-49cf-b7fa-3a22da0240b3","Type":"ContainerStarted","Data":"ec4d3c01e3b418bab5c1a403550ef75a801d9e4a3ec0caad7a7361b364f77f9c"}
Nov 03 20:28:44.698757 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:44.698793    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-eb87f429-6b49f7957f-bc4ph" event={"ID":"f578561f-2a3b-49cf-b7fa-3a22da0240b3","Type":"ContainerStarted","Data":"c7ad3a2e50b57f264536ccd3ee7d9c23b238355fadca444ed300b55439c0f95f"}
Nov 03 20:28:44.699047 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:44.698826    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-eb87f429-6b49f7957f-bc4ph"
Nov 03 20:28:44.700469 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:44.700441    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-eb87f429-7fbff5576b-89srx" event={"ID":"fc90ef38-bcf6-4df4-a07c-cb9726b731af","Type":"ContainerStarted","Data":"0e5a3aee5347afddb2d3ee7cd1065d0920bc6a99ccab810d2b962744d9a11e49"}
Nov 03 20:28:44.700600 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:44.700494    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-eb87f429-7fbff5576b-89srx"
Nov 03 20:28:44.700600 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:44.700521    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-eb87f429-7fbff5576b-89srx" event={"ID":"fc90ef38-bcf6-4df4-a07c-cb9726b731af","Type":"ContainerStarted","Data":"aa599f8f8e6611c1ad2f72df2a56964b147d3247d4e7088b7df25375a83d3cd0"}
Nov 03 20:28:44.701821 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:44.701800    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-6fc748-6b8d84dfb-4tvlr" event={"ID":"75db83cf-ba54-414a-9041-094aa630a60a","Type":"ContainerDied","Data":"a9afc22fe0d84c757e6fa956bb7652781c88cdd767dc124612294b26216b2019"}
Nov 03 20:28:44.701908 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:44.701819    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-6fc748-6b8d84dfb-4tvlr"
Nov 03 20:28:44.701908 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:44.701851    5570 scope.go:122] "RemoveContainer" containerID="a9afc22fe0d84c757e6fa956bb7652781c88cdd767dc124612294b26216b2019"
Nov 03 20:28:44.701908 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:44.701769    5570 generic.go:356] "Generic (PLEG): container finished" podID="75db83cf-ba54-414a-9041-094aa630a60a" containerID="a9afc22fe0d84c757e6fa956bb7652781c88cdd767dc124612294b26216b2019" exitCode=0
Nov 03 20:28:44.702026 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:44.701928    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-6fc748-6b8d84dfb-4tvlr" event={"ID":"75db83cf-ba54-414a-9041-094aa630a60a","Type":"ContainerDied","Data":"a04523c80e655466a5e50ecf8433a76315e6c506f131466c9dd0befbe1207282"}
Nov 03 20:28:44.716020 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:44.715993    5570 scope.go:122] "RemoveContainer" containerID="a9afc22fe0d84c757e6fa956bb7652781c88cdd767dc124612294b26216b2019"
Nov 03 20:28:44.716317 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:28:44.716296    5570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9afc22fe0d84c757e6fa956bb7652781c88cdd767dc124612294b26216b2019\": container with ID starting with a9afc22fe0d84c757e6fa956bb7652781c88cdd767dc124612294b26216b2019 not found: ID does not exist" containerID="a9afc22fe0d84c757e6fa956bb7652781c88cdd767dc124612294b26216b2019"
Nov 03 20:28:44.716394 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:44.716329    5570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9afc22fe0d84c757e6fa956bb7652781c88cdd767dc124612294b26216b2019"} err="failed to get container status \"a9afc22fe0d84c757e6fa956bb7652781c88cdd767dc124612294b26216b2019\": rpc error: code = NotFound desc = could not find container \"a9afc22fe0d84c757e6fa956bb7652781c88cdd767dc124612294b26216b2019\": container with ID starting with a9afc22fe0d84c757e6fa956bb7652781c88cdd767dc124612294b26216b2019 not found: ID does not exist"
Nov 03 20:28:44.741271 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:44.741218    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-eb87f429-6b49f7957f-bc4ph" podStartSLOduration=1.74120465 podStartE2EDuration="1.74120465s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:28:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:28:44.737976886 +0000 UTC m=+1458.890638763" watchObservedRunningTime="2026-11-03 20:28:44.74120465 +0000 UTC m=+1458.893866536"
Nov 03 20:28:44.777941 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:44.777907    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-6fc748-6b8d84dfb-4tvlr"]
Nov 03 20:28:44.781682 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:44.781658    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-6fc748-6b8d84dfb-4tvlr"]
Nov 03 20:28:44.819089 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:44.819035    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-eb87f429-7fbff5576b-89srx" podStartSLOduration=1.819021036 podStartE2EDuration="1.819021036s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:28:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:28:44.817901174 +0000 UTC m=+1458.970563059" watchObservedRunningTime="2026-11-03 20:28:44.819021036 +0000 UTC m=+1458.971682923"
Nov 03 20:28:45.966516 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:45.966481    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75db83cf-ba54-414a-9041-094aa630a60a" path="/var/lib/kubelet/pods/75db83cf-ba54-414a-9041-094aa630a60a/volumes"
Nov 03 20:28:47.977431 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:47.977349    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-addf1a-56979459f6-bt9cd"]
Nov 03 20:28:47.978130 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:47.978114    5570 cpu_manager.go:414] "containerMap: removing container" podUID="75db83cf-ba54-414a-9041-094aa630a60a" containerName="payload-processing"
Nov 03 20:28:47.978185 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:47.978131    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75db83cf-ba54-414a-9041-094aa630a60a" containerName="payload-processing"
Nov 03 20:28:47.978266 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:47.978257    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="75db83cf-ba54-414a-9041-094aa630a60a" containerName="payload-processing"
Nov 03 20:28:47.982457 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:47.982435    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-addf1a-56979459f6-bt9cd"
Nov 03 20:28:47.998521 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:47.998475    5570 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-6fc748/premium-w1-6fc748-kserve-7474db9cd7-nf2xk" podUID="830e062e-83fc-4de7-91a7-5534dac1d46b" containerName="main" containerID="cri-o://a24442a935efa7166e38699d8b414a492546737e47c8e40770013fb03915efb0" gracePeriod=45
Nov 03 20:28:48.000915 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:48.000701    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-addf1a-56979459f6-bt9cd"]
Nov 03 20:28:48.018238 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:48.018207    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-addf1a-serving-cert\""
Nov 03 20:28:48.066887 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:48.066845    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d450803-ade0-428d-9029-d71459b2fc22-metrics-certs\") pod \"maas-api-e2e-api-a-addf1a-56979459f6-bt9cd\" (UID: \"5d450803-ade0-428d-9029-d71459b2fc22\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-addf1a-56979459f6-bt9cd"
Nov 03 20:28:48.067065 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:48.066988    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkrq4\" (UniqueName: \"kubernetes.io/projected/5d450803-ade0-428d-9029-d71459b2fc22-kube-api-access-qkrq4\") pod \"maas-api-e2e-api-a-addf1a-56979459f6-bt9cd\" (UID: \"5d450803-ade0-428d-9029-d71459b2fc22\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-addf1a-56979459f6-bt9cd"
Nov 03 20:28:48.067203 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:48.067176    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d450803-ade0-428d-9029-d71459b2fc22-openshift-service-ca\") pod \"maas-api-e2e-api-a-addf1a-56979459f6-bt9cd\" (UID: \"5d450803-ade0-428d-9029-d71459b2fc22\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-addf1a-56979459f6-bt9cd"
Nov 03 20:28:48.067388 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:48.067368    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d450803-ade0-428d-9029-d71459b2fc22-maas-api-tls\") pod \"maas-api-e2e-api-a-addf1a-56979459f6-bt9cd\" (UID: \"5d450803-ade0-428d-9029-d71459b2fc22\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-addf1a-56979459f6-bt9cd"
Nov 03 20:28:48.156143 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:48.156094    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6fc748/premium-w1-6fc748-kserve-7474db9cd7-nf2xk" podUID="830e062e-83fc-4de7-91a7-5534dac1d46b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": dial tcp 10.134.0.56:8000: connect: connection refused"
Nov 03 20:28:48.181374 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:48.181279    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d450803-ade0-428d-9029-d71459b2fc22-maas-api-tls\") pod \"maas-api-e2e-api-a-addf1a-56979459f6-bt9cd\" (UID: \"5d450803-ade0-428d-9029-d71459b2fc22\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-addf1a-56979459f6-bt9cd"
Nov 03 20:28:48.200827 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:28:48.189282    5570 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-addf1a-serving-cert: secret "maas-api-e2e-api-a-addf1a-serving-cert" not found
Nov 03 20:28:48.200827 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:28:48.196280    5570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d450803-ade0-428d-9029-d71459b2fc22-maas-api-tls podName:5d450803-ade0-428d-9029-d71459b2fc22 nodeName:}" failed. No retries permitted until 2026-11-03 20:28:48.696251294 +0000 UTC m=+1462.848913179 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5d450803-ade0-428d-9029-d71459b2fc22-maas-api-tls") pod "maas-api-e2e-api-a-addf1a-56979459f6-bt9cd" (UID: "5d450803-ade0-428d-9029-d71459b2fc22") : secret "maas-api-e2e-api-a-addf1a-serving-cert" not found
Nov 03 20:28:48.200827 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:48.196816    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d450803-ade0-428d-9029-d71459b2fc22-metrics-certs\") pod \"maas-api-e2e-api-a-addf1a-56979459f6-bt9cd\" (UID: \"5d450803-ade0-428d-9029-d71459b2fc22\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-addf1a-56979459f6-bt9cd"
Nov 03 20:28:48.200827 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:48.196901    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkrq4\" (UniqueName: \"kubernetes.io/projected/5d450803-ade0-428d-9029-d71459b2fc22-kube-api-access-qkrq4\") pod \"maas-api-e2e-api-a-addf1a-56979459f6-bt9cd\" (UID: \"5d450803-ade0-428d-9029-d71459b2fc22\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-addf1a-56979459f6-bt9cd"
Nov 03 20:28:48.200827 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:48.197055    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d450803-ade0-428d-9029-d71459b2fc22-openshift-service-ca\") pod \"maas-api-e2e-api-a-addf1a-56979459f6-bt9cd\" (UID: \"5d450803-ade0-428d-9029-d71459b2fc22\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-addf1a-56979459f6-bt9cd"
Nov 03 20:28:48.219823 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:48.202211    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d450803-ade0-428d-9029-d71459b2fc22-openshift-service-ca\") pod \"maas-api-e2e-api-a-addf1a-56979459f6-bt9cd\" (UID: \"5d450803-ade0-428d-9029-d71459b2fc22\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-addf1a-56979459f6-bt9cd"
Nov 03 20:28:48.307690 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:48.232436    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d450803-ade0-428d-9029-d71459b2fc22-metrics-certs\") pod \"maas-api-e2e-api-a-addf1a-56979459f6-bt9cd\" (UID: \"5d450803-ade0-428d-9029-d71459b2fc22\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-addf1a-56979459f6-bt9cd"
Nov 03 20:28:48.307690 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:48.253914    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkrq4\" (UniqueName: \"kubernetes.io/projected/5d450803-ade0-428d-9029-d71459b2fc22-kube-api-access-qkrq4\") pod \"maas-api-e2e-api-a-addf1a-56979459f6-bt9cd\" (UID: \"5d450803-ade0-428d-9029-d71459b2fc22\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-addf1a-56979459f6-bt9cd"
Nov 03 20:28:48.469187 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:48.469155    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-6fc748/premium-w1-6fc748-kserve-7474db9cd7-nf2xk"
Nov 03 20:28:48.606468 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:48.606363    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/830e062e-83fc-4de7-91a7-5534dac1d46b-kube-api-access-sc2wj\" (UniqueName: \"kubernetes.io/projected/830e062e-83fc-4de7-91a7-5534dac1d46b-kube-api-access-sc2wj\") pod \"830e062e-83fc-4de7-91a7-5534dac1d46b\" (UID: \"830e062e-83fc-4de7-91a7-5534dac1d46b\") "
Nov 03 20:28:48.606653 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:48.606476    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/830e062e-83fc-4de7-91a7-5534dac1d46b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/830e062e-83fc-4de7-91a7-5534dac1d46b-dshm\") pod \"830e062e-83fc-4de7-91a7-5534dac1d46b\" (UID: \"830e062e-83fc-4de7-91a7-5534dac1d46b\") "
Nov 03 20:28:48.606653 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:48.606518    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/830e062e-83fc-4de7-91a7-5534dac1d46b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/830e062e-83fc-4de7-91a7-5534dac1d46b-model-cache\") pod \"830e062e-83fc-4de7-91a7-5534dac1d46b\" (UID: \"830e062e-83fc-4de7-91a7-5534dac1d46b\") "
Nov 03 20:28:48.606800 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:48.606684    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/830e062e-83fc-4de7-91a7-5534dac1d46b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/830e062e-83fc-4de7-91a7-5534dac1d46b-tmp-dir\") pod \"830e062e-83fc-4de7-91a7-5534dac1d46b\" (UID: \"830e062e-83fc-4de7-91a7-5534dac1d46b\") "
Nov 03 20:28:48.606800 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:48.606736    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/830e062e-83fc-4de7-91a7-5534dac1d46b-home\" (UniqueName: \"kubernetes.io/empty-dir/830e062e-83fc-4de7-91a7-5534dac1d46b-home\") pod \"830e062e-83fc-4de7-91a7-5534dac1d46b\" (UID: \"830e062e-83fc-4de7-91a7-5534dac1d46b\") "
Nov 03 20:28:48.606922 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:48.606862    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/830e062e-83fc-4de7-91a7-5534dac1d46b-tls-certs\" (UniqueName: \"kubernetes.io/secret/830e062e-83fc-4de7-91a7-5534dac1d46b-tls-certs\") pod \"830e062e-83fc-4de7-91a7-5534dac1d46b\" (UID: \"830e062e-83fc-4de7-91a7-5534dac1d46b\") "
Nov 03 20:28:48.607268 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:48.607215    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/830e062e-83fc-4de7-91a7-5534dac1d46b-tmp-dir" pod "830e062e-83fc-4de7-91a7-5534dac1d46b" (UID: "830e062e-83fc-4de7-91a7-5534dac1d46b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:28:48.607490 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:48.607304    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/830e062e-83fc-4de7-91a7-5534dac1d46b-home" pod "830e062e-83fc-4de7-91a7-5534dac1d46b" (UID: "830e062e-83fc-4de7-91a7-5534dac1d46b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:28:48.607588 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:48.607121    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/830e062e-83fc-4de7-91a7-5534dac1d46b-model-cache" pod "830e062e-83fc-4de7-91a7-5534dac1d46b" (UID: "830e062e-83fc-4de7-91a7-5534dac1d46b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:28:48.607983 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:48.607955    5570 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/830e062e-83fc-4de7-91a7-5534dac1d46b-tmp-dir\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:28:48.608315 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:48.607984    5570 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/830e062e-83fc-4de7-91a7-5534dac1d46b-home\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:28:48.608315 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:48.608000    5570 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/830e062e-83fc-4de7-91a7-5534dac1d46b-model-cache\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:28:48.609337 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:48.609277    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/830e062e-83fc-4de7-91a7-5534dac1d46b-dshm" pod "830e062e-83fc-4de7-91a7-5534dac1d46b" (UID: "830e062e-83fc-4de7-91a7-5534dac1d46b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:28:48.609337 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:48.609299    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/830e062e-83fc-4de7-91a7-5534dac1d46b-kube-api-access-sc2wj" pod "830e062e-83fc-4de7-91a7-5534dac1d46b" (UID: "830e062e-83fc-4de7-91a7-5534dac1d46b"). InnerVolumeSpecName "kube-api-access-sc2wj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:28:48.609484 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:48.609354    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/830e062e-83fc-4de7-91a7-5534dac1d46b-tls-certs" pod "830e062e-83fc-4de7-91a7-5534dac1d46b" (UID: "830e062e-83fc-4de7-91a7-5534dac1d46b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:28:48.709129 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:48.709081    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d450803-ade0-428d-9029-d71459b2fc22-maas-api-tls\") pod \"maas-api-e2e-api-a-addf1a-56979459f6-bt9cd\" (UID: \"5d450803-ade0-428d-9029-d71459b2fc22\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-addf1a-56979459f6-bt9cd"
Nov 03 20:28:48.709481 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:48.709459    5570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/830e062e-83fc-4de7-91a7-5534dac1d46b-tls-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:28:48.709566 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:48.709485    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sc2wj\" (UniqueName: \"kubernetes.io/projected/830e062e-83fc-4de7-91a7-5534dac1d46b-kube-api-access-sc2wj\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:28:48.709566 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:48.709500    5570 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/830e062e-83fc-4de7-91a7-5534dac1d46b-dshm\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:28:48.712146 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:48.712116    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d450803-ade0-428d-9029-d71459b2fc22-maas-api-tls\") pod \"maas-api-e2e-api-a-addf1a-56979459f6-bt9cd\" (UID: \"5d450803-ade0-428d-9029-d71459b2fc22\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-addf1a-56979459f6-bt9cd"
Nov 03 20:28:48.734062 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:48.734027    5570 generic.go:356] "Generic (PLEG): container finished" podID="830e062e-83fc-4de7-91a7-5534dac1d46b" containerID="a24442a935efa7166e38699d8b414a492546737e47c8e40770013fb03915efb0" exitCode=0
Nov 03 20:28:48.734242 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:48.734075    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-6fc748/premium-w1-6fc748-kserve-7474db9cd7-nf2xk"
Nov 03 20:28:48.734242 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:48.734106    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-6fc748/premium-w1-6fc748-kserve-7474db9cd7-nf2xk" event={"ID":"830e062e-83fc-4de7-91a7-5534dac1d46b","Type":"ContainerDied","Data":"a24442a935efa7166e38699d8b414a492546737e47c8e40770013fb03915efb0"}
Nov 03 20:28:48.734242 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:48.734149    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-6fc748/premium-w1-6fc748-kserve-7474db9cd7-nf2xk" event={"ID":"830e062e-83fc-4de7-91a7-5534dac1d46b","Type":"ContainerDied","Data":"5fe19cda93a67d49b6b53e87a796287bcb8391ea55c870857d69d6a53ea458f6"}
Nov 03 20:28:48.734242 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:48.734167    5570 scope.go:122] "RemoveContainer" containerID="a24442a935efa7166e38699d8b414a492546737e47c8e40770013fb03915efb0"
Nov 03 20:28:48.746334 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:48.746310    5570 scope.go:122] "RemoveContainer" containerID="a24442a935efa7166e38699d8b414a492546737e47c8e40770013fb03915efb0"
Nov 03 20:28:48.746679 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:28:48.746655    5570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a24442a935efa7166e38699d8b414a492546737e47c8e40770013fb03915efb0\": container with ID starting with a24442a935efa7166e38699d8b414a492546737e47c8e40770013fb03915efb0 not found: ID does not exist" containerID="a24442a935efa7166e38699d8b414a492546737e47c8e40770013fb03915efb0"
Nov 03 20:28:48.746728 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:48.746691    5570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a24442a935efa7166e38699d8b414a492546737e47c8e40770013fb03915efb0"} err="failed to get container status \"a24442a935efa7166e38699d8b414a492546737e47c8e40770013fb03915efb0\": rpc error: code = NotFound desc = could not find container \"a24442a935efa7166e38699d8b414a492546737e47c8e40770013fb03915efb0\": container with ID starting with a24442a935efa7166e38699d8b414a492546737e47c8e40770013fb03915efb0 not found: ID does not exist"
Nov 03 20:28:48.792015 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:48.791973    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-6fc748/premium-w1-6fc748-kserve-7474db9cd7-nf2xk"]
Nov 03 20:28:48.800899 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:48.800861    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-6fc748/premium-w1-6fc748-kserve-7474db9cd7-nf2xk"]
Nov 03 20:28:48.902718 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:48.902026    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-addf1a-56979459f6-bt9cd"
Nov 03 20:28:49.361589 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:49.359819    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-addf1a-56979459f6-bt9cd"]
Nov 03 20:28:49.745368 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:49.745319    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-addf1a-56979459f6-bt9cd" event={"ID":"5d450803-ade0-428d-9029-d71459b2fc22","Type":"ContainerStarted","Data":"81f9208c3c9ccc487af5a750532b9a3dac39d3ae15210fcaab1225e4d5c85492"}
Nov 03 20:28:49.990228 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:49.987813    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="830e062e-83fc-4de7-91a7-5534dac1d46b" path="/var/lib/kubelet/pods/830e062e-83fc-4de7-91a7-5534dac1d46b/volumes"
Nov 03 20:28:50.762662 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:50.762609    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-addf1a-56979459f6-bt9cd" event={"ID":"5d450803-ade0-428d-9029-d71459b2fc22","Type":"ContainerStarted","Data":"e99ba3bf4478c424bb398929088eeb78d64e97916e71dc67e18a610c4719b75c"}
Nov 03 20:28:50.763225 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:50.762719    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-addf1a-56979459f6-bt9cd"
Nov 03 20:28:50.810080 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:50.810025    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-addf1a-56979459f6-bt9cd" podStartSLOduration=3.495006325 podStartE2EDuration="3.81000993s" totalImagesPullingTime="315.003605ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:28:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:28:50.807834227 +0000 UTC m=+1464.960496117" watchObservedRunningTime="2026-11-03 20:28:50.81000993 +0000 UTC m=+1464.962671816"
Nov 03 20:28:52.673800 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:52.673179    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-c4a71d-5c8bb448f4-jsmhw"
Nov 03 20:28:52.675264 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:52.674580    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c4a71d-665484ddc9-csmw4"
Nov 03 20:28:54.402939 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:54.402889    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/distinct-2-w0-e40816-kserve-79fb46654d-s242x" podUID="4ad9b048-925c-4d01-a839-a5f25d4dd02d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:28:54.600916 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:54.597911    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/premium-w3-482722-kserve-689dc7bd5d-x8rjp" podUID="369045c3-4d6a-49a7-9bc6-f2939883bee8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:28:54.790806 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:54.784871    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz" podUID="3a715ef1-38e9-4386-9eec-e10e8df11a10" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:28:54.928814 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:54.919520    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-2-w6-9e1bcb-kserve-7d96d9f6f6-4f7fl" podUID="56353b0f-08ee-4ada-9dc0-ac5afd5a59c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded"
Nov 03 20:28:55.713060 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:55.713029    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-eb87f429-6b49f7957f-bc4ph"
Nov 03 20:28:55.713587 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:55.713099    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-eb87f429-7fbff5576b-89srx"
Nov 03 20:28:56.920277 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:28:56.920243    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-addf1a-56979459f6-bt9cd"
Nov 03 20:28:57.307845 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:28:57.307786    5570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8bwhz" containerName="extractor"
Nov 03 20:29:05.356071 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:05.356036    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-eb87f429-59cb4f857f-xkbqm"]
Nov 03 20:29:05.356753 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:05.356737    5570 cpu_manager.go:414] "containerMap: removing container" podUID="830e062e-83fc-4de7-91a7-5534dac1d46b" containerName="main"
Nov 03 20:29:05.356869 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:05.356754    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="830e062e-83fc-4de7-91a7-5534dac1d46b" containerName="main"
Nov 03 20:29:05.356955 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:05.356943    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="830e062e-83fc-4de7-91a7-5534dac1d46b" containerName="main"
Nov 03 20:29:05.362193 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:05.362173    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-eb87f429-59cb4f857f-xkbqm"
Nov 03 20:29:05.392226 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:05.392028    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-eb87f429-serving-cert\""
Nov 03 20:29:05.397843 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:05.397812    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-eb87f429-59cb4f857f-xkbqm"]
Nov 03 20:29:05.448485 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:05.448452    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/135caa8e-c038-4c19-8876-07c5fd69a4ae-metrics-certs\") pod \"maas-api-e2e-mt-eb87f429-59cb4f857f-xkbqm\" (UID: \"135caa8e-c038-4c19-8876-07c5fd69a4ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-eb87f429-59cb4f857f-xkbqm"
Nov 03 20:29:05.448698 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:05.448619    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6f8t\" (UniqueName: \"kubernetes.io/projected/135caa8e-c038-4c19-8876-07c5fd69a4ae-kube-api-access-m6f8t\") pod \"maas-api-e2e-mt-eb87f429-59cb4f857f-xkbqm\" (UID: \"135caa8e-c038-4c19-8876-07c5fd69a4ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-eb87f429-59cb4f857f-xkbqm"
Nov 03 20:29:05.448790 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:05.448714    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/135caa8e-c038-4c19-8876-07c5fd69a4ae-maas-api-tls\") pod \"maas-api-e2e-mt-eb87f429-59cb4f857f-xkbqm\" (UID: \"135caa8e-c038-4c19-8876-07c5fd69a4ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-eb87f429-59cb4f857f-xkbqm"
Nov 03 20:29:05.448897 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:05.448882    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/135caa8e-c038-4c19-8876-07c5fd69a4ae-openshift-service-ca\") pod \"maas-api-e2e-mt-eb87f429-59cb4f857f-xkbqm\" (UID: \"135caa8e-c038-4c19-8876-07c5fd69a4ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-eb87f429-59cb4f857f-xkbqm"
Nov 03 20:29:05.550843 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:05.550770    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/135caa8e-c038-4c19-8876-07c5fd69a4ae-openshift-service-ca\") pod \"maas-api-e2e-mt-eb87f429-59cb4f857f-xkbqm\" (UID: \"135caa8e-c038-4c19-8876-07c5fd69a4ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-eb87f429-59cb4f857f-xkbqm"
Nov 03 20:29:05.551038 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:05.550941    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/135caa8e-c038-4c19-8876-07c5fd69a4ae-metrics-certs\") pod \"maas-api-e2e-mt-eb87f429-59cb4f857f-xkbqm\" (UID: \"135caa8e-c038-4c19-8876-07c5fd69a4ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-eb87f429-59cb4f857f-xkbqm"
Nov 03 20:29:05.551109 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:05.551058    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6f8t\" (UniqueName: \"kubernetes.io/projected/135caa8e-c038-4c19-8876-07c5fd69a4ae-kube-api-access-m6f8t\") pod \"maas-api-e2e-mt-eb87f429-59cb4f857f-xkbqm\" (UID: \"135caa8e-c038-4c19-8876-07c5fd69a4ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-eb87f429-59cb4f857f-xkbqm"
Nov 03 20:29:05.551175 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:05.551134    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/135caa8e-c038-4c19-8876-07c5fd69a4ae-maas-api-tls\") pod \"maas-api-e2e-mt-eb87f429-59cb4f857f-xkbqm\" (UID: \"135caa8e-c038-4c19-8876-07c5fd69a4ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-eb87f429-59cb4f857f-xkbqm"
Nov 03 20:29:05.551555 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:05.551531    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/135caa8e-c038-4c19-8876-07c5fd69a4ae-openshift-service-ca\") pod \"maas-api-e2e-mt-eb87f429-59cb4f857f-xkbqm\" (UID: \"135caa8e-c038-4c19-8876-07c5fd69a4ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-eb87f429-59cb4f857f-xkbqm"
Nov 03 20:29:05.553551 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:05.553528    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/135caa8e-c038-4c19-8876-07c5fd69a4ae-maas-api-tls\") pod \"maas-api-e2e-mt-eb87f429-59cb4f857f-xkbqm\" (UID: \"135caa8e-c038-4c19-8876-07c5fd69a4ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-eb87f429-59cb4f857f-xkbqm"
Nov 03 20:29:05.553805 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:05.553770    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/135caa8e-c038-4c19-8876-07c5fd69a4ae-metrics-certs\") pod \"maas-api-e2e-mt-eb87f429-59cb4f857f-xkbqm\" (UID: \"135caa8e-c038-4c19-8876-07c5fd69a4ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-eb87f429-59cb4f857f-xkbqm"
Nov 03 20:29:05.571764 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:05.571739    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6f8t\" (UniqueName: \"kubernetes.io/projected/135caa8e-c038-4c19-8876-07c5fd69a4ae-kube-api-access-m6f8t\") pod \"maas-api-e2e-mt-eb87f429-59cb4f857f-xkbqm\" (UID: \"135caa8e-c038-4c19-8876-07c5fd69a4ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-eb87f429-59cb4f857f-xkbqm"
Nov 03 20:29:05.690612 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:05.689366    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-eb87f429-59cb4f857f-xkbqm"
Nov 03 20:29:06.071052 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:06.071019    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-eb87f429-59cb4f857f-xkbqm"]
Nov 03 20:29:06.072022 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:29:06.071996    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod135caa8e_c038_4c19_8876_07c5fd69a4ae.slice/crio-98d773afd781bd329da775fdc91a92626cf1cc9360e314db0a90e4d562bbdec1 WatchSource:0}: Error finding container 98d773afd781bd329da775fdc91a92626cf1cc9360e314db0a90e4d562bbdec1: Status 404 returned error can't find the container with id 98d773afd781bd329da775fdc91a92626cf1cc9360e314db0a90e4d562bbdec1
Nov 03 20:29:06.345178 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:06.345090    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-eb87f429-59cb4f857f-xkbqm" event={"ID":"135caa8e-c038-4c19-8876-07c5fd69a4ae","Type":"ContainerStarted","Data":"98d773afd781bd329da775fdc91a92626cf1cc9360e314db0a90e4d562bbdec1"}
Nov 03 20:29:07.397282 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:07.397247    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-eb87f429-59cb4f857f-xkbqm" event={"ID":"135caa8e-c038-4c19-8876-07c5fd69a4ae","Type":"ContainerStarted","Data":"222ea5a0c94594722028fb848dbba5a2c3839f4b32b00a775bae04ce4be7307b"}
Nov 03 20:29:07.397693 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:07.397370    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-eb87f429-59cb4f857f-xkbqm"
Nov 03 20:29:07.441220 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:07.441160    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-eb87f429-59cb4f857f-xkbqm" podStartSLOduration=2.085786876 podStartE2EDuration="2.441146664s" totalImagesPullingTime="355.359788ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:29:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:29:07.438320797 +0000 UTC m=+1481.590982713" watchObservedRunningTime="2026-11-03 20:29:07.441146664 +0000 UTC m=+1481.593808550"
Nov 03 20:29:08.881675 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:08.881644    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c4a71d-585549556c-x2sdc"]
Nov 03 20:29:08.885938 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:08.885921    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c4a71d-585549556c-x2sdc"
Nov 03 20:29:08.903838 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:08.903806    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c4a71d-585549556c-x2sdc"]
Nov 03 20:29:08.912273 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:08.912048    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-c4a71d-serving-cert\""
Nov 03 20:29:08.970948 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:08.970907    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18e420ed-849e-4e73-b1b0-ef8167ddd383-openshift-service-ca\") pod \"maas-api-e2e-api-a-c4a71d-585549556c-x2sdc\" (UID: \"18e420ed-849e-4e73-b1b0-ef8167ddd383\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c4a71d-585549556c-x2sdc"
Nov 03 20:29:08.971143 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:08.971043    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18e420ed-849e-4e73-b1b0-ef8167ddd383-maas-api-tls\") pod \"maas-api-e2e-api-a-c4a71d-585549556c-x2sdc\" (UID: \"18e420ed-849e-4e73-b1b0-ef8167ddd383\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c4a71d-585549556c-x2sdc"
Nov 03 20:29:08.971143 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:08.971079    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmvz4\" (UniqueName: \"kubernetes.io/projected/18e420ed-849e-4e73-b1b0-ef8167ddd383-kube-api-access-kmvz4\") pod \"maas-api-e2e-api-a-c4a71d-585549556c-x2sdc\" (UID: \"18e420ed-849e-4e73-b1b0-ef8167ddd383\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c4a71d-585549556c-x2sdc"
Nov 03 20:29:08.971253 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:08.971231    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18e420ed-849e-4e73-b1b0-ef8167ddd383-metrics-certs\") pod \"maas-api-e2e-api-a-c4a71d-585549556c-x2sdc\" (UID: \"18e420ed-849e-4e73-b1b0-ef8167ddd383\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c4a71d-585549556c-x2sdc"
Nov 03 20:29:09.097167 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:09.097126    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18e420ed-849e-4e73-b1b0-ef8167ddd383-openshift-service-ca\") pod \"maas-api-e2e-api-a-c4a71d-585549556c-x2sdc\" (UID: \"18e420ed-849e-4e73-b1b0-ef8167ddd383\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c4a71d-585549556c-x2sdc"
Nov 03 20:29:09.103876 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:09.103837    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18e420ed-849e-4e73-b1b0-ef8167ddd383-maas-api-tls\") pod \"maas-api-e2e-api-a-c4a71d-585549556c-x2sdc\" (UID: \"18e420ed-849e-4e73-b1b0-ef8167ddd383\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c4a71d-585549556c-x2sdc"
Nov 03 20:29:09.104171 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:09.104143    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmvz4\" (UniqueName: \"kubernetes.io/projected/18e420ed-849e-4e73-b1b0-ef8167ddd383-kube-api-access-kmvz4\") pod \"maas-api-e2e-api-a-c4a71d-585549556c-x2sdc\" (UID: \"18e420ed-849e-4e73-b1b0-ef8167ddd383\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c4a71d-585549556c-x2sdc"
Nov 03 20:29:09.104585 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:09.104553    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18e420ed-849e-4e73-b1b0-ef8167ddd383-metrics-certs\") pod \"maas-api-e2e-api-a-c4a71d-585549556c-x2sdc\" (UID: \"18e420ed-849e-4e73-b1b0-ef8167ddd383\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c4a71d-585549556c-x2sdc"
Nov 03 20:29:09.128812 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:29:09.112602    5570 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-c4a71d-serving-cert: secret "maas-api-e2e-api-a-c4a71d-serving-cert" not found
Nov 03 20:29:09.129333 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:09.127142    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18e420ed-849e-4e73-b1b0-ef8167ddd383-openshift-service-ca\") pod \"maas-api-e2e-api-a-c4a71d-585549556c-x2sdc\" (UID: \"18e420ed-849e-4e73-b1b0-ef8167ddd383\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c4a71d-585549556c-x2sdc"
Nov 03 20:29:09.129567 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:29:09.129382    5570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18e420ed-849e-4e73-b1b0-ef8167ddd383-maas-api-tls podName:18e420ed-849e-4e73-b1b0-ef8167ddd383 nodeName:}" failed. No retries permitted until 2026-11-03 20:29:09.629336637 +0000 UTC m=+1483.781998507 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/18e420ed-849e-4e73-b1b0-ef8167ddd383-maas-api-tls") pod "maas-api-e2e-api-a-c4a71d-585549556c-x2sdc" (UID: "18e420ed-849e-4e73-b1b0-ef8167ddd383") : secret "maas-api-e2e-api-a-c4a71d-serving-cert" not found
Nov 03 20:29:09.131396 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:09.131367    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18e420ed-849e-4e73-b1b0-ef8167ddd383-metrics-certs\") pod \"maas-api-e2e-api-a-c4a71d-585549556c-x2sdc\" (UID: \"18e420ed-849e-4e73-b1b0-ef8167ddd383\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c4a71d-585549556c-x2sdc"
Nov 03 20:29:09.147743 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:09.147659    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmvz4\" (UniqueName: \"kubernetes.io/projected/18e420ed-849e-4e73-b1b0-ef8167ddd383-kube-api-access-kmvz4\") pod \"maas-api-e2e-api-a-c4a71d-585549556c-x2sdc\" (UID: \"18e420ed-849e-4e73-b1b0-ef8167ddd383\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c4a71d-585549556c-x2sdc"
Nov 03 20:29:09.721324 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:09.721293    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18e420ed-849e-4e73-b1b0-ef8167ddd383-maas-api-tls\") pod \"maas-api-e2e-api-a-c4a71d-585549556c-x2sdc\" (UID: \"18e420ed-849e-4e73-b1b0-ef8167ddd383\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c4a71d-585549556c-x2sdc"
Nov 03 20:29:09.723762 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:09.723734    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18e420ed-849e-4e73-b1b0-ef8167ddd383-maas-api-tls\") pod \"maas-api-e2e-api-a-c4a71d-585549556c-x2sdc\" (UID: \"18e420ed-849e-4e73-b1b0-ef8167ddd383\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c4a71d-585549556c-x2sdc"
Nov 03 20:29:09.804375 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:09.804228    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c4a71d-585549556c-x2sdc"
Nov 03 20:29:10.223134 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:10.214716    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c4a71d-585549556c-x2sdc"]
Nov 03 20:29:10.257916 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:29:10.257875    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18e420ed_849e_4e73_b1b0_ef8167ddd383.slice/crio-950b85956f79d239ac6555dcc0934a356f0d459c94d8f627ad556f5e06185cd4 WatchSource:0}: Error finding container 950b85956f79d239ac6555dcc0934a356f0d459c94d8f627ad556f5e06185cd4: Status 404 returned error can't find the container with id 950b85956f79d239ac6555dcc0934a356f0d459c94d8f627ad556f5e06185cd4
Nov 03 20:29:10.420007 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:10.419972    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c4a71d-585549556c-x2sdc" event={"ID":"18e420ed-849e-4e73-b1b0-ef8167ddd383","Type":"ContainerStarted","Data":"950b85956f79d239ac6555dcc0934a356f0d459c94d8f627ad556f5e06185cd4"}
Nov 03 20:29:11.429634 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:11.429596    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c4a71d-585549556c-x2sdc" event={"ID":"18e420ed-849e-4e73-b1b0-ef8167ddd383","Type":"ContainerStarted","Data":"4662eba89a9576abeccb320306f55399dd68ad648321d3b7748aa9e695abdb92"}
Nov 03 20:29:11.430621 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:11.429700    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c4a71d-585549556c-x2sdc"
Nov 03 20:29:11.471374 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:11.471309    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c4a71d-585549556c-x2sdc" podStartSLOduration=3.139615295 podStartE2EDuration="3.471291915s" totalImagesPullingTime="331.67662ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:29:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:29:11.470128782 +0000 UTC m=+1485.622790668" watchObservedRunningTime="2026-11-03 20:29:11.471291915 +0000 UTC m=+1485.623953803"
Nov 03 20:29:13.409333 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:13.409302    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-eb87f429-59cb4f857f-xkbqm"
Nov 03 20:29:13.919388 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:13.919355    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-fab871-openshift-default-698b94dc5f-l6ggn"]
Nov 03 20:29:13.924888 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:13.924857    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-fab871-openshift-default-698b94dc5f-l6ggn"
Nov 03 20:29:13.943620 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:13.943593    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-fab871-openshift-default-698b94dc5f-l6ggn"]
Nov 03 20:29:13.958270 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:13.957475    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-fab871-openshift-default-dockercfg-kr9jd\""
Nov 03 20:29:13.982440 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:13.982409    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46af9c86-b058-4323-97a2-af8d89d3dc8f-istio-data\") pod \"e2e-api-a-fab871-openshift-default-698b94dc5f-l6ggn\" (UID: \"46af9c86-b058-4323-97a2-af8d89d3dc8f\") " pod="openshift-ingress/e2e-api-a-fab871-openshift-default-698b94dc5f-l6ggn"
Nov 03 20:29:13.982622 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:13.982453    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46af9c86-b058-4323-97a2-af8d89d3dc8f-istio-token\") pod \"e2e-api-a-fab871-openshift-default-698b94dc5f-l6ggn\" (UID: \"46af9c86-b058-4323-97a2-af8d89d3dc8f\") " pod="openshift-ingress/e2e-api-a-fab871-openshift-default-698b94dc5f-l6ggn"
Nov 03 20:29:13.982622 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:13.982549    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46af9c86-b058-4323-97a2-af8d89d3dc8f-istio-podinfo\") pod \"e2e-api-a-fab871-openshift-default-698b94dc5f-l6ggn\" (UID: \"46af9c86-b058-4323-97a2-af8d89d3dc8f\") " pod="openshift-ingress/e2e-api-a-fab871-openshift-default-698b94dc5f-l6ggn"
Nov 03 20:29:13.982764 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:13.982634    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46af9c86-b058-4323-97a2-af8d89d3dc8f-credential-socket\") pod \"e2e-api-a-fab871-openshift-default-698b94dc5f-l6ggn\" (UID: \"46af9c86-b058-4323-97a2-af8d89d3dc8f\") " pod="openshift-ingress/e2e-api-a-fab871-openshift-default-698b94dc5f-l6ggn"
Nov 03 20:29:13.982764 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:13.982697    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46af9c86-b058-4323-97a2-af8d89d3dc8f-workload-socket\") pod \"e2e-api-a-fab871-openshift-default-698b94dc5f-l6ggn\" (UID: \"46af9c86-b058-4323-97a2-af8d89d3dc8f\") " pod="openshift-ingress/e2e-api-a-fab871-openshift-default-698b94dc5f-l6ggn"
Nov 03 20:29:13.982764 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:13.982758    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46af9c86-b058-4323-97a2-af8d89d3dc8f-istiod-ca-cert\") pod \"e2e-api-a-fab871-openshift-default-698b94dc5f-l6ggn\" (UID: \"46af9c86-b058-4323-97a2-af8d89d3dc8f\") " pod="openshift-ingress/e2e-api-a-fab871-openshift-default-698b94dc5f-l6ggn"
Nov 03 20:29:13.982949 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:13.982923    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46af9c86-b058-4323-97a2-af8d89d3dc8f-workload-certs\") pod \"e2e-api-a-fab871-openshift-default-698b94dc5f-l6ggn\" (UID: \"46af9c86-b058-4323-97a2-af8d89d3dc8f\") " pod="openshift-ingress/e2e-api-a-fab871-openshift-default-698b94dc5f-l6ggn"
Nov 03 20:29:13.983011 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:13.982994    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztn88\" (UniqueName: \"kubernetes.io/projected/46af9c86-b058-4323-97a2-af8d89d3dc8f-kube-api-access-ztn88\") pod \"e2e-api-a-fab871-openshift-default-698b94dc5f-l6ggn\" (UID: \"46af9c86-b058-4323-97a2-af8d89d3dc8f\") " pod="openshift-ingress/e2e-api-a-fab871-openshift-default-698b94dc5f-l6ggn"
Nov 03 20:29:13.983170 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:13.983154    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46af9c86-b058-4323-97a2-af8d89d3dc8f-istio-envoy\") pod \"e2e-api-a-fab871-openshift-default-698b94dc5f-l6ggn\" (UID: \"46af9c86-b058-4323-97a2-af8d89d3dc8f\") " pod="openshift-ingress/e2e-api-a-fab871-openshift-default-698b94dc5f-l6ggn"
Nov 03 20:29:14.094259 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:14.094212    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46af9c86-b058-4323-97a2-af8d89d3dc8f-istio-envoy\") pod \"e2e-api-a-fab871-openshift-default-698b94dc5f-l6ggn\" (UID: \"46af9c86-b058-4323-97a2-af8d89d3dc8f\") " pod="openshift-ingress/e2e-api-a-fab871-openshift-default-698b94dc5f-l6ggn"
Nov 03 20:29:14.094760 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:14.094714    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46af9c86-b058-4323-97a2-af8d89d3dc8f-istio-data\") pod \"e2e-api-a-fab871-openshift-default-698b94dc5f-l6ggn\" (UID: \"46af9c86-b058-4323-97a2-af8d89d3dc8f\") " pod="openshift-ingress/e2e-api-a-fab871-openshift-default-698b94dc5f-l6ggn"
Nov 03 20:29:14.094760 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:14.094806    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46af9c86-b058-4323-97a2-af8d89d3dc8f-istio-token\") pod \"e2e-api-a-fab871-openshift-default-698b94dc5f-l6ggn\" (UID: \"46af9c86-b058-4323-97a2-af8d89d3dc8f\") " pod="openshift-ingress/e2e-api-a-fab871-openshift-default-698b94dc5f-l6ggn"
Nov 03 20:29:14.094760 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:14.094855    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46af9c86-b058-4323-97a2-af8d89d3dc8f-istio-podinfo\") pod \"e2e-api-a-fab871-openshift-default-698b94dc5f-l6ggn\" (UID: \"46af9c86-b058-4323-97a2-af8d89d3dc8f\") " pod="openshift-ingress/e2e-api-a-fab871-openshift-default-698b94dc5f-l6ggn"
Nov 03 20:29:14.094760 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:14.094930    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46af9c86-b058-4323-97a2-af8d89d3dc8f-credential-socket\") pod \"e2e-api-a-fab871-openshift-default-698b94dc5f-l6ggn\" (UID: \"46af9c86-b058-4323-97a2-af8d89d3dc8f\") " pod="openshift-ingress/e2e-api-a-fab871-openshift-default-698b94dc5f-l6ggn"
Nov 03 20:29:14.095300 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:14.094973    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46af9c86-b058-4323-97a2-af8d89d3dc8f-workload-socket\") pod \"e2e-api-a-fab871-openshift-default-698b94dc5f-l6ggn\" (UID: \"46af9c86-b058-4323-97a2-af8d89d3dc8f\") " pod="openshift-ingress/e2e-api-a-fab871-openshift-default-698b94dc5f-l6ggn"
Nov 03 20:29:14.095300 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:14.095039    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46af9c86-b058-4323-97a2-af8d89d3dc8f-istiod-ca-cert\") pod \"e2e-api-a-fab871-openshift-default-698b94dc5f-l6ggn\" (UID: \"46af9c86-b058-4323-97a2-af8d89d3dc8f\") " pod="openshift-ingress/e2e-api-a-fab871-openshift-default-698b94dc5f-l6ggn"
Nov 03 20:29:14.095300 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:14.095143    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46af9c86-b058-4323-97a2-af8d89d3dc8f-workload-certs\") pod \"e2e-api-a-fab871-openshift-default-698b94dc5f-l6ggn\" (UID: \"46af9c86-b058-4323-97a2-af8d89d3dc8f\") " pod="openshift-ingress/e2e-api-a-fab871-openshift-default-698b94dc5f-l6ggn"
Nov 03 20:29:14.095300 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:14.095248    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztn88\" (UniqueName: \"kubernetes.io/projected/46af9c86-b058-4323-97a2-af8d89d3dc8f-kube-api-access-ztn88\") pod \"e2e-api-a-fab871-openshift-default-698b94dc5f-l6ggn\" (UID: \"46af9c86-b058-4323-97a2-af8d89d3dc8f\") " pod="openshift-ingress/e2e-api-a-fab871-openshift-default-698b94dc5f-l6ggn"
Nov 03 20:29:14.097364 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:14.097277    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46af9c86-b058-4323-97a2-af8d89d3dc8f-istiod-ca-cert\") pod \"e2e-api-a-fab871-openshift-default-698b94dc5f-l6ggn\" (UID: \"46af9c86-b058-4323-97a2-af8d89d3dc8f\") " pod="openshift-ingress/e2e-api-a-fab871-openshift-default-698b94dc5f-l6ggn"
Nov 03 20:29:14.099262 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:14.097858    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46af9c86-b058-4323-97a2-af8d89d3dc8f-workload-socket\") pod \"e2e-api-a-fab871-openshift-default-698b94dc5f-l6ggn\" (UID: \"46af9c86-b058-4323-97a2-af8d89d3dc8f\") " pod="openshift-ingress/e2e-api-a-fab871-openshift-default-698b94dc5f-l6ggn"
Nov 03 20:29:14.099262 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:14.098584    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46af9c86-b058-4323-97a2-af8d89d3dc8f-istio-data\") pod \"e2e-api-a-fab871-openshift-default-698b94dc5f-l6ggn\" (UID: \"46af9c86-b058-4323-97a2-af8d89d3dc8f\") " pod="openshift-ingress/e2e-api-a-fab871-openshift-default-698b94dc5f-l6ggn"
Nov 03 20:29:14.099262 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:14.098860    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46af9c86-b058-4323-97a2-af8d89d3dc8f-workload-certs\") pod \"e2e-api-a-fab871-openshift-default-698b94dc5f-l6ggn\" (UID: \"46af9c86-b058-4323-97a2-af8d89d3dc8f\") " pod="openshift-ingress/e2e-api-a-fab871-openshift-default-698b94dc5f-l6ggn"
Nov 03 20:29:14.101154 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:14.101131    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46af9c86-b058-4323-97a2-af8d89d3dc8f-credential-socket\") pod \"e2e-api-a-fab871-openshift-default-698b94dc5f-l6ggn\" (UID: \"46af9c86-b058-4323-97a2-af8d89d3dc8f\") " pod="openshift-ingress/e2e-api-a-fab871-openshift-default-698b94dc5f-l6ggn"
Nov 03 20:29:14.144552 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:14.142115    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46af9c86-b058-4323-97a2-af8d89d3dc8f-istio-envoy\") pod \"e2e-api-a-fab871-openshift-default-698b94dc5f-l6ggn\" (UID: \"46af9c86-b058-4323-97a2-af8d89d3dc8f\") " pod="openshift-ingress/e2e-api-a-fab871-openshift-default-698b94dc5f-l6ggn"
Nov 03 20:29:14.145378 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:14.145315    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46af9c86-b058-4323-97a2-af8d89d3dc8f-istio-podinfo\") pod \"e2e-api-a-fab871-openshift-default-698b94dc5f-l6ggn\" (UID: \"46af9c86-b058-4323-97a2-af8d89d3dc8f\") " pod="openshift-ingress/e2e-api-a-fab871-openshift-default-698b94dc5f-l6ggn"
Nov 03 20:29:14.157820 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:14.145657    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztn88\" (UniqueName: \"kubernetes.io/projected/46af9c86-b058-4323-97a2-af8d89d3dc8f-kube-api-access-ztn88\") pod \"e2e-api-a-fab871-openshift-default-698b94dc5f-l6ggn\" (UID: \"46af9c86-b058-4323-97a2-af8d89d3dc8f\") " pod="openshift-ingress/e2e-api-a-fab871-openshift-default-698b94dc5f-l6ggn"
Nov 03 20:29:14.183201 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:14.182140    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46af9c86-b058-4323-97a2-af8d89d3dc8f-istio-token\") pod \"e2e-api-a-fab871-openshift-default-698b94dc5f-l6ggn\" (UID: \"46af9c86-b058-4323-97a2-af8d89d3dc8f\") " pod="openshift-ingress/e2e-api-a-fab871-openshift-default-698b94dc5f-l6ggn"
Nov 03 20:29:14.254396 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:14.254108    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-fab871-openshift-default-698b94dc5f-l6ggn"
Nov 03 20:29:14.844918 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:14.844884    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-fab871-openshift-default-698b94dc5f-l6ggn"]
Nov 03 20:29:14.845342 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:29:14.845320    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46af9c86_b058_4323_97a2_af8d89d3dc8f.slice/crio-79aac0184b8eb138f726e3ecd811259ca798561e8b42ade7acc18c7218a9f64b WatchSource:0}: Error finding container 79aac0184b8eb138f726e3ecd811259ca798561e8b42ade7acc18c7218a9f64b: Status 404 returned error can't find the container with id 79aac0184b8eb138f726e3ecd811259ca798561e8b42ade7acc18c7218a9f64b
Nov 03 20:29:14.848634 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:14.848599    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:29:14.848767 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:14.848682    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:29:14.848767 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:14.848727    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:29:15.485083 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:15.485043    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-fab871-openshift-default-698b94dc5f-l6ggn" event={"ID":"46af9c86-b058-4323-97a2-af8d89d3dc8f","Type":"ContainerStarted","Data":"3fef9e5c8d3757cb3a7f4684106e6b30c1b1f324ad8bb942240b0cf6e9a436d8"}
Nov 03 20:29:15.485262 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:15.485135    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-fab871-openshift-default-698b94dc5f-l6ggn" event={"ID":"46af9c86-b058-4323-97a2-af8d89d3dc8f","Type":"ContainerStarted","Data":"79aac0184b8eb138f726e3ecd811259ca798561e8b42ade7acc18c7218a9f64b"}
Nov 03 20:29:16.255538 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:16.255502    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-fab871-openshift-default-698b94dc5f-l6ggn"
Nov 03 20:29:16.261748 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:16.261723    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-fab871-openshift-default-698b94dc5f-l6ggn"
Nov 03 20:29:16.335699 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:16.335639    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-fab871-openshift-default-698b94dc5f-l6ggn" podStartSLOduration=3.335624002 podStartE2EDuration="3.335624002s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:29:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:29:15.569328973 +0000 UTC m=+1489.721990860" watchObservedRunningTime="2026-11-03 20:29:16.335624002 +0000 UTC m=+1490.488285889"
Nov 03 20:29:16.492627 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:16.492587    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-fab871-openshift-default-698b94dc5f-l6ggn"
Nov 03 20:29:16.493523 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:16.493498    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-fab871-openshift-default-698b94dc5f-l6ggn"
Nov 03 20:29:17.455025 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:17.454993    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c4a71d-585549556c-x2sdc"
Nov 03 20:29:19.751710 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:19.751679    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-fab871-5c8bd9f8dc-qv2jl"]
Nov 03 20:29:19.756643 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:19.756621    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fab871-5c8bd9f8dc-qv2jl"
Nov 03 20:29:19.772235 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:19.772207    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-fab871-5c8bd9f8dc-qv2jl"]
Nov 03 20:29:19.780203 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:19.780176    5570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-fab871\""
Nov 03 20:29:19.868181 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:19.868122    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-fab871-dockercfg-g5vsc\""
Nov 03 20:29:19.956801 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:19.894431    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mlbr\" (UniqueName: \"kubernetes.io/projected/ef03b5c7-3817-4d77-b2c8-e450267276cc-kube-api-access-9mlbr\") pod \"payload-pre-processing-e2e-api-a-fab871-5c8bd9f8dc-qv2jl\" (UID: \"ef03b5c7-3817-4d77-b2c8-e450267276cc\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-fab871-5c8bd9f8dc-qv2jl"
Nov 03 20:29:19.968798 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:19.957344    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef03b5c7-3817-4d77-b2c8-e450267276cc-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-fab871-5c8bd9f8dc-qv2jl\" (UID: \"ef03b5c7-3817-4d77-b2c8-e450267276cc\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-fab871-5c8bd9f8dc-qv2jl"
Nov 03 20:29:20.114855 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:20.101314    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mlbr\" (UniqueName: \"kubernetes.io/projected/ef03b5c7-3817-4d77-b2c8-e450267276cc-kube-api-access-9mlbr\") pod \"payload-pre-processing-e2e-api-a-fab871-5c8bd9f8dc-qv2jl\" (UID: \"ef03b5c7-3817-4d77-b2c8-e450267276cc\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-fab871-5c8bd9f8dc-qv2jl"
Nov 03 20:29:20.114855 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:20.101507    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef03b5c7-3817-4d77-b2c8-e450267276cc-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-fab871-5c8bd9f8dc-qv2jl\" (UID: \"ef03b5c7-3817-4d77-b2c8-e450267276cc\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-fab871-5c8bd9f8dc-qv2jl"
Nov 03 20:29:20.164809 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:20.144634    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef03b5c7-3817-4d77-b2c8-e450267276cc-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-fab871-5c8bd9f8dc-qv2jl\" (UID: \"ef03b5c7-3817-4d77-b2c8-e450267276cc\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-fab871-5c8bd9f8dc-qv2jl"
Nov 03 20:29:20.221952 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:20.217509    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mlbr\" (UniqueName: \"kubernetes.io/projected/ef03b5c7-3817-4d77-b2c8-e450267276cc-kube-api-access-9mlbr\") pod \"payload-pre-processing-e2e-api-a-fab871-5c8bd9f8dc-qv2jl\" (UID: \"ef03b5c7-3817-4d77-b2c8-e450267276cc\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-fab871-5c8bd9f8dc-qv2jl"
Nov 03 20:29:20.240557 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:20.240523    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-fab871-5c95886dbd-x49r6"]
Nov 03 20:29:20.245281 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:20.245259    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-fab871-5c95886dbd-x49r6"]
Nov 03 20:29:20.245405 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:20.245344    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-fab871-5c95886dbd-x49r6"
Nov 03 20:29:20.377186 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:20.377092    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fab871-5c8bd9f8dc-qv2jl"
Nov 03 20:29:20.407324 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:20.407291    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b83e8f6-3b31-427c-8c6c-57e57d1bc4d4-plugins-config-volume\") pod \"payload-processing-e2e-api-a-fab871-5c95886dbd-x49r6\" (UID: \"1b83e8f6-3b31-427c-8c6c-57e57d1bc4d4\") " pod="openshift-ingress/payload-processing-e2e-api-a-fab871-5c95886dbd-x49r6"
Nov 03 20:29:20.407471 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:20.407434    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcnvf\" (UniqueName: \"kubernetes.io/projected/1b83e8f6-3b31-427c-8c6c-57e57d1bc4d4-kube-api-access-fcnvf\") pod \"payload-processing-e2e-api-a-fab871-5c95886dbd-x49r6\" (UID: \"1b83e8f6-3b31-427c-8c6c-57e57d1bc4d4\") " pod="openshift-ingress/payload-processing-e2e-api-a-fab871-5c95886dbd-x49r6"
Nov 03 20:29:20.509287 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:20.509253    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b83e8f6-3b31-427c-8c6c-57e57d1bc4d4-plugins-config-volume\") pod \"payload-processing-e2e-api-a-fab871-5c95886dbd-x49r6\" (UID: \"1b83e8f6-3b31-427c-8c6c-57e57d1bc4d4\") " pod="openshift-ingress/payload-processing-e2e-api-a-fab871-5c95886dbd-x49r6"
Nov 03 20:29:20.509472 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:20.509334    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcnvf\" (UniqueName: \"kubernetes.io/projected/1b83e8f6-3b31-427c-8c6c-57e57d1bc4d4-kube-api-access-fcnvf\") pod \"payload-processing-e2e-api-a-fab871-5c95886dbd-x49r6\" (UID: \"1b83e8f6-3b31-427c-8c6c-57e57d1bc4d4\") " pod="openshift-ingress/payload-processing-e2e-api-a-fab871-5c95886dbd-x49r6"
Nov 03 20:29:20.510041 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:20.510022    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b83e8f6-3b31-427c-8c6c-57e57d1bc4d4-plugins-config-volume\") pod \"payload-processing-e2e-api-a-fab871-5c95886dbd-x49r6\" (UID: \"1b83e8f6-3b31-427c-8c6c-57e57d1bc4d4\") " pod="openshift-ingress/payload-processing-e2e-api-a-fab871-5c95886dbd-x49r6"
Nov 03 20:29:20.529622 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:20.529585    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcnvf\" (UniqueName: \"kubernetes.io/projected/1b83e8f6-3b31-427c-8c6c-57e57d1bc4d4-kube-api-access-fcnvf\") pod \"payload-processing-e2e-api-a-fab871-5c95886dbd-x49r6\" (UID: \"1b83e8f6-3b31-427c-8c6c-57e57d1bc4d4\") " pod="openshift-ingress/payload-processing-e2e-api-a-fab871-5c95886dbd-x49r6"
Nov 03 20:29:20.561386 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:20.561358    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-fab871-5c95886dbd-x49r6"
Nov 03 20:29:20.682080 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:20.682048    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-fab871-5c8bd9f8dc-qv2jl"]
Nov 03 20:29:20.687581 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:29:20.687542    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef03b5c7_3817_4d77_b2c8_e450267276cc.slice/crio-c63c5161e9acba9128d56132603bba955ef50833ae0165eec30db2bce3d9a56d WatchSource:0}: Error finding container c63c5161e9acba9128d56132603bba955ef50833ae0165eec30db2bce3d9a56d: Status 404 returned error can't find the container with id c63c5161e9acba9128d56132603bba955ef50833ae0165eec30db2bce3d9a56d
Nov 03 20:29:20.868990 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:20.868965    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-fab871-5c95886dbd-x49r6"]
Nov 03 20:29:20.871996 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:29:20.871967    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b83e8f6_3b31_427c_8c6c_57e57d1bc4d4.slice/crio-f524d4b41605a282ffa347ade405170c16f829ef0f24f5060702df5626de69ed WatchSource:0}: Error finding container f524d4b41605a282ffa347ade405170c16f829ef0f24f5060702df5626de69ed: Status 404 returned error can't find the container with id f524d4b41605a282ffa347ade405170c16f829ef0f24f5060702df5626de69ed
Nov 03 20:29:21.625870 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:21.625763    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fab871-5c8bd9f8dc-qv2jl" event={"ID":"ef03b5c7-3817-4d77-b2c8-e450267276cc","Type":"ContainerStarted","Data":"a34dcc00f4c853fe62f1d9a0716d2150d699d4107fbb70da668bbb6965c676d9"}
Nov 03 20:29:21.626101 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:21.625897    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fab871-5c8bd9f8dc-qv2jl" event={"ID":"ef03b5c7-3817-4d77-b2c8-e450267276cc","Type":"ContainerStarted","Data":"c63c5161e9acba9128d56132603bba955ef50833ae0165eec30db2bce3d9a56d"}
Nov 03 20:29:21.627081 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:21.626716    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fab871-5c8bd9f8dc-qv2jl"
Nov 03 20:29:21.636056 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:21.629459    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-fab871-5c95886dbd-x49r6" event={"ID":"1b83e8f6-3b31-427c-8c6c-57e57d1bc4d4","Type":"ContainerStarted","Data":"f8a37f0f08f601fa35ac6be2ab6ab87b4c71ce32b11bb77a0c2d8381bfb28596"}
Nov 03 20:29:21.636056 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:21.629514    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-fab871-5c95886dbd-x49r6" event={"ID":"1b83e8f6-3b31-427c-8c6c-57e57d1bc4d4","Type":"ContainerStarted","Data":"f524d4b41605a282ffa347ade405170c16f829ef0f24f5060702df5626de69ed"}
Nov 03 20:29:21.636056 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:21.630166    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-fab871-5c95886dbd-x49r6"
Nov 03 20:29:21.672028 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:21.671975    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fab871-5c8bd9f8dc-qv2jl" podStartSLOduration=2.6719631440000002 podStartE2EDuration="2.671963144s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:29:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:29:21.670148069 +0000 UTC m=+1495.822809956" watchObservedRunningTime="2026-11-03 20:29:21.671963144 +0000 UTC m=+1495.824625031"
Nov 03 20:29:21.714328 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:21.714274    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-fab871-5c95886dbd-x49r6" podStartSLOduration=2.714256813 podStartE2EDuration="2.714256813s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:29:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:29:21.711037394 +0000 UTC m=+1495.863699280" watchObservedRunningTime="2026-11-03 20:29:21.714256813 +0000 UTC m=+1495.866918702"
Nov 03 20:29:24.566789 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:24.566742    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-d1aa66-openshift-default-8457c96657-lt2kw"]
Nov 03 20:29:24.572788 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:24.572744    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d1aa66-openshift-default-8457c96657-lt2kw"
Nov 03 20:29:24.592528 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:24.592477    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-d1aa66-openshift-default-dockercfg-5dgnx\""
Nov 03 20:29:24.596517 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:24.596487    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-d1aa66-openshift-default-8457c96657-lt2kw"]
Nov 03 20:29:24.620146 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:24.620107    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a701695-c67d-4cda-a46d-7834b22750ed-workload-socket\") pod \"e2e-api-b-d1aa66-openshift-default-8457c96657-lt2kw\" (UID: \"5a701695-c67d-4cda-a46d-7834b22750ed\") " pod="openshift-ingress/e2e-api-b-d1aa66-openshift-default-8457c96657-lt2kw"
Nov 03 20:29:24.620306 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:24.620233    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a701695-c67d-4cda-a46d-7834b22750ed-credential-socket\") pod \"e2e-api-b-d1aa66-openshift-default-8457c96657-lt2kw\" (UID: \"5a701695-c67d-4cda-a46d-7834b22750ed\") " pod="openshift-ingress/e2e-api-b-d1aa66-openshift-default-8457c96657-lt2kw"
Nov 03 20:29:24.620306 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:24.620276    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a701695-c67d-4cda-a46d-7834b22750ed-istio-data\") pod \"e2e-api-b-d1aa66-openshift-default-8457c96657-lt2kw\" (UID: \"5a701695-c67d-4cda-a46d-7834b22750ed\") " pod="openshift-ingress/e2e-api-b-d1aa66-openshift-default-8457c96657-lt2kw"
Nov 03 20:29:24.620306 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:24.620305    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a701695-c67d-4cda-a46d-7834b22750ed-istio-envoy\") pod \"e2e-api-b-d1aa66-openshift-default-8457c96657-lt2kw\" (UID: \"5a701695-c67d-4cda-a46d-7834b22750ed\") " pod="openshift-ingress/e2e-api-b-d1aa66-openshift-default-8457c96657-lt2kw"
Nov 03 20:29:24.620523 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:24.620320    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a701695-c67d-4cda-a46d-7834b22750ed-istiod-ca-cert\") pod \"e2e-api-b-d1aa66-openshift-default-8457c96657-lt2kw\" (UID: \"5a701695-c67d-4cda-a46d-7834b22750ed\") " pod="openshift-ingress/e2e-api-b-d1aa66-openshift-default-8457c96657-lt2kw"
Nov 03 20:29:24.620523 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:24.620392    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a701695-c67d-4cda-a46d-7834b22750ed-workload-certs\") pod \"e2e-api-b-d1aa66-openshift-default-8457c96657-lt2kw\" (UID: \"5a701695-c67d-4cda-a46d-7834b22750ed\") " pod="openshift-ingress/e2e-api-b-d1aa66-openshift-default-8457c96657-lt2kw"
Nov 03 20:29:24.620640 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:24.620557    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a701695-c67d-4cda-a46d-7834b22750ed-istio-token\") pod \"e2e-api-b-d1aa66-openshift-default-8457c96657-lt2kw\" (UID: \"5a701695-c67d-4cda-a46d-7834b22750ed\") " pod="openshift-ingress/e2e-api-b-d1aa66-openshift-default-8457c96657-lt2kw"
Nov 03 20:29:24.620699 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:24.620664    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbcw7\" (UniqueName: \"kubernetes.io/projected/5a701695-c67d-4cda-a46d-7834b22750ed-kube-api-access-bbcw7\") pod \"e2e-api-b-d1aa66-openshift-default-8457c96657-lt2kw\" (UID: \"5a701695-c67d-4cda-a46d-7834b22750ed\") " pod="openshift-ingress/e2e-api-b-d1aa66-openshift-default-8457c96657-lt2kw"
Nov 03 20:29:24.620852 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:24.620836    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a701695-c67d-4cda-a46d-7834b22750ed-istio-podinfo\") pod \"e2e-api-b-d1aa66-openshift-default-8457c96657-lt2kw\" (UID: \"5a701695-c67d-4cda-a46d-7834b22750ed\") " pod="openshift-ingress/e2e-api-b-d1aa66-openshift-default-8457c96657-lt2kw"
Nov 03 20:29:24.722000 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:24.721961    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbcw7\" (UniqueName: \"kubernetes.io/projected/5a701695-c67d-4cda-a46d-7834b22750ed-kube-api-access-bbcw7\") pod \"e2e-api-b-d1aa66-openshift-default-8457c96657-lt2kw\" (UID: \"5a701695-c67d-4cda-a46d-7834b22750ed\") " pod="openshift-ingress/e2e-api-b-d1aa66-openshift-default-8457c96657-lt2kw"
Nov 03 20:29:24.722199 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:24.722112    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a701695-c67d-4cda-a46d-7834b22750ed-istio-podinfo\") pod \"e2e-api-b-d1aa66-openshift-default-8457c96657-lt2kw\" (UID: \"5a701695-c67d-4cda-a46d-7834b22750ed\") " pod="openshift-ingress/e2e-api-b-d1aa66-openshift-default-8457c96657-lt2kw"
Nov 03 20:29:24.722312 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:24.722297    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a701695-c67d-4cda-a46d-7834b22750ed-workload-socket\") pod \"e2e-api-b-d1aa66-openshift-default-8457c96657-lt2kw\" (UID: \"5a701695-c67d-4cda-a46d-7834b22750ed\") " pod="openshift-ingress/e2e-api-b-d1aa66-openshift-default-8457c96657-lt2kw"
Nov 03 20:29:24.722406 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:24.722392    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a701695-c67d-4cda-a46d-7834b22750ed-credential-socket\") pod \"e2e-api-b-d1aa66-openshift-default-8457c96657-lt2kw\" (UID: \"5a701695-c67d-4cda-a46d-7834b22750ed\") " pod="openshift-ingress/e2e-api-b-d1aa66-openshift-default-8457c96657-lt2kw"
Nov 03 20:29:24.722479 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:24.722431    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a701695-c67d-4cda-a46d-7834b22750ed-istio-data\") pod \"e2e-api-b-d1aa66-openshift-default-8457c96657-lt2kw\" (UID: \"5a701695-c67d-4cda-a46d-7834b22750ed\") " pod="openshift-ingress/e2e-api-b-d1aa66-openshift-default-8457c96657-lt2kw"
Nov 03 20:29:24.722479 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:24.722470    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a701695-c67d-4cda-a46d-7834b22750ed-istio-envoy\") pod \"e2e-api-b-d1aa66-openshift-default-8457c96657-lt2kw\" (UID: \"5a701695-c67d-4cda-a46d-7834b22750ed\") " pod="openshift-ingress/e2e-api-b-d1aa66-openshift-default-8457c96657-lt2kw"
Nov 03 20:29:24.722587 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:24.722493    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a701695-c67d-4cda-a46d-7834b22750ed-istiod-ca-cert\") pod \"e2e-api-b-d1aa66-openshift-default-8457c96657-lt2kw\" (UID: \"5a701695-c67d-4cda-a46d-7834b22750ed\") " pod="openshift-ingress/e2e-api-b-d1aa66-openshift-default-8457c96657-lt2kw"
Nov 03 20:29:24.722587 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:24.722533    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a701695-c67d-4cda-a46d-7834b22750ed-workload-certs\") pod \"e2e-api-b-d1aa66-openshift-default-8457c96657-lt2kw\" (UID: \"5a701695-c67d-4cda-a46d-7834b22750ed\") " pod="openshift-ingress/e2e-api-b-d1aa66-openshift-default-8457c96657-lt2kw"
Nov 03 20:29:24.723405 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:24.722606    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a701695-c67d-4cda-a46d-7834b22750ed-istio-token\") pod \"e2e-api-b-d1aa66-openshift-default-8457c96657-lt2kw\" (UID: \"5a701695-c67d-4cda-a46d-7834b22750ed\") " pod="openshift-ingress/e2e-api-b-d1aa66-openshift-default-8457c96657-lt2kw"
Nov 03 20:29:24.723405 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:24.722728    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a701695-c67d-4cda-a46d-7834b22750ed-workload-socket\") pod \"e2e-api-b-d1aa66-openshift-default-8457c96657-lt2kw\" (UID: \"5a701695-c67d-4cda-a46d-7834b22750ed\") " pod="openshift-ingress/e2e-api-b-d1aa66-openshift-default-8457c96657-lt2kw"
Nov 03 20:29:24.723405 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:24.722841    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a701695-c67d-4cda-a46d-7834b22750ed-istio-data\") pod \"e2e-api-b-d1aa66-openshift-default-8457c96657-lt2kw\" (UID: \"5a701695-c67d-4cda-a46d-7834b22750ed\") " pod="openshift-ingress/e2e-api-b-d1aa66-openshift-default-8457c96657-lt2kw"
Nov 03 20:29:24.723405 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:24.722851    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a701695-c67d-4cda-a46d-7834b22750ed-credential-socket\") pod \"e2e-api-b-d1aa66-openshift-default-8457c96657-lt2kw\" (UID: \"5a701695-c67d-4cda-a46d-7834b22750ed\") " pod="openshift-ingress/e2e-api-b-d1aa66-openshift-default-8457c96657-lt2kw"
Nov 03 20:29:24.723405 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:24.723122    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a701695-c67d-4cda-a46d-7834b22750ed-workload-certs\") pod \"e2e-api-b-d1aa66-openshift-default-8457c96657-lt2kw\" (UID: \"5a701695-c67d-4cda-a46d-7834b22750ed\") " pod="openshift-ingress/e2e-api-b-d1aa66-openshift-default-8457c96657-lt2kw"
Nov 03 20:29:24.723405 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:24.723348    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a701695-c67d-4cda-a46d-7834b22750ed-istiod-ca-cert\") pod \"e2e-api-b-d1aa66-openshift-default-8457c96657-lt2kw\" (UID: \"5a701695-c67d-4cda-a46d-7834b22750ed\") " pod="openshift-ingress/e2e-api-b-d1aa66-openshift-default-8457c96657-lt2kw"
Nov 03 20:29:24.724759 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:24.724733    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a701695-c67d-4cda-a46d-7834b22750ed-istio-envoy\") pod \"e2e-api-b-d1aa66-openshift-default-8457c96657-lt2kw\" (UID: \"5a701695-c67d-4cda-a46d-7834b22750ed\") " pod="openshift-ingress/e2e-api-b-d1aa66-openshift-default-8457c96657-lt2kw"
Nov 03 20:29:24.724885 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:24.724822    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a701695-c67d-4cda-a46d-7834b22750ed-istio-podinfo\") pod \"e2e-api-b-d1aa66-openshift-default-8457c96657-lt2kw\" (UID: \"5a701695-c67d-4cda-a46d-7834b22750ed\") " pod="openshift-ingress/e2e-api-b-d1aa66-openshift-default-8457c96657-lt2kw"
Nov 03 20:29:24.887875 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:24.875599    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbcw7\" (UniqueName: \"kubernetes.io/projected/5a701695-c67d-4cda-a46d-7834b22750ed-kube-api-access-bbcw7\") pod \"e2e-api-b-d1aa66-openshift-default-8457c96657-lt2kw\" (UID: \"5a701695-c67d-4cda-a46d-7834b22750ed\") " pod="openshift-ingress/e2e-api-b-d1aa66-openshift-default-8457c96657-lt2kw"
Nov 03 20:29:24.961806 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:24.961646    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a701695-c67d-4cda-a46d-7834b22750ed-istio-token\") pod \"e2e-api-b-d1aa66-openshift-default-8457c96657-lt2kw\" (UID: \"5a701695-c67d-4cda-a46d-7834b22750ed\") " pod="openshift-ingress/e2e-api-b-d1aa66-openshift-default-8457c96657-lt2kw"
Nov 03 20:29:25.208300 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:25.208245    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d1aa66-openshift-default-8457c96657-lt2kw"
Nov 03 20:29:25.552667 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:25.552636    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-d1aa66-openshift-default-8457c96657-lt2kw"]
Nov 03 20:29:25.554479 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:29:25.554452    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a701695_c67d_4cda_a46d_7834b22750ed.slice/crio-251e355a12f83f023b75ede730b776523c78acdb5d02bcb71d3e83a86b354ffe WatchSource:0}: Error finding container 251e355a12f83f023b75ede730b776523c78acdb5d02bcb71d3e83a86b354ffe: Status 404 returned error can't find the container with id 251e355a12f83f023b75ede730b776523c78acdb5d02bcb71d3e83a86b354ffe
Nov 03 20:29:25.557454 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:25.557424    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:29:25.557529 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:25.557492    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:29:25.557529 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:25.557522    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:29:25.707192 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:25.707146    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d1aa66-openshift-default-8457c96657-lt2kw" event={"ID":"5a701695-c67d-4cda-a46d-7834b22750ed","Type":"ContainerStarted","Data":"8aaebcb8af28523aa8fb9da47a8dd3c1e0d18b4d1ea123656dcb34aa2d36d88c"}
Nov 03 20:29:25.707706 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:25.707220    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d1aa66-openshift-default-8457c96657-lt2kw" event={"ID":"5a701695-c67d-4cda-a46d-7834b22750ed","Type":"ContainerStarted","Data":"251e355a12f83f023b75ede730b776523c78acdb5d02bcb71d3e83a86b354ffe"}
Nov 03 20:29:25.759953 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:25.759894    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-d1aa66-openshift-default-8457c96657-lt2kw" podStartSLOduration=1.759877639 podStartE2EDuration="1.759877639s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:29:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:29:25.753810202 +0000 UTC m=+1499.906472088" watchObservedRunningTime="2026-11-03 20:29:25.759877639 +0000 UTC m=+1499.912539526"
Nov 03 20:29:26.209275 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:26.209238    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-d1aa66-openshift-default-8457c96657-lt2kw"
Nov 03 20:29:26.217225 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:26.217195    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-d1aa66-openshift-default-8457c96657-lt2kw"
Nov 03 20:29:26.405603 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:26.405562    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-dc7b94-667b5756b8-ns2z5"]
Nov 03 20:29:26.406208 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:26.405872    5570 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dc7b94-667b5756b8-ns2z5" podUID="2635dffb-14f1-49cb-8d32-8f539c28ed30" containerName="maas-api" containerID="cri-o://e2d6437514e7ae9649085f2ab58a2c8c15874dadd45dda93d9e35c8a43ca8260" gracePeriod=30
Nov 03 20:29:27.035094 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:27.035049    5570 generic.go:356] "Generic (PLEG): container finished" podID="2635dffb-14f1-49cb-8d32-8f539c28ed30" containerID="e2d6437514e7ae9649085f2ab58a2c8c15874dadd45dda93d9e35c8a43ca8260" exitCode=0
Nov 03 20:29:27.056849 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:27.035929    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dc7b94-667b5756b8-ns2z5" event={"ID":"2635dffb-14f1-49cb-8d32-8f539c28ed30","Type":"ContainerDied","Data":"e2d6437514e7ae9649085f2ab58a2c8c15874dadd45dda93d9e35c8a43ca8260"}
Nov 03 20:29:27.056849 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:27.044466    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-d1aa66-openshift-default-8457c96657-lt2kw"
Nov 03 20:29:27.115683 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:27.115262    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-d1aa66-openshift-default-8457c96657-lt2kw"
Nov 03 20:29:27.146841 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:27.145926    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dc7b94-667b5756b8-ns2z5"
Nov 03 20:29:27.329966 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:29:27.329876    5570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8bwhz" containerName="extractor"
Nov 03 20:29:27.332188 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:27.332161    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2635dffb-14f1-49cb-8d32-8f539c28ed30-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2635dffb-14f1-49cb-8d32-8f539c28ed30-openshift-service-ca\") pod \"2635dffb-14f1-49cb-8d32-8f539c28ed30\" (UID: \"2635dffb-14f1-49cb-8d32-8f539c28ed30\") "
Nov 03 20:29:27.332353 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:27.332336    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2635dffb-14f1-49cb-8d32-8f539c28ed30-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2635dffb-14f1-49cb-8d32-8f539c28ed30-metrics-certs\") pod \"2635dffb-14f1-49cb-8d32-8f539c28ed30\" (UID: \"2635dffb-14f1-49cb-8d32-8f539c28ed30\") "
Nov 03 20:29:27.332582 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:27.332569    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2635dffb-14f1-49cb-8d32-8f539c28ed30-kube-api-access-mxmdh\" (UniqueName: \"kubernetes.io/projected/2635dffb-14f1-49cb-8d32-8f539c28ed30-kube-api-access-mxmdh\") pod \"2635dffb-14f1-49cb-8d32-8f539c28ed30\" (UID: \"2635dffb-14f1-49cb-8d32-8f539c28ed30\") "
Nov 03 20:29:27.332949 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:27.332932    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2635dffb-14f1-49cb-8d32-8f539c28ed30-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2635dffb-14f1-49cb-8d32-8f539c28ed30-maas-api-tls\") pod \"2635dffb-14f1-49cb-8d32-8f539c28ed30\" (UID: \"2635dffb-14f1-49cb-8d32-8f539c28ed30\") "
Nov 03 20:29:27.335981 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:27.335952    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2635dffb-14f1-49cb-8d32-8f539c28ed30-maas-api-tls" pod "2635dffb-14f1-49cb-8d32-8f539c28ed30" (UID: "2635dffb-14f1-49cb-8d32-8f539c28ed30"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:29:27.336309 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:27.336285    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2635dffb-14f1-49cb-8d32-8f539c28ed30-openshift-service-ca" pod "2635dffb-14f1-49cb-8d32-8f539c28ed30" (UID: "2635dffb-14f1-49cb-8d32-8f539c28ed30"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:29:27.344943 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:27.344911    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2635dffb-14f1-49cb-8d32-8f539c28ed30-metrics-certs" pod "2635dffb-14f1-49cb-8d32-8f539c28ed30" (UID: "2635dffb-14f1-49cb-8d32-8f539c28ed30"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:29:27.346630 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:27.346605    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2635dffb-14f1-49cb-8d32-8f539c28ed30-kube-api-access-mxmdh" pod "2635dffb-14f1-49cb-8d32-8f539c28ed30" (UID: "2635dffb-14f1-49cb-8d32-8f539c28ed30"). InnerVolumeSpecName "kube-api-access-mxmdh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:29:27.453796 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:27.443444    5570 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2635dffb-14f1-49cb-8d32-8f539c28ed30-maas-api-tls\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:29:27.453796 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:27.443482    5570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2635dffb-14f1-49cb-8d32-8f539c28ed30-openshift-service-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:29:27.453796 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:27.443496    5570 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2635dffb-14f1-49cb-8d32-8f539c28ed30-metrics-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:29:27.453796 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:27.443511    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxmdh\" (UniqueName: \"kubernetes.io/projected/2635dffb-14f1-49cb-8d32-8f539c28ed30-kube-api-access-mxmdh\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:29:28.044146 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:28.044111    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dc7b94-667b5756b8-ns2z5"
Nov 03 20:29:28.044146 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:28.044133    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dc7b94-667b5756b8-ns2z5" event={"ID":"2635dffb-14f1-49cb-8d32-8f539c28ed30","Type":"ContainerDied","Data":"a995b94940a51a8f8665ed1cf56377df2d8e2210b8165c602e9a00aaf73e7127"}
Nov 03 20:29:28.044723 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:28.044202    5570 scope.go:122] "RemoveContainer" containerID="e2d6437514e7ae9649085f2ab58a2c8c15874dadd45dda93d9e35c8a43ca8260"
Nov 03 20:29:28.101945 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:28.101908    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-dc7b94-667b5756b8-ns2z5"]
Nov 03 20:29:28.114863 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:28.114832    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-dc7b94-667b5756b8-ns2z5"]
Nov 03 20:29:29.967286 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:29.967248    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2635dffb-14f1-49cb-8d32-8f539c28ed30" path="/var/lib/kubelet/pods/2635dffb-14f1-49cb-8d32-8f539c28ed30/volumes"
Nov 03 20:29:30.301870 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:30.301844    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d1aa66-7b7c6c6455-n8n8v"]
Nov 03 20:29:30.302614 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:30.302600    5570 cpu_manager.go:414] "containerMap: removing container" podUID="2635dffb-14f1-49cb-8d32-8f539c28ed30" containerName="maas-api"
Nov 03 20:29:30.302658 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:30.302616    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2635dffb-14f1-49cb-8d32-8f539c28ed30" containerName="maas-api"
Nov 03 20:29:30.302794 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:30.302765    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="2635dffb-14f1-49cb-8d32-8f539c28ed30" containerName="maas-api"
Nov 03 20:29:30.307924 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:30.307900    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d1aa66-7b7c6c6455-n8n8v"
Nov 03 20:29:30.323930 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:30.323901    5570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-d1aa66\""
Nov 03 20:29:30.324233 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:30.324205    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d1aa66-7b7c6c6455-n8n8v"]
Nov 03 20:29:30.331022 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:30.330996    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-d1aa66-dockercfg-c7h7z\""
Nov 03 20:29:30.518852 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:30.515213    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51d6f999-63b2-4461-a82f-65ff671546f3-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-d1aa66-7b7c6c6455-n8n8v\" (UID: \"51d6f999-63b2-4461-a82f-65ff671546f3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d1aa66-7b7c6c6455-n8n8v"
Nov 03 20:29:30.518852 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:30.515341    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5spxf\" (UniqueName: \"kubernetes.io/projected/51d6f999-63b2-4461-a82f-65ff671546f3-kube-api-access-5spxf\") pod \"payload-pre-processing-e2e-api-b-d1aa66-7b7c6c6455-n8n8v\" (UID: \"51d6f999-63b2-4461-a82f-65ff671546f3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d1aa66-7b7c6c6455-n8n8v"
Nov 03 20:29:30.594916 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:30.593540    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d1aa66-bcd7bb5b6-d9dcs"]
Nov 03 20:29:30.613822 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:30.612034    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d1aa66-bcd7bb5b6-d9dcs"]
Nov 03 20:29:30.613822 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:30.612166    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-d1aa66-bcd7bb5b6-d9dcs"
Nov 03 20:29:30.642798 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:30.638809    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5spxf\" (UniqueName: \"kubernetes.io/projected/51d6f999-63b2-4461-a82f-65ff671546f3-kube-api-access-5spxf\") pod \"payload-pre-processing-e2e-api-b-d1aa66-7b7c6c6455-n8n8v\" (UID: \"51d6f999-63b2-4461-a82f-65ff671546f3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d1aa66-7b7c6c6455-n8n8v"
Nov 03 20:29:30.642798 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:30.639081    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2c2g\" (UniqueName: \"kubernetes.io/projected/1da1c326-d687-48d8-aa52-42392ef0e1d2-kube-api-access-v2c2g\") pod \"payload-processing-e2e-api-b-d1aa66-bcd7bb5b6-d9dcs\" (UID: \"1da1c326-d687-48d8-aa52-42392ef0e1d2\") " pod="openshift-ingress/payload-processing-e2e-api-b-d1aa66-bcd7bb5b6-d9dcs"
Nov 03 20:29:30.642798 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:30.639922    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1da1c326-d687-48d8-aa52-42392ef0e1d2-plugins-config-volume\") pod \"payload-processing-e2e-api-b-d1aa66-bcd7bb5b6-d9dcs\" (UID: \"1da1c326-d687-48d8-aa52-42392ef0e1d2\") " pod="openshift-ingress/payload-processing-e2e-api-b-d1aa66-bcd7bb5b6-d9dcs"
Nov 03 20:29:30.642798 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:30.640017    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51d6f999-63b2-4461-a82f-65ff671546f3-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-d1aa66-7b7c6c6455-n8n8v\" (UID: \"51d6f999-63b2-4461-a82f-65ff671546f3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d1aa66-7b7c6c6455-n8n8v"
Nov 03 20:29:30.642798 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:30.641839    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51d6f999-63b2-4461-a82f-65ff671546f3-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-d1aa66-7b7c6c6455-n8n8v\" (UID: \"51d6f999-63b2-4461-a82f-65ff671546f3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d1aa66-7b7c6c6455-n8n8v"
Nov 03 20:29:30.723807 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:30.716804    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5spxf\" (UniqueName: \"kubernetes.io/projected/51d6f999-63b2-4461-a82f-65ff671546f3-kube-api-access-5spxf\") pod \"payload-pre-processing-e2e-api-b-d1aa66-7b7c6c6455-n8n8v\" (UID: \"51d6f999-63b2-4461-a82f-65ff671546f3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d1aa66-7b7c6c6455-n8n8v"
Nov 03 20:29:30.761799 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:30.750913    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2c2g\" (UniqueName: \"kubernetes.io/projected/1da1c326-d687-48d8-aa52-42392ef0e1d2-kube-api-access-v2c2g\") pod \"payload-processing-e2e-api-b-d1aa66-bcd7bb5b6-d9dcs\" (UID: \"1da1c326-d687-48d8-aa52-42392ef0e1d2\") " pod="openshift-ingress/payload-processing-e2e-api-b-d1aa66-bcd7bb5b6-d9dcs"
Nov 03 20:29:30.761799 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:30.751118    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1da1c326-d687-48d8-aa52-42392ef0e1d2-plugins-config-volume\") pod \"payload-processing-e2e-api-b-d1aa66-bcd7bb5b6-d9dcs\" (UID: \"1da1c326-d687-48d8-aa52-42392ef0e1d2\") " pod="openshift-ingress/payload-processing-e2e-api-b-d1aa66-bcd7bb5b6-d9dcs"
Nov 03 20:29:30.761799 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:30.752211    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1da1c326-d687-48d8-aa52-42392ef0e1d2-plugins-config-volume\") pod \"payload-processing-e2e-api-b-d1aa66-bcd7bb5b6-d9dcs\" (UID: \"1da1c326-d687-48d8-aa52-42392ef0e1d2\") " pod="openshift-ingress/payload-processing-e2e-api-b-d1aa66-bcd7bb5b6-d9dcs"
Nov 03 20:29:30.809254 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:30.786654    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2c2g\" (UniqueName: \"kubernetes.io/projected/1da1c326-d687-48d8-aa52-42392ef0e1d2-kube-api-access-v2c2g\") pod \"payload-processing-e2e-api-b-d1aa66-bcd7bb5b6-d9dcs\" (UID: \"1da1c326-d687-48d8-aa52-42392ef0e1d2\") " pod="openshift-ingress/payload-processing-e2e-api-b-d1aa66-bcd7bb5b6-d9dcs"
Nov 03 20:29:30.926122 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:30.926087    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d1aa66-7b7c6c6455-n8n8v"
Nov 03 20:29:31.019612 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:31.018862    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-d1aa66-bcd7bb5b6-d9dcs"
Nov 03 20:29:31.245364 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:31.245334    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d1aa66-7b7c6c6455-n8n8v"]
Nov 03 20:29:31.248719 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:29:31.247711    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51d6f999_63b2_4461_a82f_65ff671546f3.slice/crio-2ff7dec511122d2725ecce94b248d5501eac506fc207cc0b547ddfb264d69487 WatchSource:0}: Error finding container 2ff7dec511122d2725ecce94b248d5501eac506fc207cc0b547ddfb264d69487: Status 404 returned error can't find the container with id 2ff7dec511122d2725ecce94b248d5501eac506fc207cc0b547ddfb264d69487
Nov 03 20:29:31.418591 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:31.418562    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d1aa66-bcd7bb5b6-d9dcs"]
Nov 03 20:29:31.419870 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:29:31.419839    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1da1c326_d687_48d8_aa52_42392ef0e1d2.slice/crio-870aaf0381fa3bfbf2baa50551ae333d317b66fcf81d992b4b871c519bcdcbb4 WatchSource:0}: Error finding container 870aaf0381fa3bfbf2baa50551ae333d317b66fcf81d992b4b871c519bcdcbb4: Status 404 returned error can't find the container with id 870aaf0381fa3bfbf2baa50551ae333d317b66fcf81d992b4b871c519bcdcbb4
Nov 03 20:29:32.140720 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:32.140684    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-d1aa66-bcd7bb5b6-d9dcs" event={"ID":"1da1c326-d687-48d8-aa52-42392ef0e1d2","Type":"ContainerStarted","Data":"2980b208da095e8ce4d4bb2471ca321f444bf4bde5f56d9fc62e72ab0035cfa8"}
Nov 03 20:29:32.141244 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:32.140748    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-d1aa66-bcd7bb5b6-d9dcs" event={"ID":"1da1c326-d687-48d8-aa52-42392ef0e1d2","Type":"ContainerStarted","Data":"870aaf0381fa3bfbf2baa50551ae333d317b66fcf81d992b4b871c519bcdcbb4"}
Nov 03 20:29:32.141244 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:32.140950    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-d1aa66-bcd7bb5b6-d9dcs"
Nov 03 20:29:32.142695 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:32.142666    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d1aa66-7b7c6c6455-n8n8v" event={"ID":"51d6f999-63b2-4461-a82f-65ff671546f3","Type":"ContainerStarted","Data":"5602baac61cd7ea0e2f1947dda9357aee0ec96356550d84113e2ab3bd9ceda1d"}
Nov 03 20:29:32.142823 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:32.142714    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d1aa66-7b7c6c6455-n8n8v" event={"ID":"51d6f999-63b2-4461-a82f-65ff671546f3","Type":"ContainerStarted","Data":"2ff7dec511122d2725ecce94b248d5501eac506fc207cc0b547ddfb264d69487"}
Nov 03 20:29:32.142823 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:32.142765    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d1aa66-7b7c6c6455-n8n8v"
Nov 03 20:29:32.195082 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:32.195029    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-d1aa66-bcd7bb5b6-d9dcs" podStartSLOduration=2.19501507 podStartE2EDuration="2.19501507s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:29:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:29:32.180569029 +0000 UTC m=+1506.333230935" watchObservedRunningTime="2026-11-03 20:29:32.19501507 +0000 UTC m=+1506.347676971"
Nov 03 20:29:32.223541 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:32.223488    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d1aa66-7b7c6c6455-n8n8v" podStartSLOduration=2.223472875 podStartE2EDuration="2.223472875s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:29:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:29:32.219337003 +0000 UTC m=+1506.371998911" watchObservedRunningTime="2026-11-03 20:29:32.223472875 +0000 UTC m=+1506.376134762"
Nov 03 20:29:32.297685 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:32.297643    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g69f8"]
Nov 03 20:29:33.699810 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:33.697250    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-fab871-5c95886dbd-x49r6"
Nov 03 20:29:33.699810 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:33.697664    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fab871-5c8bd9f8dc-qv2jl"
Nov 03 20:29:35.930931 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:35.930879    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-2-w6-9e1bcb-kserve-7d96d9f6f6-4f7fl" podUID="56353b0f-08ee-4ada-9dc0-ac5afd5a59c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:29:36.919494 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:36.919441    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/distinct-2-w4-e80203-kserve-756f9b9bc6-69gxj" podUID="ccf5dc2f-f116-4917-bb1a-b136d8ed17f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:29:36.935264 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:36.935221    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-2-w6-9e1bcb-kserve-7d96d9f6f6-4f7fl" podUID="56353b0f-08ee-4ada-9dc0-ac5afd5a59c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded"
Nov 03 20:29:36.937117 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:36.937092    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-2-w6-9e1bcb-kserve-7d96d9f6f6-4f7fl"
Nov 03 20:29:37.275275 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:37.218310    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/premium-w2-9973d9-kserve-5c6dd4669c-5gv8x" podUID="cefc89ad-7ae1-433b-979d-6601678a6d98" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Nov 03 20:29:37.275275 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:37.218631    5570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-2-w6-9e1bcb-kserve-7d96d9f6f6-4f7fl" podUID="56353b0f-08ee-4ada-9dc0-ac5afd5a59c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": context deadline exceeded"
Nov 03 20:29:37.343832 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:37.343658    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/distinct-2-w0-e40816-kserve-79fb46654d-s242x" podUID="4ad9b048-925c-4d01-a839-a5f25d4dd02d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:29:37.399832 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:37.369162    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh" podUID="2d3647f2-89e9-4908-a8ee-d84e527abeb0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Nov 03 20:29:37.399832 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:37.372733    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/simulator-w0-e40816-kserve-688f79796f-2tvw5" podUID="c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:29:37.404172 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:37.401661    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/premium-w4-e80203-kserve-5f9f445456-bms9d" podUID="6ad76efc-20c2-4951-b22a-c5a39cd39ed4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Nov 03 20:29:37.587857 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:37.571616    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/distinct-2-w3-482722-kserve-74bf58cf7d-v9cjw" podUID="9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:29:37.614803 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:37.593217    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/premium-w3-482722-kserve-689dc7bd5d-x8rjp" podUID="369045c3-4d6a-49a7-9bc6-f2939883bee8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:29:37.697805 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:37.689619    5570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-9e1bcb/premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh" podUID="2d3647f2-89e9-4908-a8ee-d84e527abeb0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:29:37.765906 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:37.764870    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz" podUID="3a715ef1-38e9-4386-9eec-e10e8df11a10" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": context deadline exceeded"
Nov 03 20:29:37.852118 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:37.852006    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/distinct-w3-482722-kserve-d876967c-kd488" podUID="9a97f9cc-2261-471d-9053-51ee965850bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:29:37.923251 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:37.920897    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/distinct-2-w4-e80203-kserve-756f9b9bc6-69gxj" podUID="ccf5dc2f-f116-4917-bb1a-b136d8ed17f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:29:37.923251 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:37.921524    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-e80203/distinct-2-w4-e80203-kserve-756f9b9bc6-69gxj"
Nov 03 20:29:37.937394 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:37.936854    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-2-w6-9e1bcb-kserve-7d96d9f6f6-4f7fl" podUID="56353b0f-08ee-4ada-9dc0-ac5afd5a59c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:29:38.234868 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:38.226421    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/premium-w2-9973d9-kserve-5c6dd4669c-5gv8x" podUID="cefc89ad-7ae1-433b-979d-6601678a6d98" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:29:38.234868 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:38.233521    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9973d9/premium-w2-9973d9-kserve-5c6dd4669c-5gv8x"
Nov 03 20:29:38.465924 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:38.445896    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/distinct-2-w0-e40816-kserve-79fb46654d-s242x" podUID="4ad9b048-925c-4d01-a839-a5f25d4dd02d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded"
Nov 03 20:29:38.465924 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:38.446309    5570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-e40816/simulator-w0-e40816-kserve-688f79796f-2tvw5" podUID="c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:29:38.465924 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:38.446382    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh" podUID="2d3647f2-89e9-4908-a8ee-d84e527abeb0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Nov 03 20:29:38.467897 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:38.467855    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/premium-w4-e80203-kserve-5f9f445456-bms9d" podUID="6ad76efc-20c2-4951-b22a-c5a39cd39ed4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Nov 03 20:29:38.511890 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:38.467910    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/simulator-w0-e40816-kserve-688f79796f-2tvw5" podUID="c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Nov 03 20:29:38.511890 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:38.468516    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e40816/distinct-2-w0-e40816-kserve-79fb46654d-s242x"
Nov 03 20:29:38.511890 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:38.508036    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-9e1bcb/premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh"
Nov 03 20:29:38.511890 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:38.508099    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-e80203/premium-w4-e80203-kserve-5f9f445456-bms9d"
Nov 03 20:29:38.511890 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:38.508154    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e40816/simulator-w0-e40816-kserve-688f79796f-2tvw5"
Nov 03 20:29:38.986901 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:38.864058    5570 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-657vf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:29:38.986901 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:38.864139    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-657vf" podUID="f089dd20-d6a6-4c63-88a5-3fbe9a2ff315" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:29:39.038803 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:39.024817    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/distinct-w3-482722-kserve-d876967c-kd488" podUID="9a97f9cc-2261-471d-9053-51ee965850bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded"
Nov 03 20:29:39.136557 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:39.048396    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-482722/distinct-w3-482722-kserve-d876967c-kd488"
Nov 03 20:29:39.211490 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:39.051149    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-2-w6-9e1bcb-kserve-7d96d9f6f6-4f7fl" podUID="56353b0f-08ee-4ada-9dc0-ac5afd5a59c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:29:39.211490 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:39.099331    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/premium-w3-482722-kserve-689dc7bd5d-x8rjp" podUID="369045c3-4d6a-49a7-9bc6-f2939883bee8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Nov 03 20:29:39.211490 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:39.100755    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz" podUID="3a715ef1-38e9-4386-9eec-e10e8df11a10" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:29:39.211490 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:39.100822    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/distinct-2-w4-e80203-kserve-756f9b9bc6-69gxj" podUID="ccf5dc2f-f116-4917-bb1a-b136d8ed17f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded"
Nov 03 20:29:39.211490 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:39.163483    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/distinct-2-w3-482722-kserve-74bf58cf7d-v9cjw" podUID="9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded"
Nov 03 20:29:39.405860 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:39.373361    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-482722/premium-w3-482722-kserve-689dc7bd5d-x8rjp"
Nov 03 20:29:39.457802 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:39.448940    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh" podUID="2d3647f2-89e9-4908-a8ee-d84e527abeb0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:29:39.457802 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:39.449107    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/distinct-2-w0-e40816-kserve-79fb46654d-s242x" podUID="4ad9b048-925c-4d01-a839-a5f25d4dd02d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded"
Nov 03 20:29:39.488796 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:39.462069    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/premium-w2-9973d9-kserve-5c6dd4669c-5gv8x" podUID="cefc89ad-7ae1-433b-979d-6601678a6d98" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Nov 03 20:29:39.528239 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:39.507923    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/simulator-w0-e40816-kserve-688f79796f-2tvw5" podUID="c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:29:39.528239 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:39.508062    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/premium-w4-e80203-kserve-5f9f445456-bms9d" podUID="6ad76efc-20c2-4951-b22a-c5a39cd39ed4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:29:40.035274 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:40.035234    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz"
Nov 03 20:29:40.060119 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:40.039561    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-482722/distinct-2-w3-482722-kserve-74bf58cf7d-v9cjw"
Nov 03 20:29:40.153082 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:40.099912    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/distinct-w3-482722-kserve-d876967c-kd488" podUID="9a97f9cc-2261-471d-9053-51ee965850bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:29:40.270182 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:40.270094    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz" podUID="3a715ef1-38e9-4386-9eec-e10e8df11a10" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:29:40.290706 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:40.270448    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-2-w6-9e1bcb-kserve-7d96d9f6f6-4f7fl" podUID="56353b0f-08ee-4ada-9dc0-ac5afd5a59c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded"
Nov 03 20:29:40.310360 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:40.270482    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/distinct-2-w4-e80203-kserve-756f9b9bc6-69gxj" podUID="ccf5dc2f-f116-4917-bb1a-b136d8ed17f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded"
Nov 03 20:29:40.351515 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:40.270508    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/distinct-2-w3-482722-kserve-74bf58cf7d-v9cjw" podUID="9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded"
Nov 03 20:29:40.380006 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:40.271238    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/premium-w3-482722-kserve-689dc7bd5d-x8rjp" podUID="369045c3-4d6a-49a7-9bc6-f2939883bee8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:29:40.805125 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:40.805090    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-e80203/distinct-2-w4-e80203-kserve-756f9b9bc6-69gxj"
Nov 03 20:29:40.909729 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:40.909693    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9973d9/premium-w2-9973d9-kserve-5c6dd4669c-5gv8x"
Nov 03 20:29:40.923898 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:40.923870    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-482722/distinct-w3-482722-kserve-d876967c-kd488"
Nov 03 20:29:40.989813 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:40.974342    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e40816/distinct-2-w0-e40816-kserve-79fb46654d-s242x"
Nov 03 20:29:41.044236 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:41.044205    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e40816/simulator-w0-e40816-kserve-688f79796f-2tvw5"
Nov 03 20:29:41.048722 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:41.048690    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/premium-w4-e80203-kserve-5f9f445456-bms9d" podUID="6ad76efc-20c2-4951-b22a-c5a39cd39ed4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:29:41.048963 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:41.048932    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh" podUID="2d3647f2-89e9-4908-a8ee-d84e527abeb0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:29:41.110153 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:41.110064    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-e80203/premium-w4-e80203-kserve-5f9f445456-bms9d"
Nov 03 20:29:41.122797 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:41.119392    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-482722/premium-w3-482722-kserve-689dc7bd5d-x8rjp"
Nov 03 20:29:41.182058 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:41.182020    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-2-w6-9e1bcb-kserve-7d96d9f6f6-4f7fl"
Nov 03 20:29:41.207110 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:41.204318    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz"
Nov 03 20:29:41.207110 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:41.204385    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-9e1bcb/premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh"
Nov 03 20:29:41.211077 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:41.211046    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-482722/distinct-2-w3-482722-kserve-74bf58cf7d-v9cjw"
Nov 03 20:29:43.283793 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:43.278063    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d1aa66-7b7c6c6455-n8n8v"
Nov 03 20:29:43.284556 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:43.284530    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-d1aa66-bcd7bb5b6-d9dcs"
Nov 03 20:29:46.084889 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:46.084844    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-4cdbeb-77575c99cb-7pz9r"]
Nov 03 20:29:46.118675 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:46.118634    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4cdbeb-7c468dc747-9kqqx"]
Nov 03 20:29:46.612180 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:46.612139    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-2zvr4"]
Nov 03 20:29:46.612485 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:46.612390    5570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-2zvr4" podUID="c8edff27-9b0c-40e5-9a72-4a85d7efd3e9" containerName="istio-proxy" containerID="cri-o://511ac9d6b28ec82c5a67a5a17fb5cfd8ca9042b566e6369939735547e1f06b1f" gracePeriod=30
Nov 03 20:29:48.140927 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:48.140890    5570 patch_prober.go:29] interesting pod/e2e-api-a-addf1a-openshift-default-69ffb7968c-pgb22 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:29:48.141419 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:48.140955    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-addf1a-openshift-default-69ffb7968c-pgb22" podUID="e5d67e4b-c834-426f-8df5-8d6dc4ae77ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:29:48.844959 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:48.844918    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-9pg4l"]
Nov 03 20:29:48.865036 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:48.865008    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-9pg4l"
Nov 03 20:29:48.871687 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:48.871657    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-9pg4l"]
Nov 03 20:29:49.003141 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:49.003111    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/623f50bf-e25b-4647-9ffe-db4d163b2284-credential-socket\") pod \"e2e-api-a-c4a71d-openshift-default-7647f5cdbf-9pg4l\" (UID: \"623f50bf-e25b-4647-9ffe-db4d163b2284\") " pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-9pg4l"
Nov 03 20:29:49.003291 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:49.003151    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fv28w\" (UniqueName: \"kubernetes.io/projected/623f50bf-e25b-4647-9ffe-db4d163b2284-kube-api-access-fv28w\") pod \"e2e-api-a-c4a71d-openshift-default-7647f5cdbf-9pg4l\" (UID: \"623f50bf-e25b-4647-9ffe-db4d163b2284\") " pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-9pg4l"
Nov 03 20:29:49.003291 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:49.003246    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/623f50bf-e25b-4647-9ffe-db4d163b2284-workload-socket\") pod \"e2e-api-a-c4a71d-openshift-default-7647f5cdbf-9pg4l\" (UID: \"623f50bf-e25b-4647-9ffe-db4d163b2284\") " pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-9pg4l"
Nov 03 20:29:49.003401 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:49.003310    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/623f50bf-e25b-4647-9ffe-db4d163b2284-istio-token\") pod \"e2e-api-a-c4a71d-openshift-default-7647f5cdbf-9pg4l\" (UID: \"623f50bf-e25b-4647-9ffe-db4d163b2284\") " pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-9pg4l"
Nov 03 20:29:49.003401 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:49.003332    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/623f50bf-e25b-4647-9ffe-db4d163b2284-istio-podinfo\") pod \"e2e-api-a-c4a71d-openshift-default-7647f5cdbf-9pg4l\" (UID: \"623f50bf-e25b-4647-9ffe-db4d163b2284\") " pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-9pg4l"
Nov 03 20:29:49.003481 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:49.003462    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/623f50bf-e25b-4647-9ffe-db4d163b2284-istio-data\") pod \"e2e-api-a-c4a71d-openshift-default-7647f5cdbf-9pg4l\" (UID: \"623f50bf-e25b-4647-9ffe-db4d163b2284\") " pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-9pg4l"
Nov 03 20:29:49.003565 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:49.003545    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/623f50bf-e25b-4647-9ffe-db4d163b2284-istiod-ca-cert\") pod \"e2e-api-a-c4a71d-openshift-default-7647f5cdbf-9pg4l\" (UID: \"623f50bf-e25b-4647-9ffe-db4d163b2284\") " pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-9pg4l"
Nov 03 20:29:49.003659 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:49.003644    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/623f50bf-e25b-4647-9ffe-db4d163b2284-workload-certs\") pod \"e2e-api-a-c4a71d-openshift-default-7647f5cdbf-9pg4l\" (UID: \"623f50bf-e25b-4647-9ffe-db4d163b2284\") " pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-9pg4l"
Nov 03 20:29:49.003722 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:49.003707    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/623f50bf-e25b-4647-9ffe-db4d163b2284-istio-envoy\") pod \"e2e-api-a-c4a71d-openshift-default-7647f5cdbf-9pg4l\" (UID: \"623f50bf-e25b-4647-9ffe-db4d163b2284\") " pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-9pg4l"
Nov 03 20:29:49.106061 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:49.105985    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/623f50bf-e25b-4647-9ffe-db4d163b2284-istio-token\") pod \"e2e-api-a-c4a71d-openshift-default-7647f5cdbf-9pg4l\" (UID: \"623f50bf-e25b-4647-9ffe-db4d163b2284\") " pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-9pg4l"
Nov 03 20:29:49.106061 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:49.106024    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/623f50bf-e25b-4647-9ffe-db4d163b2284-istio-podinfo\") pod \"e2e-api-a-c4a71d-openshift-default-7647f5cdbf-9pg4l\" (UID: \"623f50bf-e25b-4647-9ffe-db4d163b2284\") " pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-9pg4l"
Nov 03 20:29:49.106305 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:49.106077    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/623f50bf-e25b-4647-9ffe-db4d163b2284-istio-data\") pod \"e2e-api-a-c4a71d-openshift-default-7647f5cdbf-9pg4l\" (UID: \"623f50bf-e25b-4647-9ffe-db4d163b2284\") " pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-9pg4l"
Nov 03 20:29:49.106305 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:49.106150    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/623f50bf-e25b-4647-9ffe-db4d163b2284-istiod-ca-cert\") pod \"e2e-api-a-c4a71d-openshift-default-7647f5cdbf-9pg4l\" (UID: \"623f50bf-e25b-4647-9ffe-db4d163b2284\") " pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-9pg4l"
Nov 03 20:29:49.106305 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:49.106239    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/623f50bf-e25b-4647-9ffe-db4d163b2284-workload-certs\") pod \"e2e-api-a-c4a71d-openshift-default-7647f5cdbf-9pg4l\" (UID: \"623f50bf-e25b-4647-9ffe-db4d163b2284\") " pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-9pg4l"
Nov 03 20:29:49.106305 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:49.106284    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/623f50bf-e25b-4647-9ffe-db4d163b2284-istio-envoy\") pod \"e2e-api-a-c4a71d-openshift-default-7647f5cdbf-9pg4l\" (UID: \"623f50bf-e25b-4647-9ffe-db4d163b2284\") " pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-9pg4l"
Nov 03 20:29:49.106609 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:49.106571    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/623f50bf-e25b-4647-9ffe-db4d163b2284-istio-data\") pod \"e2e-api-a-c4a71d-openshift-default-7647f5cdbf-9pg4l\" (UID: \"623f50bf-e25b-4647-9ffe-db4d163b2284\") " pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-9pg4l"
Nov 03 20:29:49.106609 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:49.106584    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/623f50bf-e25b-4647-9ffe-db4d163b2284-credential-socket\") pod \"e2e-api-a-c4a71d-openshift-default-7647f5cdbf-9pg4l\" (UID: \"623f50bf-e25b-4647-9ffe-db4d163b2284\") " pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-9pg4l"
Nov 03 20:29:49.106738 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:49.106641    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fv28w\" (UniqueName: \"kubernetes.io/projected/623f50bf-e25b-4647-9ffe-db4d163b2284-kube-api-access-fv28w\") pod \"e2e-api-a-c4a71d-openshift-default-7647f5cdbf-9pg4l\" (UID: \"623f50bf-e25b-4647-9ffe-db4d163b2284\") " pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-9pg4l"
Nov 03 20:29:49.106738 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:49.106668    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/623f50bf-e25b-4647-9ffe-db4d163b2284-workload-certs\") pod \"e2e-api-a-c4a71d-openshift-default-7647f5cdbf-9pg4l\" (UID: \"623f50bf-e25b-4647-9ffe-db4d163b2284\") " pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-9pg4l"
Nov 03 20:29:49.106901 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:49.106749    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/623f50bf-e25b-4647-9ffe-db4d163b2284-workload-socket\") pod \"e2e-api-a-c4a71d-openshift-default-7647f5cdbf-9pg4l\" (UID: \"623f50bf-e25b-4647-9ffe-db4d163b2284\") " pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-9pg4l"
Nov 03 20:29:49.106901 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:49.106864    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/623f50bf-e25b-4647-9ffe-db4d163b2284-credential-socket\") pod \"e2e-api-a-c4a71d-openshift-default-7647f5cdbf-9pg4l\" (UID: \"623f50bf-e25b-4647-9ffe-db4d163b2284\") " pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-9pg4l"
Nov 03 20:29:49.107182 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:49.107157    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/623f50bf-e25b-4647-9ffe-db4d163b2284-workload-socket\") pod \"e2e-api-a-c4a71d-openshift-default-7647f5cdbf-9pg4l\" (UID: \"623f50bf-e25b-4647-9ffe-db4d163b2284\") " pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-9pg4l"
Nov 03 20:29:49.107305 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:49.107241    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/623f50bf-e25b-4647-9ffe-db4d163b2284-istiod-ca-cert\") pod \"e2e-api-a-c4a71d-openshift-default-7647f5cdbf-9pg4l\" (UID: \"623f50bf-e25b-4647-9ffe-db4d163b2284\") " pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-9pg4l"
Nov 03 20:29:49.108656 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:49.108627    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/623f50bf-e25b-4647-9ffe-db4d163b2284-istio-envoy\") pod \"e2e-api-a-c4a71d-openshift-default-7647f5cdbf-9pg4l\" (UID: \"623f50bf-e25b-4647-9ffe-db4d163b2284\") " pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-9pg4l"
Nov 03 20:29:49.108977 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:49.108960    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/623f50bf-e25b-4647-9ffe-db4d163b2284-istio-podinfo\") pod \"e2e-api-a-c4a71d-openshift-default-7647f5cdbf-9pg4l\" (UID: \"623f50bf-e25b-4647-9ffe-db4d163b2284\") " pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-9pg4l"
Nov 03 20:29:49.127637 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:49.127611    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/623f50bf-e25b-4647-9ffe-db4d163b2284-istio-token\") pod \"e2e-api-a-c4a71d-openshift-default-7647f5cdbf-9pg4l\" (UID: \"623f50bf-e25b-4647-9ffe-db4d163b2284\") " pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-9pg4l"
Nov 03 20:29:49.140252 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:49.140228    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fv28w\" (UniqueName: \"kubernetes.io/projected/623f50bf-e25b-4647-9ffe-db4d163b2284-kube-api-access-fv28w\") pod \"e2e-api-a-c4a71d-openshift-default-7647f5cdbf-9pg4l\" (UID: \"623f50bf-e25b-4647-9ffe-db4d163b2284\") " pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-9pg4l"
Nov 03 20:29:49.185702 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:49.180939    5570 patch_prober.go:29] interesting pod/e2e-worker-w1-6fc748-openshift-default-665546959f-2zvr4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 20:29:49.185702 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:49.181000    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-2zvr4" podUID="c8edff27-9b0c-40e5-9a72-4a85d7efd3e9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 20:29:49.230685 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:49.229129    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-9pg4l"
Nov 03 20:29:49.314932 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:49.314898    5570 generic.go:356] "Generic (PLEG): container finished" podID="ca63212a-77c0-4b97-8a23-5d0a518a55ef" containerID="a4842be231f4b13a0568dacd707e44fd989a4a1f6d171db686a6aac8a246eb0f" exitCode=0
Nov 03 20:29:49.315066 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:49.315047    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-bx2dr" event={"ID":"ca63212a-77c0-4b97-8a23-5d0a518a55ef","Type":"ContainerDied","Data":"a4842be231f4b13a0568dacd707e44fd989a4a1f6d171db686a6aac8a246eb0f"}
Nov 03 20:29:49.315675 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:49.315652    5570 scope.go:122] "RemoveContainer" containerID="a4842be231f4b13a0568dacd707e44fd989a4a1f6d171db686a6aac8a246eb0f"
Nov 03 20:29:49.317986 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:49.317949    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:29:49.318179 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:49.318156    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:29:49.318265 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:49.318214    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:29:49.712005 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:49.711971    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-9pg4l"]
Nov 03 20:29:49.718578 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:49.718537    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:29:49.718677 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:49.718627    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:29:49.718677 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:49.718670    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:29:50.291016 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:50.290927    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-7e4389-openshift-default-5b5bd7f55-kpk7n"]
Nov 03 20:29:50.295071 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:50.294992    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-7e4389-openshift-default-5b5bd7f55-kpk7n"
Nov 03 20:29:50.324298 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:50.324261    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-7e4389-openshift-default-dockercfg-gdbqq\""
Nov 03 20:29:50.331735 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:50.331676    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-7e4389-openshift-default-5b5bd7f55-kpk7n"]
Nov 03 20:29:50.376808 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:50.376753    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-9pg4l" event={"ID":"623f50bf-e25b-4647-9ffe-db4d163b2284","Type":"ContainerStarted","Data":"2a450d72048e8485b746d6c3a806cdc70b988882550514fbc8566ee1f05860a2"}
Nov 03 20:29:50.377050 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:50.376834    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-9pg4l" event={"ID":"623f50bf-e25b-4647-9ffe-db4d163b2284","Type":"ContainerStarted","Data":"262b79c83c2bf9bcaac123740b14d31ea69c65209743767d859ca15cbcbc0f80"}
Nov 03 20:29:50.379006 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:50.378970    5570 generic.go:356] "Generic (PLEG): container finished" podID="e5d67e4b-c834-426f-8df5-8d6dc4ae77ab" containerID="5ede59604e62a22821fb828b83994fcbd9b4102722a967137e0cf46fa579cc50" exitCode=0
Nov 03 20:29:50.379176 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:50.379051    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-addf1a-openshift-default-69ffb7968c-pgb22" event={"ID":"e5d67e4b-c834-426f-8df5-8d6dc4ae77ab","Type":"ContainerDied","Data":"5ede59604e62a22821fb828b83994fcbd9b4102722a967137e0cf46fa579cc50"}
Nov 03 20:29:50.379576 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:50.379548    5570 scope.go:122] "RemoveContainer" containerID="5ede59604e62a22821fb828b83994fcbd9b4102722a967137e0cf46fa579cc50"
Nov 03 20:29:50.381638 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:50.381615    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-bx2dr" event={"ID":"ca63212a-77c0-4b97-8a23-5d0a518a55ef","Type":"ContainerStarted","Data":"2f27b981b1b00f2a6830b65b62ed19e37456b99a6f874e45cef4ab2c310ef747"}
Nov 03 20:29:50.381912 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:50.381883    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:29:50.381987 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:50.381958    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:29:50.382039 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:50.381990    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:29:50.387516 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:50.386681    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f5aeb82-01f6-483f-8273-10bd0714e03c-istiod-ca-cert\") pod \"e2e-api-b-7e4389-openshift-default-5b5bd7f55-kpk7n\" (UID: \"1f5aeb82-01f6-483f-8273-10bd0714e03c\") " pod="openshift-ingress/e2e-api-b-7e4389-openshift-default-5b5bd7f55-kpk7n"
Nov 03 20:29:50.387516 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:50.386755    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f5aeb82-01f6-483f-8273-10bd0714e03c-workload-socket\") pod \"e2e-api-b-7e4389-openshift-default-5b5bd7f55-kpk7n\" (UID: \"1f5aeb82-01f6-483f-8273-10bd0714e03c\") " pod="openshift-ingress/e2e-api-b-7e4389-openshift-default-5b5bd7f55-kpk7n"
Nov 03 20:29:50.387516 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:50.386828    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f5aeb82-01f6-483f-8273-10bd0714e03c-istio-envoy\") pod \"e2e-api-b-7e4389-openshift-default-5b5bd7f55-kpk7n\" (UID: \"1f5aeb82-01f6-483f-8273-10bd0714e03c\") " pod="openshift-ingress/e2e-api-b-7e4389-openshift-default-5b5bd7f55-kpk7n"
Nov 03 20:29:50.387516 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:50.386859    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f5aeb82-01f6-483f-8273-10bd0714e03c-workload-certs\") pod \"e2e-api-b-7e4389-openshift-default-5b5bd7f55-kpk7n\" (UID: \"1f5aeb82-01f6-483f-8273-10bd0714e03c\") " pod="openshift-ingress/e2e-api-b-7e4389-openshift-default-5b5bd7f55-kpk7n"
Nov 03 20:29:50.387516 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:50.387051    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9sw9\" (UniqueName: \"kubernetes.io/projected/1f5aeb82-01f6-483f-8273-10bd0714e03c-kube-api-access-t9sw9\") pod \"e2e-api-b-7e4389-openshift-default-5b5bd7f55-kpk7n\" (UID: \"1f5aeb82-01f6-483f-8273-10bd0714e03c\") " pod="openshift-ingress/e2e-api-b-7e4389-openshift-default-5b5bd7f55-kpk7n"
Nov 03 20:29:50.387516 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:50.387153    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f5aeb82-01f6-483f-8273-10bd0714e03c-istio-podinfo\") pod \"e2e-api-b-7e4389-openshift-default-5b5bd7f55-kpk7n\" (UID: \"1f5aeb82-01f6-483f-8273-10bd0714e03c\") " pod="openshift-ingress/e2e-api-b-7e4389-openshift-default-5b5bd7f55-kpk7n"
Nov 03 20:29:50.387516 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:50.387190    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f5aeb82-01f6-483f-8273-10bd0714e03c-credential-socket\") pod \"e2e-api-b-7e4389-openshift-default-5b5bd7f55-kpk7n\" (UID: \"1f5aeb82-01f6-483f-8273-10bd0714e03c\") " pod="openshift-ingress/e2e-api-b-7e4389-openshift-default-5b5bd7f55-kpk7n"
Nov 03 20:29:50.387516 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:50.387213    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f5aeb82-01f6-483f-8273-10bd0714e03c-istio-token\") pod \"e2e-api-b-7e4389-openshift-default-5b5bd7f55-kpk7n\" (UID: \"1f5aeb82-01f6-483f-8273-10bd0714e03c\") " pod="openshift-ingress/e2e-api-b-7e4389-openshift-default-5b5bd7f55-kpk7n"
Nov 03 20:29:50.387516 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:50.387236    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f5aeb82-01f6-483f-8273-10bd0714e03c-istio-data\") pod \"e2e-api-b-7e4389-openshift-default-5b5bd7f55-kpk7n\" (UID: \"1f5aeb82-01f6-483f-8273-10bd0714e03c\") " pod="openshift-ingress/e2e-api-b-7e4389-openshift-default-5b5bd7f55-kpk7n"
Nov 03 20:29:50.426922 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:50.426870    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-9pg4l" podStartSLOduration=2.4268549139999998 podStartE2EDuration="2.426854914s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:29:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:29:50.425092086 +0000 UTC m=+1524.577753976" watchObservedRunningTime="2026-11-03 20:29:50.426854914 +0000 UTC m=+1524.579516801"
Nov 03 20:29:50.564887 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:50.537910    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f5aeb82-01f6-483f-8273-10bd0714e03c-workload-socket\") pod \"e2e-api-b-7e4389-openshift-default-5b5bd7f55-kpk7n\" (UID: \"1f5aeb82-01f6-483f-8273-10bd0714e03c\") " pod="openshift-ingress/e2e-api-b-7e4389-openshift-default-5b5bd7f55-kpk7n"
Nov 03 20:29:50.565669 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:50.565638    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f5aeb82-01f6-483f-8273-10bd0714e03c-istio-envoy\") pod \"e2e-api-b-7e4389-openshift-default-5b5bd7f55-kpk7n\" (UID: \"1f5aeb82-01f6-483f-8273-10bd0714e03c\") " pod="openshift-ingress/e2e-api-b-7e4389-openshift-default-5b5bd7f55-kpk7n"
Nov 03 20:29:50.566008 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:50.565964    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f5aeb82-01f6-483f-8273-10bd0714e03c-workload-certs\") pod \"e2e-api-b-7e4389-openshift-default-5b5bd7f55-kpk7n\" (UID: \"1f5aeb82-01f6-483f-8273-10bd0714e03c\") " pod="openshift-ingress/e2e-api-b-7e4389-openshift-default-5b5bd7f55-kpk7n"
Nov 03 20:29:50.566865 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:50.566820    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9sw9\" (UniqueName: \"kubernetes.io/projected/1f5aeb82-01f6-483f-8273-10bd0714e03c-kube-api-access-t9sw9\") pod \"e2e-api-b-7e4389-openshift-default-5b5bd7f55-kpk7n\" (UID: \"1f5aeb82-01f6-483f-8273-10bd0714e03c\") " pod="openshift-ingress/e2e-api-b-7e4389-openshift-default-5b5bd7f55-kpk7n"
Nov 03 20:29:50.567086 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:50.567059    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f5aeb82-01f6-483f-8273-10bd0714e03c-istio-podinfo\") pod \"e2e-api-b-7e4389-openshift-default-5b5bd7f55-kpk7n\" (UID: \"1f5aeb82-01f6-483f-8273-10bd0714e03c\") " pod="openshift-ingress/e2e-api-b-7e4389-openshift-default-5b5bd7f55-kpk7n"
Nov 03 20:29:50.567353 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:50.567319    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f5aeb82-01f6-483f-8273-10bd0714e03c-credential-socket\") pod \"e2e-api-b-7e4389-openshift-default-5b5bd7f55-kpk7n\" (UID: \"1f5aeb82-01f6-483f-8273-10bd0714e03c\") " pod="openshift-ingress/e2e-api-b-7e4389-openshift-default-5b5bd7f55-kpk7n"
Nov 03 20:29:50.619799 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:50.611030    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f5aeb82-01f6-483f-8273-10bd0714e03c-istio-token\") pod \"e2e-api-b-7e4389-openshift-default-5b5bd7f55-kpk7n\" (UID: \"1f5aeb82-01f6-483f-8273-10bd0714e03c\") " pod="openshift-ingress/e2e-api-b-7e4389-openshift-default-5b5bd7f55-kpk7n"
Nov 03 20:29:50.619799 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:50.611123    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f5aeb82-01f6-483f-8273-10bd0714e03c-istio-data\") pod \"e2e-api-b-7e4389-openshift-default-5b5bd7f55-kpk7n\" (UID: \"1f5aeb82-01f6-483f-8273-10bd0714e03c\") " pod="openshift-ingress/e2e-api-b-7e4389-openshift-default-5b5bd7f55-kpk7n"
Nov 03 20:29:50.635573 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:50.544455    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f5aeb82-01f6-483f-8273-10bd0714e03c-workload-socket\") pod \"e2e-api-b-7e4389-openshift-default-5b5bd7f55-kpk7n\" (UID: \"1f5aeb82-01f6-483f-8273-10bd0714e03c\") " pod="openshift-ingress/e2e-api-b-7e4389-openshift-default-5b5bd7f55-kpk7n"
Nov 03 20:29:50.655346 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:50.654669    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f5aeb82-01f6-483f-8273-10bd0714e03c-credential-socket\") pod \"e2e-api-b-7e4389-openshift-default-5b5bd7f55-kpk7n\" (UID: \"1f5aeb82-01f6-483f-8273-10bd0714e03c\") " pod="openshift-ingress/e2e-api-b-7e4389-openshift-default-5b5bd7f55-kpk7n"
Nov 03 20:29:50.655346 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:50.654970    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f5aeb82-01f6-483f-8273-10bd0714e03c-istio-data\") pod \"e2e-api-b-7e4389-openshift-default-5b5bd7f55-kpk7n\" (UID: \"1f5aeb82-01f6-483f-8273-10bd0714e03c\") " pod="openshift-ingress/e2e-api-b-7e4389-openshift-default-5b5bd7f55-kpk7n"
Nov 03 20:29:50.661277 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:50.661149    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f5aeb82-01f6-483f-8273-10bd0714e03c-istiod-ca-cert\") pod \"e2e-api-b-7e4389-openshift-default-5b5bd7f55-kpk7n\" (UID: \"1f5aeb82-01f6-483f-8273-10bd0714e03c\") " pod="openshift-ingress/e2e-api-b-7e4389-openshift-default-5b5bd7f55-kpk7n"
Nov 03 20:29:50.673020 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:50.672873    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f5aeb82-01f6-483f-8273-10bd0714e03c-workload-certs\") pod \"e2e-api-b-7e4389-openshift-default-5b5bd7f55-kpk7n\" (UID: \"1f5aeb82-01f6-483f-8273-10bd0714e03c\") " pod="openshift-ingress/e2e-api-b-7e4389-openshift-default-5b5bd7f55-kpk7n"
Nov 03 20:29:50.680449 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:50.680411    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f5aeb82-01f6-483f-8273-10bd0714e03c-istiod-ca-cert\") pod \"e2e-api-b-7e4389-openshift-default-5b5bd7f55-kpk7n\" (UID: \"1f5aeb82-01f6-483f-8273-10bd0714e03c\") " pod="openshift-ingress/e2e-api-b-7e4389-openshift-default-5b5bd7f55-kpk7n"
Nov 03 20:29:50.920142 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:50.858199    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f5aeb82-01f6-483f-8273-10bd0714e03c-istio-podinfo\") pod \"e2e-api-b-7e4389-openshift-default-5b5bd7f55-kpk7n\" (UID: \"1f5aeb82-01f6-483f-8273-10bd0714e03c\") " pod="openshift-ingress/e2e-api-b-7e4389-openshift-default-5b5bd7f55-kpk7n"
Nov 03 20:29:50.921029 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:50.920995    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f5aeb82-01f6-483f-8273-10bd0714e03c-istio-envoy\") pod \"e2e-api-b-7e4389-openshift-default-5b5bd7f55-kpk7n\" (UID: \"1f5aeb82-01f6-483f-8273-10bd0714e03c\") " pod="openshift-ingress/e2e-api-b-7e4389-openshift-default-5b5bd7f55-kpk7n"
Nov 03 20:29:51.016525 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:51.015191    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f5aeb82-01f6-483f-8273-10bd0714e03c-istio-token\") pod \"e2e-api-b-7e4389-openshift-default-5b5bd7f55-kpk7n\" (UID: \"1f5aeb82-01f6-483f-8273-10bd0714e03c\") " pod="openshift-ingress/e2e-api-b-7e4389-openshift-default-5b5bd7f55-kpk7n"
Nov 03 20:29:51.016525 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:51.015978    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9sw9\" (UniqueName: \"kubernetes.io/projected/1f5aeb82-01f6-483f-8273-10bd0714e03c-kube-api-access-t9sw9\") pod \"e2e-api-b-7e4389-openshift-default-5b5bd7f55-kpk7n\" (UID: \"1f5aeb82-01f6-483f-8273-10bd0714e03c\") " pod="openshift-ingress/e2e-api-b-7e4389-openshift-default-5b5bd7f55-kpk7n"
Nov 03 20:29:51.088670 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:51.086887    5570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-4cdbeb-77575c99cb-7pz9r" podUID="3716281c-9234-41d2-96c3-dbc38c692bbb" containerName="payload-processing" containerID="cri-o://4be17df6901c7c6d2804455f4b35dab727aa9a50b40068440dca691f0ba75af3" gracePeriod=25
Nov 03 20:29:51.119719 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:51.119678    5570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4cdbeb-7c468dc747-9kqqx" podUID="880ebe10-49c7-4960-9fb2-2c699047ebb6" containerName="payload-pre-processing" containerID="cri-o://5f27f224d717a2b67dd1594ec6e976ea0589888f16a8aa5e5a94c06420e38d46" gracePeriod=25
Nov 03 20:29:51.183351 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:51.183191    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-bx2dr"
Nov 03 20:29:51.206942 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:51.206567    5570 patch_prober.go:29] interesting pod/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-bx2dr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body=
Nov 03 20:29:51.206942 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:51.206632    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-bx2dr" podUID="ca63212a-77c0-4b97-8a23-5d0a518a55ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused"
Nov 03 20:29:51.222276 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:51.222043    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-7e4389-openshift-default-5b5bd7f55-kpk7n"
Nov 03 20:29:51.231353 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:51.231256    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-9pg4l"
Nov 03 20:29:51.232693 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:51.232589    5570 patch_prober.go:29] interesting pod/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-9pg4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body=
Nov 03 20:29:51.232693 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:51.232642    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-9pg4l" podUID="623f50bf-e25b-4647-9ffe-db4d163b2284" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused"
Nov 03 20:29:51.367816 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:51.365647    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-bx2dr"
Nov 03 20:29:51.569804 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:29:51.568535    5570 logging.go:55] [core] [Channel #1073 SubChannel #1074]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.123:50052", ServerName: "10.134.0.123:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.123:50052: connect: connection refused"
Nov 03 20:29:51.715708 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:51.715657    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-addf1a-openshift-default-69ffb7968c-pgb22" event={"ID":"e5d67e4b-c834-426f-8df5-8d6dc4ae77ab","Type":"ContainerStarted","Data":"0288429d09834921619e75ac8123d49724e489c65d28a7167ef2b79afb94848f"}
Nov 03 20:29:51.719539 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:51.719469    5570 generic.go:356] "Generic (PLEG): container finished" podID="880ebe10-49c7-4960-9fb2-2c699047ebb6" containerID="5f27f224d717a2b67dd1594ec6e976ea0589888f16a8aa5e5a94c06420e38d46" exitCode=0
Nov 03 20:29:51.721013 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:51.719802    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4cdbeb-7c468dc747-9kqqx" event={"ID":"880ebe10-49c7-4960-9fb2-2c699047ebb6","Type":"ContainerDied","Data":"5f27f224d717a2b67dd1594ec6e976ea0589888f16a8aa5e5a94c06420e38d46"}
Nov 03 20:29:51.722736 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:51.722710    5570 generic.go:356] "Generic (PLEG): container finished" podID="3716281c-9234-41d2-96c3-dbc38c692bbb" containerID="4be17df6901c7c6d2804455f4b35dab727aa9a50b40068440dca691f0ba75af3" exitCode=0
Nov 03 20:29:51.723688 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:51.723267    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-4cdbeb-77575c99cb-7pz9r" event={"ID":"3716281c-9234-41d2-96c3-dbc38c692bbb","Type":"ContainerDied","Data":"4be17df6901c7c6d2804455f4b35dab727aa9a50b40068440dca691f0ba75af3"}
Nov 03 20:29:51.914635 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:51.914602    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4cdbeb-7c468dc747-9kqqx"
Nov 03 20:29:52.019430 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:52.018363    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/880ebe10-49c7-4960-9fb2-2c699047ebb6-kube-api-access-rrhgz\" (UniqueName: \"kubernetes.io/projected/880ebe10-49c7-4960-9fb2-2c699047ebb6-kube-api-access-rrhgz\") pod \"880ebe10-49c7-4960-9fb2-2c699047ebb6\" (UID: \"880ebe10-49c7-4960-9fb2-2c699047ebb6\") "
Nov 03 20:29:52.019430 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:52.018632    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/880ebe10-49c7-4960-9fb2-2c699047ebb6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/880ebe10-49c7-4960-9fb2-2c699047ebb6-plugins-config-volume\") pod \"880ebe10-49c7-4960-9fb2-2c699047ebb6\" (UID: \"880ebe10-49c7-4960-9fb2-2c699047ebb6\") "
Nov 03 20:29:52.020384 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:52.020082    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/880ebe10-49c7-4960-9fb2-2c699047ebb6-plugins-config-volume" pod "880ebe10-49c7-4960-9fb2-2c699047ebb6" (UID: "880ebe10-49c7-4960-9fb2-2c699047ebb6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:29:52.079128 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:52.079015    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/880ebe10-49c7-4960-9fb2-2c699047ebb6-kube-api-access-rrhgz" pod "880ebe10-49c7-4960-9fb2-2c699047ebb6" (UID: "880ebe10-49c7-4960-9fb2-2c699047ebb6"). InnerVolumeSpecName "kube-api-access-rrhgz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:29:52.146204 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:52.142448    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrhgz\" (UniqueName: \"kubernetes.io/projected/880ebe10-49c7-4960-9fb2-2c699047ebb6-kube-api-access-rrhgz\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:29:52.146204 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:52.142490    5570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/880ebe10-49c7-4960-9fb2-2c699047ebb6-plugins-config-volume\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:29:52.207799 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:52.202976    5570 patch_prober.go:29] interesting pod/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-bx2dr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body=
Nov 03 20:29:52.207799 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:52.203045    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-bx2dr" podUID="ca63212a-77c0-4b97-8a23-5d0a518a55ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused"
Nov 03 20:29:52.245834 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:52.239452    5570 patch_prober.go:29] interesting pod/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-9pg4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body=
Nov 03 20:29:52.245834 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:52.239511    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-9pg4l" podUID="623f50bf-e25b-4647-9ffe-db4d163b2284" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused"
Nov 03 20:29:52.334705 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:29:52.276892    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f5aeb82_01f6_483f_8273_10bd0714e03c.slice/crio-b8aff8106a2f472d381d6827c2ccaf4419eb482ade529313331ca5e8096c7b19 WatchSource:0}: Error finding container b8aff8106a2f472d381d6827c2ccaf4419eb482ade529313331ca5e8096c7b19: Status 404 returned error can't find the container with id b8aff8106a2f472d381d6827c2ccaf4419eb482ade529313331ca5e8096c7b19
Nov 03 20:29:52.334705 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:52.328548    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:29:52.334705 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:52.328670    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:29:52.334705 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:52.328716    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:29:52.334705 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:52.331476    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-7e4389-openshift-default-5b5bd7f55-kpk7n"]
Nov 03 20:29:52.426259 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:52.426211    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-addf1a-openshift-default-69ffb7968c-pgb22"
Nov 03 20:29:52.497798 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:52.436183    5570 patch_prober.go:29] interesting pod/e2e-api-a-addf1a-openshift-default-69ffb7968c-pgb22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body=
Nov 03 20:29:52.497798 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:52.436236    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-addf1a-openshift-default-69ffb7968c-pgb22" podUID="e5d67e4b-c834-426f-8df5-8d6dc4ae77ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused"
Nov 03 20:29:52.600274 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:52.591603    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-api-b-4cdbeb-77575c99cb-7pz9r" podUID="3716281c-9234-41d2-96c3-dbc38c692bbb" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.123:50052\" within 1s: context deadline exceeded"
Nov 03 20:29:52.631414 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:52.631028    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-4cdbeb-77575c99cb-7pz9r"
Nov 03 20:29:52.807927 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:52.790003    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3716281c-9234-41d2-96c3-dbc38c692bbb-kube-api-access-kwb9b\" (UniqueName: \"kubernetes.io/projected/3716281c-9234-41d2-96c3-dbc38c692bbb-kube-api-access-kwb9b\") pod \"3716281c-9234-41d2-96c3-dbc38c692bbb\" (UID: \"3716281c-9234-41d2-96c3-dbc38c692bbb\") "
Nov 03 20:29:52.807927 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:52.790067    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3716281c-9234-41d2-96c3-dbc38c692bbb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3716281c-9234-41d2-96c3-dbc38c692bbb-plugins-config-volume\") pod \"3716281c-9234-41d2-96c3-dbc38c692bbb\" (UID: \"3716281c-9234-41d2-96c3-dbc38c692bbb\") "
Nov 03 20:29:52.819403 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:52.809715    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3716281c-9234-41d2-96c3-dbc38c692bbb-plugins-config-volume" pod "3716281c-9234-41d2-96c3-dbc38c692bbb" (UID: "3716281c-9234-41d2-96c3-dbc38c692bbb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:29:52.916738 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:52.910869    5570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3716281c-9234-41d2-96c3-dbc38c692bbb-plugins-config-volume\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:29:52.916738 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:52.914957    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3716281c-9234-41d2-96c3-dbc38c692bbb-kube-api-access-kwb9b" pod "3716281c-9234-41d2-96c3-dbc38c692bbb" (UID: "3716281c-9234-41d2-96c3-dbc38c692bbb"). InnerVolumeSpecName "kube-api-access-kwb9b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:29:52.931034 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:52.931002    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-2zvr4"
Nov 03 20:29:52.935691 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:52.935666    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-4cdbeb-77575c99cb-7pz9r" event={"ID":"3716281c-9234-41d2-96c3-dbc38c692bbb","Type":"ContainerDied","Data":"d992d82915b787a6337392b782cf3a127f25831eac12d01d25cfb43a6bd0ac16"}
Nov 03 20:29:52.935917 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:52.935896    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-4cdbeb-77575c99cb-7pz9r"
Nov 03 20:29:52.936048 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:52.936031    5570 scope.go:122] "RemoveContainer" containerID="4be17df6901c7c6d2804455f4b35dab727aa9a50b40068440dca691f0ba75af3"
Nov 03 20:29:53.035719 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:53.035680    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwb9b\" (UniqueName: \"kubernetes.io/projected/3716281c-9234-41d2-96c3-dbc38c692bbb-kube-api-access-kwb9b\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:29:53.148629 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:53.148554    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8edff27-9b0c-40e5-9a72-4a85d7efd3e9-istio-token\" (UniqueName: \"kubernetes.io/projected/c8edff27-9b0c-40e5-9a72-4a85d7efd3e9-istio-token\") pod \"c8edff27-9b0c-40e5-9a72-4a85d7efd3e9\" (UID: \"c8edff27-9b0c-40e5-9a72-4a85d7efd3e9\") "
Nov 03 20:29:53.148629 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:53.148623    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8edff27-9b0c-40e5-9a72-4a85d7efd3e9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8edff27-9b0c-40e5-9a72-4a85d7efd3e9-workload-socket\") pod \"c8edff27-9b0c-40e5-9a72-4a85d7efd3e9\" (UID: \"c8edff27-9b0c-40e5-9a72-4a85d7efd3e9\") "
Nov 03 20:29:53.148936 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:53.148657    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8edff27-9b0c-40e5-9a72-4a85d7efd3e9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8edff27-9b0c-40e5-9a72-4a85d7efd3e9-istio-data\") pod \"c8edff27-9b0c-40e5-9a72-4a85d7efd3e9\" (UID: \"c8edff27-9b0c-40e5-9a72-4a85d7efd3e9\") "
Nov 03 20:29:53.148936 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:53.148724    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c8edff27-9b0c-40e5-9a72-4a85d7efd3e9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8edff27-9b0c-40e5-9a72-4a85d7efd3e9-istio-podinfo\") pod \"c8edff27-9b0c-40e5-9a72-4a85d7efd3e9\" (UID: \"c8edff27-9b0c-40e5-9a72-4a85d7efd3e9\") "
Nov 03 20:29:53.152896 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:53.152856    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8edff27-9b0c-40e5-9a72-4a85d7efd3e9-workload-socket" pod "c8edff27-9b0c-40e5-9a72-4a85d7efd3e9" (UID: "c8edff27-9b0c-40e5-9a72-4a85d7efd3e9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:29:53.154028 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:53.153403    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8edff27-9b0c-40e5-9a72-4a85d7efd3e9-istio-data" pod "c8edff27-9b0c-40e5-9a72-4a85d7efd3e9" (UID: "c8edff27-9b0c-40e5-9a72-4a85d7efd3e9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:29:53.183909 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:53.178095    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8edff27-9b0c-40e5-9a72-4a85d7efd3e9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8edff27-9b0c-40e5-9a72-4a85d7efd3e9-istio-envoy\") pod \"c8edff27-9b0c-40e5-9a72-4a85d7efd3e9\" (UID: \"c8edff27-9b0c-40e5-9a72-4a85d7efd3e9\") "
Nov 03 20:29:53.183909 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:53.178173    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8edff27-9b0c-40e5-9a72-4a85d7efd3e9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8edff27-9b0c-40e5-9a72-4a85d7efd3e9-credential-socket\") pod \"c8edff27-9b0c-40e5-9a72-4a85d7efd3e9\" (UID: \"c8edff27-9b0c-40e5-9a72-4a85d7efd3e9\") "
Nov 03 20:29:53.183909 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:53.178317    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8edff27-9b0c-40e5-9a72-4a85d7efd3e9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8edff27-9b0c-40e5-9a72-4a85d7efd3e9-workload-certs\") pod \"c8edff27-9b0c-40e5-9a72-4a85d7efd3e9\" (UID: \"c8edff27-9b0c-40e5-9a72-4a85d7efd3e9\") "
Nov 03 20:29:53.183909 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:53.178527    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8edff27-9b0c-40e5-9a72-4a85d7efd3e9-kube-api-access-dfv7d\" (UniqueName: \"kubernetes.io/projected/c8edff27-9b0c-40e5-9a72-4a85d7efd3e9-kube-api-access-dfv7d\") pod \"c8edff27-9b0c-40e5-9a72-4a85d7efd3e9\" (UID: \"c8edff27-9b0c-40e5-9a72-4a85d7efd3e9\") "
Nov 03 20:29:53.183909 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:53.178626    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8edff27-9b0c-40e5-9a72-4a85d7efd3e9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8edff27-9b0c-40e5-9a72-4a85d7efd3e9-istiod-ca-cert\") pod \"c8edff27-9b0c-40e5-9a72-4a85d7efd3e9\" (UID: \"c8edff27-9b0c-40e5-9a72-4a85d7efd3e9\") "
Nov 03 20:29:53.183909 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:53.179794    5570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8edff27-9b0c-40e5-9a72-4a85d7efd3e9-workload-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:29:53.183909 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:53.179829    5570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8edff27-9b0c-40e5-9a72-4a85d7efd3e9-istio-data\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:29:53.183909 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:53.180184    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8edff27-9b0c-40e5-9a72-4a85d7efd3e9-workload-certs" pod "c8edff27-9b0c-40e5-9a72-4a85d7efd3e9" (UID: "c8edff27-9b0c-40e5-9a72-4a85d7efd3e9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:29:53.183909 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:53.181341    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8edff27-9b0c-40e5-9a72-4a85d7efd3e9-istiod-ca-cert" pod "c8edff27-9b0c-40e5-9a72-4a85d7efd3e9" (UID: "c8edff27-9b0c-40e5-9a72-4a85d7efd3e9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:29:53.183909 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:53.182013    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8edff27-9b0c-40e5-9a72-4a85d7efd3e9-credential-socket" pod "c8edff27-9b0c-40e5-9a72-4a85d7efd3e9" (UID: "c8edff27-9b0c-40e5-9a72-4a85d7efd3e9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:29:53.211956 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:53.196492    5570 patch_prober.go:29] interesting pod/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-bx2dr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body=
Nov 03 20:29:53.211956 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:53.196553    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-bx2dr" podUID="ca63212a-77c0-4b97-8a23-5d0a518a55ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused"
Nov 03 20:29:53.211956 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:53.197266    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4cdbeb-7c468dc747-9kqqx" event={"ID":"880ebe10-49c7-4960-9fb2-2c699047ebb6","Type":"ContainerDied","Data":"f5547614565e9560552ed9bd9f9655ff568060ec86e37f003aa1073fe240508c"}
Nov 03 20:29:53.211956 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:53.197331    5570 scope.go:122] "RemoveContainer" containerID="5f27f224d717a2b67dd1594ec6e976ea0589888f16a8aa5e5a94c06420e38d46"
Nov 03 20:29:53.211956 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:53.197436    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4cdbeb-7c468dc747-9kqqx"
Nov 03 20:29:53.244390 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:53.236500    5570 patch_prober.go:29] interesting pod/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-9pg4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body=
Nov 03 20:29:53.244390 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:53.236557    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-9pg4l" podUID="623f50bf-e25b-4647-9ffe-db4d163b2284" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused"
Nov 03 20:29:53.295938 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:53.295809    5570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8edff27-9b0c-40e5-9a72-4a85d7efd3e9-workload-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:29:53.296275 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:53.296259    5570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8edff27-9b0c-40e5-9a72-4a85d7efd3e9-istiod-ca-cert\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:29:53.296413 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:53.296399    5570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8edff27-9b0c-40e5-9a72-4a85d7efd3e9-credential-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:29:53.379056 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:53.378038    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-7e4389-openshift-default-5b5bd7f55-kpk7n" event={"ID":"1f5aeb82-01f6-483f-8273-10bd0714e03c","Type":"ContainerStarted","Data":"b8aff8106a2f472d381d6827c2ccaf4419eb482ade529313331ca5e8096c7b19"}
Nov 03 20:29:53.387147 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:53.382610    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8edff27-9b0c-40e5-9a72-4a85d7efd3e9-kube-api-access-dfv7d" pod "c8edff27-9b0c-40e5-9a72-4a85d7efd3e9" (UID: "c8edff27-9b0c-40e5-9a72-4a85d7efd3e9"). InnerVolumeSpecName "kube-api-access-dfv7d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:29:53.387147 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:53.382701    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c8edff27-9b0c-40e5-9a72-4a85d7efd3e9-istio-podinfo" pod "c8edff27-9b0c-40e5-9a72-4a85d7efd3e9" (UID: "c8edff27-9b0c-40e5-9a72-4a85d7efd3e9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:29:53.434837 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:53.406629    5570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8edff27-9b0c-40e5-9a72-4a85d7efd3e9-istio-podinfo\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:29:53.434837 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:53.406673    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dfv7d\" (UniqueName: \"kubernetes.io/projected/c8edff27-9b0c-40e5-9a72-4a85d7efd3e9-kube-api-access-dfv7d\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:29:53.523882 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:53.521991    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8edff27-9b0c-40e5-9a72-4a85d7efd3e9-istio-envoy" pod "c8edff27-9b0c-40e5-9a72-4a85d7efd3e9" (UID: "c8edff27-9b0c-40e5-9a72-4a85d7efd3e9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:29:53.546798 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:53.526039    5570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8edff27-9b0c-40e5-9a72-4a85d7efd3e9-istio-envoy\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:29:53.776716 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:53.776683    5570 patch_prober.go:29] interesting pod/e2e-api-a-addf1a-openshift-default-69ffb7968c-pgb22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body=
Nov 03 20:29:53.861030 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:53.860928    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-addf1a-openshift-default-69ffb7968c-pgb22" podUID="e5d67e4b-c834-426f-8df5-8d6dc4ae77ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused"
Nov 03 20:29:54.019265 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:54.019219    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/distinct-w3-482722-kserve-d876967c-kd488" podUID="9a97f9cc-2261-471d-9053-51ee965850bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:29:54.023796 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:54.019929    5570 generic.go:356] "Generic (PLEG): container finished" podID="c8edff27-9b0c-40e5-9a72-4a85d7efd3e9" containerID="511ac9d6b28ec82c5a67a5a17fb5cfd8ca9042b566e6369939735547e1f06b1f" exitCode=0
Nov 03 20:29:54.034917 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:54.034804    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-2zvr4"
Nov 03 20:29:54.036283 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:54.036246    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-2zvr4" event={"ID":"c8edff27-9b0c-40e5-9a72-4a85d7efd3e9","Type":"ContainerDied","Data":"511ac9d6b28ec82c5a67a5a17fb5cfd8ca9042b566e6369939735547e1f06b1f"}
Nov 03 20:29:54.128831 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:54.128409    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4cdbeb-7c468dc747-9kqqx"]
Nov 03 20:29:54.128831 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:54.128478    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-2zvr4" event={"ID":"c8edff27-9b0c-40e5-9a72-4a85d7efd3e9","Type":"ContainerDied","Data":"7359d0787ecdb6855a786aea51b75b85f1d2ce53fdaa9a4428988b88c0095fe5"}
Nov 03 20:29:54.188800 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:54.168750    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8edff27-9b0c-40e5-9a72-4a85d7efd3e9-istio-token" pod "c8edff27-9b0c-40e5-9a72-4a85d7efd3e9" (UID: "c8edff27-9b0c-40e5-9a72-4a85d7efd3e9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:29:54.199097 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:54.199052    5570 patch_prober.go:29] interesting pod/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-bx2dr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body=
Nov 03 20:29:54.200914 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:54.199344    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-bx2dr" podUID="ca63212a-77c0-4b97-8a23-5d0a518a55ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused"
Nov 03 20:29:54.223812 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:54.216887    5570 scope.go:122] "RemoveContainer" containerID="511ac9d6b28ec82c5a67a5a17fb5cfd8ca9042b566e6369939735547e1f06b1f"
Nov 03 20:29:54.305916 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:54.221291    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/distinct-2-w0-e40816-kserve-79fb46654d-s242x" podUID="4ad9b048-925c-4d01-a839-a5f25d4dd02d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded"
Nov 03 20:29:54.305916 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:54.231420    5570 patch_prober.go:29] interesting pod/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-9pg4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body=
Nov 03 20:29:54.305916 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:54.236962    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-9pg4l" podUID="623f50bf-e25b-4647-9ffe-db4d163b2284" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused"
Nov 03 20:29:54.317930 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:54.314431    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz" podUID="3a715ef1-38e9-4386-9eec-e10e8df11a10" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": context deadline exceeded"
Nov 03 20:29:54.417884 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:54.417847    5570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8edff27-9b0c-40e5-9a72-4a85d7efd3e9-istio-token\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:29:54.465801 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:54.453932    5570 patch_prober.go:29] interesting pod/e2e-api-a-addf1a-openshift-default-69ffb7968c-pgb22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body=
Nov 03 20:29:54.466467 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:54.466432    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-addf1a-openshift-default-69ffb7968c-pgb22" podUID="e5d67e4b-c834-426f-8df5-8d6dc4ae77ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused"
Nov 03 20:29:54.529804 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:54.527256    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh" podUID="2d3647f2-89e9-4908-a8ee-d84e527abeb0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:29:54.627043 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:54.626242    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-2-w6-9e1bcb-kserve-7d96d9f6f6-4f7fl" podUID="56353b0f-08ee-4ada-9dc0-ac5afd5a59c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded"
Nov 03 20:29:54.632117 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:54.631013    5570 scope.go:122] "RemoveContainer" containerID="511ac9d6b28ec82c5a67a5a17fb5cfd8ca9042b566e6369939735547e1f06b1f"
Nov 03 20:29:54.633321 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:29:54.632287    5570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"511ac9d6b28ec82c5a67a5a17fb5cfd8ca9042b566e6369939735547e1f06b1f\": container with ID starting with 511ac9d6b28ec82c5a67a5a17fb5cfd8ca9042b566e6369939735547e1f06b1f not found: ID does not exist" containerID="511ac9d6b28ec82c5a67a5a17fb5cfd8ca9042b566e6369939735547e1f06b1f"
Nov 03 20:29:54.633321 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:54.632334    5570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"511ac9d6b28ec82c5a67a5a17fb5cfd8ca9042b566e6369939735547e1f06b1f"} err="failed to get container status \"511ac9d6b28ec82c5a67a5a17fb5cfd8ca9042b566e6369939735547e1f06b1f\": rpc error: code = NotFound desc = could not find container \"511ac9d6b28ec82c5a67a5a17fb5cfd8ca9042b566e6369939735547e1f06b1f\": container with ID starting with 511ac9d6b28ec82c5a67a5a17fb5cfd8ca9042b566e6369939735547e1f06b1f not found: ID does not exist"
Nov 03 20:29:54.720798 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:54.704750    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4cdbeb-7c468dc747-9kqqx"]
Nov 03 20:29:54.729086 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:54.725922    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-4cdbeb-77575c99cb-7pz9r"]
Nov 03 20:29:54.729086 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:54.726485    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/premium-w2-9973d9-kserve-5c6dd4669c-5gv8x" podUID="cefc89ad-7ae1-433b-979d-6601678a6d98" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:29:54.733802 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:54.733176    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-4cdbeb-77575c99cb-7pz9r"]
Nov 03 20:29:55.024854 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:55.024818    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-2zvr4"]
Nov 03 20:29:55.091557 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:55.087833    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-6fc748-openshift-default-665546959f-2zvr4"]
Nov 03 20:29:55.183917 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:55.183871    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-7e4389-openshift-default-5b5bd7f55-kpk7n" event={"ID":"1f5aeb82-01f6-483f-8273-10bd0714e03c","Type":"ContainerStarted","Data":"17e044fcfdd8d20d3cc3fa7ec5897d1ad22ba6fcf7a0f47f100303c23ff48798"}
Nov 03 20:29:55.202849 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:55.202809    5570 patch_prober.go:29] interesting pod/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-bx2dr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body=
Nov 03 20:29:55.204153 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:55.204118    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-bx2dr" podUID="ca63212a-77c0-4b97-8a23-5d0a518a55ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused"
Nov 03 20:29:55.229565 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:55.229533    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-7e4389-openshift-default-5b5bd7f55-kpk7n"
Nov 03 20:29:55.244877 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:55.233524    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-7e4389-openshift-default-5b5bd7f55-kpk7n" podStartSLOduration=5.233503377 podStartE2EDuration="5.233503377s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:29:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:29:55.230652447 +0000 UTC m=+1529.383314339" watchObservedRunningTime="2026-11-03 20:29:55.233503377 +0000 UTC m=+1529.386165267"
Nov 03 20:29:55.273800 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:55.235095    5570 patch_prober.go:29] interesting pod/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-9pg4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body=
Nov 03 20:29:55.273800 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:55.259733    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-9pg4l" podUID="623f50bf-e25b-4647-9ffe-db4d163b2284" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused"
Nov 03 20:29:55.273800 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:55.262555    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-7e4389-openshift-default-5b5bd7f55-kpk7n"
Nov 03 20:29:55.412289 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:55.412193    5570 patch_prober.go:29] interesting pod/e2e-api-a-addf1a-openshift-default-69ffb7968c-pgb22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body=
Nov 03 20:29:55.413019 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:55.412464    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-addf1a-openshift-default-69ffb7968c-pgb22" podUID="e5d67e4b-c834-426f-8df5-8d6dc4ae77ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused"
Nov 03 20:29:56.029187 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:56.021516    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3716281c-9234-41d2-96c3-dbc38c692bbb" path="/var/lib/kubelet/pods/3716281c-9234-41d2-96c3-dbc38c692bbb/volumes"
Nov 03 20:29:56.052265 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:56.052230    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="880ebe10-49c7-4960-9fb2-2c699047ebb6" path="/var/lib/kubelet/pods/880ebe10-49c7-4960-9fb2-2c699047ebb6/volumes"
Nov 03 20:29:56.053398 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:56.053368    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8edff27-9b0c-40e5-9a72-4a85d7efd3e9" path="/var/lib/kubelet/pods/c8edff27-9b0c-40e5-9a72-4a85d7efd3e9/volumes"
Nov 03 20:29:56.181220 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:56.181183    5570 patch_prober.go:29] interesting pod/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-bx2dr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body=
Nov 03 20:29:56.181528 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:56.181498    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-bx2dr" podUID="ca63212a-77c0-4b97-8a23-5d0a518a55ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused"
Nov 03 20:29:56.211805 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:56.206887    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/distinct-2-w0-e40816-kserve-79fb46654d-s242x" podUID="4ad9b048-925c-4d01-a839-a5f25d4dd02d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:29:56.255799 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:56.251940    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-7e4389-openshift-default-5b5bd7f55-kpk7n"
Nov 03 20:29:56.258609 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:56.258578    5570 patch_prober.go:29] interesting pod/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-9pg4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body=
Nov 03 20:29:56.258862 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:56.258830    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-9pg4l" podUID="623f50bf-e25b-4647-9ffe-db4d163b2284" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused"
Nov 03 20:29:56.264045 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:56.264023    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-7e4389-openshift-default-5b5bd7f55-kpk7n"
Nov 03 20:29:56.312333 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:56.312143    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz" podUID="3a715ef1-38e9-4386-9eec-e10e8df11a10" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:29:56.325821 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:56.313843    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-2-w6-9e1bcb-kserve-7d96d9f6f6-4f7fl" podUID="56353b0f-08ee-4ada-9dc0-ac5afd5a59c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:29:56.367449 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:56.365836    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/distinct-2-w4-e80203-kserve-756f9b9bc6-69gxj" podUID="ccf5dc2f-f116-4917-bb1a-b136d8ed17f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:29:56.523801 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:56.505983    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/distinct-2-w3-482722-kserve-74bf58cf7d-v9cjw" podUID="9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded"
Nov 03 20:29:56.523801 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:56.518899    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/premium-w2-9973d9-kserve-5c6dd4669c-5gv8x" podUID="cefc89ad-7ae1-433b-979d-6601678a6d98" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:29:56.523801 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:56.519273    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh" podUID="2d3647f2-89e9-4908-a8ee-d84e527abeb0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:29:56.587863 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:56.572082    5570 patch_prober.go:29] interesting pod/e2e-api-a-addf1a-openshift-default-69ffb7968c-pgb22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body=
Nov 03 20:29:56.587863 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:56.572147    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-addf1a-openshift-default-69ffb7968c-pgb22" podUID="e5d67e4b-c834-426f-8df5-8d6dc4ae77ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused"
Nov 03 20:29:56.927487 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:56.927379    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/distinct-w3-482722-kserve-d876967c-kd488" podUID="9a97f9cc-2261-471d-9053-51ee965850bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:29:57.232642 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:57.232170    5570 patch_prober.go:29] interesting pod/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-9pg4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body=
Nov 03 20:29:57.232642 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:57.232235    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-9pg4l" podUID="623f50bf-e25b-4647-9ffe-db4d163b2284" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused"
Nov 03 20:29:57.309201 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:29:57.280613    5570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8bwhz" containerName="extractor"
Nov 03 20:29:57.413810 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:57.413165    5570 patch_prober.go:29] interesting pod/e2e-api-a-addf1a-openshift-default-69ffb7968c-pgb22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body=
Nov 03 20:29:57.413810 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:57.413225    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-addf1a-openshift-default-69ffb7968c-pgb22" podUID="e5d67e4b-c834-426f-8df5-8d6dc4ae77ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused"
Nov 03 20:29:57.616869 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:57.614264    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-bx2dr"
Nov 03 20:29:57.688168 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:57.664492    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-bx2dr"
Nov 03 20:29:58.332746 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:58.332713    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-9pg4l"
Nov 03 20:29:58.333648 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:58.333622    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-9pg4l"
Nov 03 20:29:58.334659 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:58.334638    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-9pg4l"
Nov 03 20:29:58.411062 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:58.411016    5570 patch_prober.go:29] interesting pod/e2e-api-a-addf1a-openshift-default-69ffb7968c-pgb22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body=
Nov 03 20:29:58.411219 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:58.411074    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-addf1a-openshift-default-69ffb7968c-pgb22" podUID="e5d67e4b-c834-426f-8df5-8d6dc4ae77ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused"
Nov 03 20:29:59.415755 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:59.415724    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-addf1a-openshift-default-69ffb7968c-pgb22"
Nov 03 20:29:59.416309 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:59.416285    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-addf1a-openshift-default-69ffb7968c-pgb22"
Nov 03 20:29:59.417232 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:29:59.417208    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-addf1a-openshift-default-69ffb7968c-pgb22"
Nov 03 20:30:00.184224 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:00.184193    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-9e1bcb-29895630-tgcnz"]
Nov 03 20:30:00.185158 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:00.185129    5570 cpu_manager.go:414] "containerMap: removing container" podUID="880ebe10-49c7-4960-9fb2-2c699047ebb6" containerName="payload-pre-processing"
Nov 03 20:30:00.185158 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:00.185156    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="880ebe10-49c7-4960-9fb2-2c699047ebb6" containerName="payload-pre-processing"
Nov 03 20:30:00.185407 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:00.185193    5570 cpu_manager.go:414] "containerMap: removing container" podUID="c8edff27-9b0c-40e5-9a72-4a85d7efd3e9" containerName="istio-proxy"
Nov 03 20:30:00.185407 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:00.185202    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8edff27-9b0c-40e5-9a72-4a85d7efd3e9" containerName="istio-proxy"
Nov 03 20:30:00.185407 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:00.185239    5570 cpu_manager.go:414] "containerMap: removing container" podUID="3716281c-9234-41d2-96c3-dbc38c692bbb" containerName="payload-processing"
Nov 03 20:30:00.185407 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:00.185247    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3716281c-9234-41d2-96c3-dbc38c692bbb" containerName="payload-processing"
Nov 03 20:30:00.185674 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:00.185470    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="880ebe10-49c7-4960-9fb2-2c699047ebb6" containerName="payload-pre-processing"
Nov 03 20:30:00.185674 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:00.185488    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8edff27-9b0c-40e5-9a72-4a85d7efd3e9" containerName="istio-proxy"
Nov 03 20:30:00.185674 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:00.185506    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="3716281c-9234-41d2-96c3-dbc38c692bbb" containerName="payload-processing"
Nov 03 20:30:00.194519 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:00.194491    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-b917c2-29895630-l49db"]
Nov 03 20:30:00.194664 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:00.194589    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-9e1bcb-29895630-tgcnz"
Nov 03 20:30:00.199441 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:00.199416    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-b917c2-29895630-l49db"
Nov 03 20:30:00.203477 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:00.203449    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-17b3d9-29895630-btts9"]
Nov 03 20:30:00.214581 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:00.214557    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-9e1bcb-29895630-tgcnz"]
Nov 03 20:30:00.214720 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:00.214689    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-17b3d9-29895630-btts9"
Nov 03 20:30:00.234570 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:00.234543    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-b917c2-29895630-l49db"]
Nov 03 20:30:00.241813 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:00.240985    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d1bfb8-29895630-m6rrt"]
Nov 03 20:30:00.246886 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:00.246860    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d1bfb8-29895630-m6rrt"
Nov 03 20:30:00.252918 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:00.252887    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e40816-29895630-4n472"]
Nov 03 20:30:00.263625 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:00.263582    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e40816-29895630-4n472"
Nov 03 20:30:00.279656 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:00.279623    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l7s5w\""
Nov 03 20:30:00.306444 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:00.306413    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-17b3d9-29895630-btts9"]
Nov 03 20:30:00.308609 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:00.308585    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d1bfb8-29895630-m6rrt"]
Nov 03 20:30:00.314866 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:00.314836    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-482722-29895630-9pjh8"]
Nov 03 20:30:00.320744 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:00.320703    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-482722-29895630-9pjh8"
Nov 03 20:30:00.323856 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:00.323832    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-addf1a-29895630-btmk6"]
Nov 03 20:30:00.331080 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:00.330736    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e40816-29895630-4n472"]
Nov 03 20:30:00.331080 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:00.330893    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-addf1a-29895630-btmk6"
Nov 03 20:30:00.334387 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:00.334322    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-482722-29895630-9pjh8"]
Nov 03 20:30:00.337317 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:00.337066    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b745fba-4594-4621-81bd-e322d6dd1efa-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-9e1bcb-29895630-tgcnz\" (UID: \"3b745fba-4594-4621-81bd-e322d6dd1efa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-9e1bcb-29895630-tgcnz"
Nov 03 20:30:00.337317 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:00.337118    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c8a1c8b-4798-435a-a859-f6d1bc27efda-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-17b3d9-29895630-btts9\" (UID: \"4c8a1c8b-4798-435a-a859-f6d1bc27efda\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-17b3d9-29895630-btts9"
Nov 03 20:30:00.337317 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:00.337199    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hp9d\" (UniqueName: \"kubernetes.io/projected/3b745fba-4594-4621-81bd-e322d6dd1efa-kube-api-access-5hp9d\") pod \"maas-api-key-cleanup-e2e-worker-w6-9e1bcb-29895630-tgcnz\" (UID: \"3b745fba-4594-4621-81bd-e322d6dd1efa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-9e1bcb-29895630-tgcnz"
Nov 03 20:30:00.337317 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:00.337243    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e73f70d-ab6c-4cc8-aff1-4d9833b5b991-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-d1bfb8-29895630-m6rrt\" (UID: \"7e73f70d-ab6c-4cc8-aff1-4d9833b5b991\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d1bfb8-29895630-m6rrt"
Nov 03 20:30:00.337590 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:00.337349    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cfvx\" (UniqueName: \"kubernetes.io/projected/7e73f70d-ab6c-4cc8-aff1-4d9833b5b991-kube-api-access-4cfvx\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-d1bfb8-29895630-m6rrt\" (UID: \"7e73f70d-ab6c-4cc8-aff1-4d9833b5b991\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d1bfb8-29895630-m6rrt"
Nov 03 20:30:00.337590 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:00.337407    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zst2g\" (UniqueName: \"kubernetes.io/projected/c9cb6b5f-bf07-48ed-846a-eae8a3c19c1d-kube-api-access-zst2g\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-b917c2-29895630-l49db\" (UID: \"c9cb6b5f-bf07-48ed-846a-eae8a3c19c1d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-b917c2-29895630-l49db"
Nov 03 20:30:00.337590 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:00.337547    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-228pp\" (UniqueName: \"kubernetes.io/projected/4c8a1c8b-4798-435a-a859-f6d1bc27efda-kube-api-access-228pp\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-17b3d9-29895630-btts9\" (UID: \"4c8a1c8b-4798-435a-a859-f6d1bc27efda\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-17b3d9-29895630-btts9"
Nov 03 20:30:00.337590 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:00.337585    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9cb6b5f-bf07-48ed-846a-eae8a3c19c1d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-b917c2-29895630-l49db\" (UID: \"c9cb6b5f-bf07-48ed-846a-eae8a3c19c1d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-b917c2-29895630-l49db"
Nov 03 20:30:00.342628 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:00.339383    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-addf1a-29895630-btmk6"]
Nov 03 20:30:00.439489 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:00.439396    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cfvx\" (UniqueName: \"kubernetes.io/projected/7e73f70d-ab6c-4cc8-aff1-4d9833b5b991-kube-api-access-4cfvx\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-d1bfb8-29895630-m6rrt\" (UID: \"7e73f70d-ab6c-4cc8-aff1-4d9833b5b991\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d1bfb8-29895630-m6rrt"
Nov 03 20:30:00.439489 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:00.439449    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spsw6\" (UniqueName: \"kubernetes.io/projected/36bf8581-603b-4a8d-8d7b-6da231cf7161-kube-api-access-spsw6\") pod \"maas-api-key-cleanup-e2e-worker-w3-482722-29895630-9pjh8\" (UID: \"36bf8581-603b-4a8d-8d7b-6da231cf7161\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-482722-29895630-9pjh8"
Nov 03 20:30:00.440076 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:00.439523    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zst2g\" (UniqueName: \"kubernetes.io/projected/c9cb6b5f-bf07-48ed-846a-eae8a3c19c1d-kube-api-access-zst2g\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-b917c2-29895630-l49db\" (UID: \"c9cb6b5f-bf07-48ed-846a-eae8a3c19c1d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-b917c2-29895630-l49db"
Nov 03 20:30:00.440076 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:00.439815    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ksn9\" (UniqueName: \"kubernetes.io/projected/11f669a0-4617-4c3e-96cc-fc288ef2aea1-kube-api-access-8ksn9\") pod \"maas-api-key-cleanup-e2e-worker-w0-e40816-29895630-4n472\" (UID: \"11f669a0-4617-4c3e-96cc-fc288ef2aea1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e40816-29895630-4n472"
Nov 03 20:30:00.440076 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:00.439974    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-228pp\" (UniqueName: \"kubernetes.io/projected/4c8a1c8b-4798-435a-a859-f6d1bc27efda-kube-api-access-228pp\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-17b3d9-29895630-btts9\" (UID: \"4c8a1c8b-4798-435a-a859-f6d1bc27efda\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-17b3d9-29895630-btts9"
Nov 03 20:30:00.440076 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:00.440021    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9cb6b5f-bf07-48ed-846a-eae8a3c19c1d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-b917c2-29895630-l49db\" (UID: \"c9cb6b5f-bf07-48ed-846a-eae8a3c19c1d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-b917c2-29895630-l49db"
Nov 03 20:30:00.440309 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:00.440284    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b3cfadc-bda0-4d0a-abff-056991a17a37-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-addf1a-29895630-btmk6\" (UID: \"4b3cfadc-bda0-4d0a-abff-056991a17a37\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-addf1a-29895630-btmk6"
Nov 03 20:30:00.440403 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:00.440382    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b745fba-4594-4621-81bd-e322d6dd1efa-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-9e1bcb-29895630-tgcnz\" (UID: \"3b745fba-4594-4621-81bd-e322d6dd1efa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-9e1bcb-29895630-tgcnz"
Nov 03 20:30:00.440453 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:00.440414    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkbjf\" (UniqueName: \"kubernetes.io/projected/4b3cfadc-bda0-4d0a-abff-056991a17a37-kube-api-access-fkbjf\") pod \"maas-api-key-cleanup-e2e-api-a-addf1a-29895630-btmk6\" (UID: \"4b3cfadc-bda0-4d0a-abff-056991a17a37\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-addf1a-29895630-btmk6"
Nov 03 20:30:00.440514 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:00.440454    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c8a1c8b-4798-435a-a859-f6d1bc27efda-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-17b3d9-29895630-btts9\" (UID: \"4c8a1c8b-4798-435a-a859-f6d1bc27efda\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-17b3d9-29895630-btts9"
Nov 03 20:30:00.440647 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:00.440626    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/11f669a0-4617-4c3e-96cc-fc288ef2aea1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-e40816-29895630-4n472\" (UID: \"11f669a0-4617-4c3e-96cc-fc288ef2aea1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e40816-29895630-4n472"
Nov 03 20:30:00.440647 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:00.440634    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9cb6b5f-bf07-48ed-846a-eae8a3c19c1d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-b917c2-29895630-l49db\" (UID: \"c9cb6b5f-bf07-48ed-846a-eae8a3c19c1d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-b917c2-29895630-l49db"
Nov 03 20:30:00.440770 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:00.440683    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/36bf8581-603b-4a8d-8d7b-6da231cf7161-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-482722-29895630-9pjh8\" (UID: \"36bf8581-603b-4a8d-8d7b-6da231cf7161\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-482722-29895630-9pjh8"
Nov 03 20:30:00.440770 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:00.440725    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hp9d\" (UniqueName: \"kubernetes.io/projected/3b745fba-4594-4621-81bd-e322d6dd1efa-kube-api-access-5hp9d\") pod \"maas-api-key-cleanup-e2e-worker-w6-9e1bcb-29895630-tgcnz\" (UID: \"3b745fba-4594-4621-81bd-e322d6dd1efa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-9e1bcb-29895630-tgcnz"
Nov 03 20:30:00.440968 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:00.440808    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e73f70d-ab6c-4cc8-aff1-4d9833b5b991-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-d1bfb8-29895630-m6rrt\" (UID: \"7e73f70d-ab6c-4cc8-aff1-4d9833b5b991\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d1bfb8-29895630-m6rrt"
Nov 03 20:30:00.441179 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:00.441148    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c8a1c8b-4798-435a-a859-f6d1bc27efda-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-17b3d9-29895630-btts9\" (UID: \"4c8a1c8b-4798-435a-a859-f6d1bc27efda\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-17b3d9-29895630-btts9"
Nov 03 20:30:00.441282 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:00.441247    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b745fba-4594-4621-81bd-e322d6dd1efa-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-9e1bcb-29895630-tgcnz\" (UID: \"3b745fba-4594-4621-81bd-e322d6dd1efa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-9e1bcb-29895630-tgcnz"
Nov 03 20:30:00.441825 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:00.441764    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e73f70d-ab6c-4cc8-aff1-4d9833b5b991-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-d1bfb8-29895630-m6rrt\" (UID: \"7e73f70d-ab6c-4cc8-aff1-4d9833b5b991\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d1bfb8-29895630-m6rrt"
Nov 03 20:30:00.464112 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:00.464073    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-228pp\" (UniqueName: \"kubernetes.io/projected/4c8a1c8b-4798-435a-a859-f6d1bc27efda-kube-api-access-228pp\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-17b3d9-29895630-btts9\" (UID: \"4c8a1c8b-4798-435a-a859-f6d1bc27efda\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-17b3d9-29895630-btts9"
Nov 03 20:30:00.464616 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:00.464592    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cfvx\" (UniqueName: \"kubernetes.io/projected/7e73f70d-ab6c-4cc8-aff1-4d9833b5b991-kube-api-access-4cfvx\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-d1bfb8-29895630-m6rrt\" (UID: \"7e73f70d-ab6c-4cc8-aff1-4d9833b5b991\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d1bfb8-29895630-m6rrt"
Nov 03 20:30:00.464696 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:00.464592    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hp9d\" (UniqueName: \"kubernetes.io/projected/3b745fba-4594-4621-81bd-e322d6dd1efa-kube-api-access-5hp9d\") pod \"maas-api-key-cleanup-e2e-worker-w6-9e1bcb-29895630-tgcnz\" (UID: \"3b745fba-4594-4621-81bd-e322d6dd1efa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-9e1bcb-29895630-tgcnz"
Nov 03 20:30:00.465021 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:00.465005    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zst2g\" (UniqueName: \"kubernetes.io/projected/c9cb6b5f-bf07-48ed-846a-eae8a3c19c1d-kube-api-access-zst2g\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-b917c2-29895630-l49db\" (UID: \"c9cb6b5f-bf07-48ed-846a-eae8a3c19c1d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-b917c2-29895630-l49db"
Nov 03 20:30:00.519485 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:00.519446    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-9e1bcb-29895630-tgcnz"
Nov 03 20:30:00.539853 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:00.539827    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-b917c2-29895630-l49db"
Nov 03 20:30:00.543379 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:00.543316    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkbjf\" (UniqueName: \"kubernetes.io/projected/4b3cfadc-bda0-4d0a-abff-056991a17a37-kube-api-access-fkbjf\") pod \"maas-api-key-cleanup-e2e-api-a-addf1a-29895630-btmk6\" (UID: \"4b3cfadc-bda0-4d0a-abff-056991a17a37\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-addf1a-29895630-btmk6"
Nov 03 20:30:00.543468 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:00.543397    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/11f669a0-4617-4c3e-96cc-fc288ef2aea1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-e40816-29895630-4n472\" (UID: \"11f669a0-4617-4c3e-96cc-fc288ef2aea1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e40816-29895630-4n472"
Nov 03 20:30:00.543468 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:00.543431    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/36bf8581-603b-4a8d-8d7b-6da231cf7161-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-482722-29895630-9pjh8\" (UID: \"36bf8581-603b-4a8d-8d7b-6da231cf7161\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-482722-29895630-9pjh8"
Nov 03 20:30:00.543637 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:00.543564    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spsw6\" (UniqueName: \"kubernetes.io/projected/36bf8581-603b-4a8d-8d7b-6da231cf7161-kube-api-access-spsw6\") pod \"maas-api-key-cleanup-e2e-worker-w3-482722-29895630-9pjh8\" (UID: \"36bf8581-603b-4a8d-8d7b-6da231cf7161\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-482722-29895630-9pjh8"
Nov 03 20:30:00.543686 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:00.543672    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ksn9\" (UniqueName: \"kubernetes.io/projected/11f669a0-4617-4c3e-96cc-fc288ef2aea1-kube-api-access-8ksn9\") pod \"maas-api-key-cleanup-e2e-worker-w0-e40816-29895630-4n472\" (UID: \"11f669a0-4617-4c3e-96cc-fc288ef2aea1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e40816-29895630-4n472"
Nov 03 20:30:00.543885 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:00.543864    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b3cfadc-bda0-4d0a-abff-056991a17a37-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-addf1a-29895630-btmk6\" (UID: \"4b3cfadc-bda0-4d0a-abff-056991a17a37\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-addf1a-29895630-btmk6"
Nov 03 20:30:00.544383 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:00.544361    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/11f669a0-4617-4c3e-96cc-fc288ef2aea1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-e40816-29895630-4n472\" (UID: \"11f669a0-4617-4c3e-96cc-fc288ef2aea1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e40816-29895630-4n472"
Nov 03 20:30:00.544601 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:00.544581    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b3cfadc-bda0-4d0a-abff-056991a17a37-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-addf1a-29895630-btmk6\" (UID: \"4b3cfadc-bda0-4d0a-abff-056991a17a37\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-addf1a-29895630-btmk6"
Nov 03 20:30:00.545173 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:00.545151    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/36bf8581-603b-4a8d-8d7b-6da231cf7161-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-482722-29895630-9pjh8\" (UID: \"36bf8581-603b-4a8d-8d7b-6da231cf7161\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-482722-29895630-9pjh8"
Nov 03 20:30:00.556733 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:00.556702    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-17b3d9-29895630-btts9"
Nov 03 20:30:00.574906 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:00.573687    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-spsw6\" (UniqueName: \"kubernetes.io/projected/36bf8581-603b-4a8d-8d7b-6da231cf7161-kube-api-access-spsw6\") pod \"maas-api-key-cleanup-e2e-worker-w3-482722-29895630-9pjh8\" (UID: \"36bf8581-603b-4a8d-8d7b-6da231cf7161\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-482722-29895630-9pjh8"
Nov 03 20:30:00.575631 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:00.575574    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkbjf\" (UniqueName: \"kubernetes.io/projected/4b3cfadc-bda0-4d0a-abff-056991a17a37-kube-api-access-fkbjf\") pod \"maas-api-key-cleanup-e2e-api-a-addf1a-29895630-btmk6\" (UID: \"4b3cfadc-bda0-4d0a-abff-056991a17a37\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-addf1a-29895630-btmk6"
Nov 03 20:30:00.576898 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:00.576865    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ksn9\" (UniqueName: \"kubernetes.io/projected/11f669a0-4617-4c3e-96cc-fc288ef2aea1-kube-api-access-8ksn9\") pod \"maas-api-key-cleanup-e2e-worker-w0-e40816-29895630-4n472\" (UID: \"11f669a0-4617-4c3e-96cc-fc288ef2aea1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e40816-29895630-4n472"
Nov 03 20:30:00.595377 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:00.595342    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d1bfb8-29895630-m6rrt"
Nov 03 20:30:00.616060 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:00.615943    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e40816-29895630-4n472"
Nov 03 20:30:00.650558 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:00.650525    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-482722-29895630-9pjh8"
Nov 03 20:30:00.656430 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:00.656326    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-addf1a-29895630-btmk6"
Nov 03 20:30:00.842304 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:30:00.842242    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b745fba_4594_4621_81bd_e322d6dd1efa.slice/crio-51dac6a9c0ca4ae1694e5011176f43f90724a342109cb1a4dc0c2faa57635002 WatchSource:0}: Error finding container 51dac6a9c0ca4ae1694e5011176f43f90724a342109cb1a4dc0c2faa57635002: Status 404 returned error can't find the container with id 51dac6a9c0ca4ae1694e5011176f43f90724a342109cb1a4dc0c2faa57635002
Nov 03 20:30:00.843679 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:00.843645    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-9e1bcb-29895630-tgcnz"]
Nov 03 20:30:00.890583 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:00.890555    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-b917c2-29895630-l49db"]
Nov 03 20:30:00.890721 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:30:00.890702    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9cb6b5f_bf07_48ed_846a_eae8a3c19c1d.slice/crio-70dde93fc27bc8a4f513203ac8518673f6102a906660dbf048590a1ee3b2bd3a WatchSource:0}: Error finding container 70dde93fc27bc8a4f513203ac8518673f6102a906660dbf048590a1ee3b2bd3a: Status 404 returned error can't find the container with id 70dde93fc27bc8a4f513203ac8518673f6102a906660dbf048590a1ee3b2bd3a
Nov 03 20:30:00.921704 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:30:00.921673    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c8a1c8b_4798_435a_a859_f6d1bc27efda.slice/crio-c36cb2ad9ee7211d545ca4aa6db77573d85573eae18b6a7cfca98a89e0b628d2 WatchSource:0}: Error finding container c36cb2ad9ee7211d545ca4aa6db77573d85573eae18b6a7cfca98a89e0b628d2: Status 404 returned error can't find the container with id c36cb2ad9ee7211d545ca4aa6db77573d85573eae18b6a7cfca98a89e0b628d2
Nov 03 20:30:00.922300 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:00.922274    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-17b3d9-29895630-btts9"]
Nov 03 20:30:00.964916 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:00.964888    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d1bfb8-29895630-m6rrt"]
Nov 03 20:30:00.967145 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:30:00.967119    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e73f70d_ab6c_4cc8_aff1_4d9833b5b991.slice/crio-29513b3174e93d4d40415b3f04ea40422e13d3c6cceff46642ba6de36e27c611 WatchSource:0}: Error finding container 29513b3174e93d4d40415b3f04ea40422e13d3c6cceff46642ba6de36e27c611: Status 404 returned error can't find the container with id 29513b3174e93d4d40415b3f04ea40422e13d3c6cceff46642ba6de36e27c611
Nov 03 20:30:01.001155 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:01.001119    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e40816-29895630-4n472"]
Nov 03 20:30:01.010182 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:30:01.010146    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11f669a0_4617_4c3e_96cc_fc288ef2aea1.slice/crio-eadbc6fed59cd16b42e9ed22f65dd7157da701ea6580ff1693f4a2765f5f607f WatchSource:0}: Error finding container eadbc6fed59cd16b42e9ed22f65dd7157da701ea6580ff1693f4a2765f5f607f: Status 404 returned error can't find the container with id eadbc6fed59cd16b42e9ed22f65dd7157da701ea6580ff1693f4a2765f5f607f
Nov 03 20:30:01.109246 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:01.109206    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-482722-29895630-9pjh8"]
Nov 03 20:30:01.131571 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:01.130721    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-addf1a-29895630-btmk6"]
Nov 03 20:30:01.354578 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:01.354545    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-addf1a-29895630-btmk6" event={"ID":"4b3cfadc-bda0-4d0a-abff-056991a17a37","Type":"ContainerStarted","Data":"88d15c9836555cd744dcd834adad28c9a90935230bd1e881218441127539e057"}
Nov 03 20:30:01.355844 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:01.355806    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e40816-29895630-4n472" event={"ID":"11f669a0-4617-4c3e-96cc-fc288ef2aea1","Type":"ContainerStarted","Data":"eadbc6fed59cd16b42e9ed22f65dd7157da701ea6580ff1693f4a2765f5f607f"}
Nov 03 20:30:01.357825 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:01.357801    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-b917c2-29895630-l49db" event={"ID":"c9cb6b5f-bf07-48ed-846a-eae8a3c19c1d","Type":"ContainerStarted","Data":"70dde93fc27bc8a4f513203ac8518673f6102a906660dbf048590a1ee3b2bd3a"}
Nov 03 20:30:01.359668 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:01.359641    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-9e1bcb-29895630-tgcnz" event={"ID":"3b745fba-4594-4621-81bd-e322d6dd1efa","Type":"ContainerStarted","Data":"e480a298621964ac57cec51cfe2fb384bc6450e50ce11ce42a5cceb203562808"}
Nov 03 20:30:01.359834 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:01.359702    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-9e1bcb-29895630-tgcnz" event={"ID":"3b745fba-4594-4621-81bd-e322d6dd1efa","Type":"ContainerStarted","Data":"51dac6a9c0ca4ae1694e5011176f43f90724a342109cb1a4dc0c2faa57635002"}
Nov 03 20:30:01.361137 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:01.361110    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d1bfb8-29895630-m6rrt" event={"ID":"7e73f70d-ab6c-4cc8-aff1-4d9833b5b991","Type":"ContainerStarted","Data":"29513b3174e93d4d40415b3f04ea40422e13d3c6cceff46642ba6de36e27c611"}
Nov 03 20:30:01.362321 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:01.362300    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-17b3d9-29895630-btts9" event={"ID":"4c8a1c8b-4798-435a-a859-f6d1bc27efda","Type":"ContainerStarted","Data":"c36cb2ad9ee7211d545ca4aa6db77573d85573eae18b6a7cfca98a89e0b628d2"}
Nov 03 20:30:01.363474 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:01.363448    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-482722-29895630-9pjh8" event={"ID":"36bf8581-603b-4a8d-8d7b-6da231cf7161","Type":"ContainerStarted","Data":"4e0344c18c145edeedad4b156e5c97da12b8ef432d60d52f49b3252dbfc6416c"}
Nov 03 20:30:01.465065 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:01.462462    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-b917c2-29895630-l49db" podStartSLOduration=1.4307552000000001 podStartE2EDuration="1.4307552s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:30:01.414087301 +0000 UTC m=+1535.566749192" watchObservedRunningTime="2026-11-03 20:30:01.4307552 +0000 UTC m=+1535.583417092"
Nov 03 20:30:01.510711 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:01.510595    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-9e1bcb-29895630-tgcnz" podStartSLOduration=1.5105755539999999 podStartE2EDuration="1.510575554s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:30:01.503624408 +0000 UTC m=+1535.656286327" watchObservedRunningTime="2026-11-03 20:30:01.510575554 +0000 UTC m=+1535.663237443"
Nov 03 20:30:02.375910 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:02.375875    5570 generic.go:356] "Generic (PLEG): container finished" podID="3b745fba-4594-4621-81bd-e322d6dd1efa" containerID="e480a298621964ac57cec51cfe2fb384bc6450e50ce11ce42a5cceb203562808" exitCode=60
Nov 03 20:30:02.376123 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:02.375958    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-9e1bcb-29895630-tgcnz" event={"ID":"3b745fba-4594-4621-81bd-e322d6dd1efa","Type":"ContainerDied","Data":"e480a298621964ac57cec51cfe2fb384bc6450e50ce11ce42a5cceb203562808"}
Nov 03 20:30:02.376376 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:02.376356    5570 scope.go:122] "RemoveContainer" containerID="e480a298621964ac57cec51cfe2fb384bc6450e50ce11ce42a5cceb203562808"
Nov 03 20:30:02.377934 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:02.377908    5570 generic.go:356] "Generic (PLEG): container finished" podID="7e73f70d-ab6c-4cc8-aff1-4d9833b5b991" containerID="ba5586a0ffbfc2870393be764d0e9c339c790adb853ff08295c44d3cc044cc93" exitCode=60
Nov 03 20:30:02.378012 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:02.377987    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d1bfb8-29895630-m6rrt" event={"ID":"7e73f70d-ab6c-4cc8-aff1-4d9833b5b991","Type":"ContainerDied","Data":"ba5586a0ffbfc2870393be764d0e9c339c790adb853ff08295c44d3cc044cc93"}
Nov 03 20:30:02.378260 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:02.378241    5570 scope.go:122] "RemoveContainer" containerID="ba5586a0ffbfc2870393be764d0e9c339c790adb853ff08295c44d3cc044cc93"
Nov 03 20:30:02.380067 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:02.380046    5570 generic.go:356] "Generic (PLEG): container finished" podID="4c8a1c8b-4798-435a-a859-f6d1bc27efda" containerID="11601267c1adb4f3d47827398cd66f607f981fe45f500a72a8c98be276b4e4a7" exitCode=60
Nov 03 20:30:02.380159 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:02.380076    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-17b3d9-29895630-btts9" event={"ID":"4c8a1c8b-4798-435a-a859-f6d1bc27efda","Type":"ContainerDied","Data":"11601267c1adb4f3d47827398cd66f607f981fe45f500a72a8c98be276b4e4a7"}
Nov 03 20:30:02.380361 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:02.380337    5570 scope.go:122] "RemoveContainer" containerID="11601267c1adb4f3d47827398cd66f607f981fe45f500a72a8c98be276b4e4a7"
Nov 03 20:30:02.382045 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:02.382019    5570 generic.go:356] "Generic (PLEG): container finished" podID="36bf8581-603b-4a8d-8d7b-6da231cf7161" containerID="243b4d4d23db7ad36f7ca14b76e1573cbac46b724d52f137231f03e6177cae45" exitCode=60
Nov 03 20:30:02.382117 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:02.382097    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-482722-29895630-9pjh8" event={"ID":"36bf8581-603b-4a8d-8d7b-6da231cf7161","Type":"ContainerDied","Data":"243b4d4d23db7ad36f7ca14b76e1573cbac46b724d52f137231f03e6177cae45"}
Nov 03 20:30:02.382308 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:02.382292    5570 scope.go:122] "RemoveContainer" containerID="243b4d4d23db7ad36f7ca14b76e1573cbac46b724d52f137231f03e6177cae45"
Nov 03 20:30:02.383874 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:02.383851    5570 generic.go:356] "Generic (PLEG): container finished" podID="4b3cfadc-bda0-4d0a-abff-056991a17a37" containerID="b6baf752e6a24af8613ca833bfd9367b19d880e8fcbf67373c5a69adcb8b2475" exitCode=60
Nov 03 20:30:02.383983 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:02.383962    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-addf1a-29895630-btmk6" event={"ID":"4b3cfadc-bda0-4d0a-abff-056991a17a37","Type":"ContainerDied","Data":"b6baf752e6a24af8613ca833bfd9367b19d880e8fcbf67373c5a69adcb8b2475"}
Nov 03 20:30:02.384206 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:02.384190    5570 scope.go:122] "RemoveContainer" containerID="b6baf752e6a24af8613ca833bfd9367b19d880e8fcbf67373c5a69adcb8b2475"
Nov 03 20:30:02.385580 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:02.385560    5570 generic.go:356] "Generic (PLEG): container finished" podID="11f669a0-4617-4c3e-96cc-fc288ef2aea1" containerID="08645a3897f1706b5def167c003a4778ae412eaf2402b688d4b0ffefee9d50dd" exitCode=60
Nov 03 20:30:02.385671 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:02.385636    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e40816-29895630-4n472" event={"ID":"11f669a0-4617-4c3e-96cc-fc288ef2aea1","Type":"ContainerDied","Data":"08645a3897f1706b5def167c003a4778ae412eaf2402b688d4b0ffefee9d50dd"}
Nov 03 20:30:02.385879 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:02.385863    5570 scope.go:122] "RemoveContainer" containerID="08645a3897f1706b5def167c003a4778ae412eaf2402b688d4b0ffefee9d50dd"
Nov 03 20:30:02.387138 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:02.387109    5570 generic.go:356] "Generic (PLEG): container finished" podID="c9cb6b5f-bf07-48ed-846a-eae8a3c19c1d" containerID="a779706851882c9381f962b5b7459ef2a5a7fe0cbc59b1c65e380186d5b184ad" exitCode=60
Nov 03 20:30:02.387238 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:02.387162    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-b917c2-29895630-l49db" event={"ID":"c9cb6b5f-bf07-48ed-846a-eae8a3c19c1d","Type":"ContainerDied","Data":"a779706851882c9381f962b5b7459ef2a5a7fe0cbc59b1c65e380186d5b184ad"}
Nov 03 20:30:02.387408 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:02.387394    5570 scope.go:122] "RemoveContainer" containerID="a779706851882c9381f962b5b7459ef2a5a7fe0cbc59b1c65e380186d5b184ad"
Nov 03 20:30:03.400270 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:03.400168    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d1bfb8-29895630-m6rrt" event={"ID":"7e73f70d-ab6c-4cc8-aff1-4d9833b5b991","Type":"ContainerStarted","Data":"9fd2123cca39d0925ffda113e83f55bc9e3d7572bae4164043482f80ac2d0442"}
Nov 03 20:30:03.402299 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:03.402264    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-17b3d9-29895630-btts9" event={"ID":"4c8a1c8b-4798-435a-a859-f6d1bc27efda","Type":"ContainerStarted","Data":"ed55a79301e63a765e0cc98f12f233acd532ceecc908376be464ed7a6b4d165c"}
Nov 03 20:30:03.404359 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:03.404329    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-482722-29895630-9pjh8" event={"ID":"36bf8581-603b-4a8d-8d7b-6da231cf7161","Type":"ContainerStarted","Data":"c6630c2e624055a7b0e840ab8ca899d5a6999fa0e557fc8a1254e56ee3842823"}
Nov 03 20:30:03.406492 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:03.406463    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-addf1a-29895630-btmk6" event={"ID":"4b3cfadc-bda0-4d0a-abff-056991a17a37","Type":"ContainerStarted","Data":"d922ff14989c4195be88eb688dc97063f64fd3115e69e56ba7c305be4b542127"}
Nov 03 20:30:03.408532 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:03.408511    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e40816-29895630-4n472" event={"ID":"11f669a0-4617-4c3e-96cc-fc288ef2aea1","Type":"ContainerStarted","Data":"4da39c147cb49c604398fb6d33112df3e7cfc29d40943e7357a680dac09002e6"}
Nov 03 20:30:03.410652 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:03.410624    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-b917c2-29895630-l49db" event={"ID":"c9cb6b5f-bf07-48ed-846a-eae8a3c19c1d","Type":"ContainerStarted","Data":"d7f0b476e96a84ba9a6c76c0b7f83997bec85cb3c3b6be593e69abe2b9a8d3bc"}
Nov 03 20:30:03.412454 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:03.412433    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-9e1bcb-29895630-tgcnz" event={"ID":"3b745fba-4594-4621-81bd-e322d6dd1efa","Type":"ContainerStarted","Data":"080916e31a6cc1fbff6fd197c71a326087e7cb9a497214125dcc3853c0ab2690"}
Nov 03 20:30:03.446289 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:03.446235    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d1bfb8-29895630-m6rrt" podStartSLOduration=3.4462202570000002 podStartE2EDuration="3.446220257s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:30:03.441280792 +0000 UTC m=+1537.593942682" watchObservedRunningTime="2026-11-03 20:30:03.446220257 +0000 UTC m=+1537.598882143"
Nov 03 20:30:03.482683 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:03.482623    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-addf1a-29895630-btmk6" podStartSLOduration=3.482605628 podStartE2EDuration="3.482605628s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:30:03.482317286 +0000 UTC m=+1537.634979170" watchObservedRunningTime="2026-11-03 20:30:03.482605628 +0000 UTC m=+1537.635267517"
Nov 03 20:30:03.523974 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:03.523919    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-17b3d9-29895630-btts9" podStartSLOduration=3.523904873 podStartE2EDuration="3.523904873s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:30:03.52099266 +0000 UTC m=+1537.673654546" watchObservedRunningTime="2026-11-03 20:30:03.523904873 +0000 UTC m=+1537.676566759"
Nov 03 20:30:03.652674 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:03.652539    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-482722-29895630-9pjh8" podStartSLOduration=3.652518898 podStartE2EDuration="3.652518898s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:30:03.648419699 +0000 UTC m=+1537.801081586" watchObservedRunningTime="2026-11-03 20:30:03.652518898 +0000 UTC m=+1537.805180787"
Nov 03 20:30:03.692321 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:03.692267    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e40816-29895630-4n472" podStartSLOduration=3.692250624 podStartE2EDuration="3.692250624s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:30:03.687844209 +0000 UTC m=+1537.840506096" watchObservedRunningTime="2026-11-03 20:30:03.692250624 +0000 UTC m=+1537.844912511"
Nov 03 20:30:04.510900 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:04.507643    5570 generic.go:356] "Generic (PLEG): container finished" podID="7e73f70d-ab6c-4cc8-aff1-4d9833b5b991" containerID="9fd2123cca39d0925ffda113e83f55bc9e3d7572bae4164043482f80ac2d0442" exitCode=60
Nov 03 20:30:04.510900 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:04.508009    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d1bfb8-29895630-m6rrt" event={"ID":"7e73f70d-ab6c-4cc8-aff1-4d9833b5b991","Type":"ContainerDied","Data":"9fd2123cca39d0925ffda113e83f55bc9e3d7572bae4164043482f80ac2d0442"}
Nov 03 20:30:04.510900 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:04.508076    5570 scope.go:122] "RemoveContainer" containerID="ba5586a0ffbfc2870393be764d0e9c339c790adb853ff08295c44d3cc044cc93"
Nov 03 20:30:04.515217 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:04.515190    5570 scope.go:122] "RemoveContainer" containerID="9fd2123cca39d0925ffda113e83f55bc9e3d7572bae4164043482f80ac2d0442"
Nov 03 20:30:04.515701 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:30:04.515667    5570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-d1bfb8-29895630-m6rrt_odh-ai-gateway-infra(7e73f70d-ab6c-4cc8-aff1-4d9833b5b991)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d1bfb8-29895630-m6rrt" podUID="7e73f70d-ab6c-4cc8-aff1-4d9833b5b991"
Nov 03 20:30:04.516981 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:04.516959    5570 generic.go:356] "Generic (PLEG): container finished" podID="4c8a1c8b-4798-435a-a859-f6d1bc27efda" containerID="ed55a79301e63a765e0cc98f12f233acd532ceecc908376be464ed7a6b4d165c" exitCode=60
Nov 03 20:30:04.517076 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:04.517061    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-17b3d9-29895630-btts9" event={"ID":"4c8a1c8b-4798-435a-a859-f6d1bc27efda","Type":"ContainerDied","Data":"ed55a79301e63a765e0cc98f12f233acd532ceecc908376be464ed7a6b4d165c"}
Nov 03 20:30:04.517542 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:04.517526    5570 scope.go:122] "RemoveContainer" containerID="ed55a79301e63a765e0cc98f12f233acd532ceecc908376be464ed7a6b4d165c"
Nov 03 20:30:04.517935 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:30:04.517915    5570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-17b3d9-29895630-btts9_odh-ai-gateway-infra(4c8a1c8b-4798-435a-a859-f6d1bc27efda)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-17b3d9-29895630-btts9" podUID="4c8a1c8b-4798-435a-a859-f6d1bc27efda"
Nov 03 20:30:04.520353 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:04.520324    5570 generic.go:356] "Generic (PLEG): container finished" podID="36bf8581-603b-4a8d-8d7b-6da231cf7161" containerID="c6630c2e624055a7b0e840ab8ca899d5a6999fa0e557fc8a1254e56ee3842823" exitCode=60
Nov 03 20:30:04.520470 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:04.520414    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-482722-29895630-9pjh8" event={"ID":"36bf8581-603b-4a8d-8d7b-6da231cf7161","Type":"ContainerDied","Data":"c6630c2e624055a7b0e840ab8ca899d5a6999fa0e557fc8a1254e56ee3842823"}
Nov 03 20:30:04.520762 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:04.520744    5570 scope.go:122] "RemoveContainer" containerID="c6630c2e624055a7b0e840ab8ca899d5a6999fa0e557fc8a1254e56ee3842823"
Nov 03 20:30:04.521186 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:30:04.521163    5570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-482722-29895630-9pjh8_odh-ai-gateway-infra(36bf8581-603b-4a8d-8d7b-6da231cf7161)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-482722-29895630-9pjh8" podUID="36bf8581-603b-4a8d-8d7b-6da231cf7161"
Nov 03 20:30:04.522821 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:04.522796    5570 generic.go:356] "Generic (PLEG): container finished" podID="4b3cfadc-bda0-4d0a-abff-056991a17a37" containerID="d922ff14989c4195be88eb688dc97063f64fd3115e69e56ba7c305be4b542127" exitCode=60
Nov 03 20:30:04.522930 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:04.522914    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-addf1a-29895630-btmk6" event={"ID":"4b3cfadc-bda0-4d0a-abff-056991a17a37","Type":"ContainerDied","Data":"d922ff14989c4195be88eb688dc97063f64fd3115e69e56ba7c305be4b542127"}
Nov 03 20:30:04.523234 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:04.523214    5570 scope.go:122] "RemoveContainer" containerID="d922ff14989c4195be88eb688dc97063f64fd3115e69e56ba7c305be4b542127"
Nov 03 20:30:04.523663 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:30:04.523637    5570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-addf1a-29895630-btmk6_odh-ai-gateway-infra(4b3cfadc-bda0-4d0a-abff-056991a17a37)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-addf1a-29895630-btmk6" podUID="4b3cfadc-bda0-4d0a-abff-056991a17a37"
Nov 03 20:30:04.524962 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:04.524938    5570 generic.go:356] "Generic (PLEG): container finished" podID="11f669a0-4617-4c3e-96cc-fc288ef2aea1" containerID="4da39c147cb49c604398fb6d33112df3e7cfc29d40943e7357a680dac09002e6" exitCode=60
Nov 03 20:30:04.525072 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:04.525031    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e40816-29895630-4n472" event={"ID":"11f669a0-4617-4c3e-96cc-fc288ef2aea1","Type":"ContainerDied","Data":"4da39c147cb49c604398fb6d33112df3e7cfc29d40943e7357a680dac09002e6"}
Nov 03 20:30:04.525346 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:04.525327    5570 scope.go:122] "RemoveContainer" containerID="4da39c147cb49c604398fb6d33112df3e7cfc29d40943e7357a680dac09002e6"
Nov 03 20:30:04.525736 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:30:04.525712    5570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-e40816-29895630-4n472_odh-ai-gateway-infra(11f669a0-4617-4c3e-96cc-fc288ef2aea1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e40816-29895630-4n472" podUID="11f669a0-4617-4c3e-96cc-fc288ef2aea1"
Nov 03 20:30:04.527352 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:04.527329    5570 generic.go:356] "Generic (PLEG): container finished" podID="c9cb6b5f-bf07-48ed-846a-eae8a3c19c1d" containerID="d7f0b476e96a84ba9a6c76c0b7f83997bec85cb3c3b6be593e69abe2b9a8d3bc" exitCode=60
Nov 03 20:30:04.527439 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:04.527409    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-b917c2-29895630-l49db" event={"ID":"c9cb6b5f-bf07-48ed-846a-eae8a3c19c1d","Type":"ContainerDied","Data":"d7f0b476e96a84ba9a6c76c0b7f83997bec85cb3c3b6be593e69abe2b9a8d3bc"}
Nov 03 20:30:04.527728 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:04.527708    5570 scope.go:122] "RemoveContainer" containerID="d7f0b476e96a84ba9a6c76c0b7f83997bec85cb3c3b6be593e69abe2b9a8d3bc"
Nov 03 20:30:04.528136 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:30:04.528112    5570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-b917c2-29895630-l49db_odh-ai-gateway-infra(c9cb6b5f-bf07-48ed-846a-eae8a3c19c1d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-b917c2-29895630-l49db" podUID="c9cb6b5f-bf07-48ed-846a-eae8a3c19c1d"
Nov 03 20:30:04.529889 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:04.529867    5570 generic.go:356] "Generic (PLEG): container finished" podID="3b745fba-4594-4621-81bd-e322d6dd1efa" containerID="080916e31a6cc1fbff6fd197c71a326087e7cb9a497214125dcc3853c0ab2690" exitCode=60
Nov 03 20:30:04.529978 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:04.529932    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-9e1bcb-29895630-tgcnz" event={"ID":"3b745fba-4594-4621-81bd-e322d6dd1efa","Type":"ContainerDied","Data":"080916e31a6cc1fbff6fd197c71a326087e7cb9a497214125dcc3853c0ab2690"}
Nov 03 20:30:04.530274 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:04.530259    5570 scope.go:122] "RemoveContainer" containerID="080916e31a6cc1fbff6fd197c71a326087e7cb9a497214125dcc3853c0ab2690"
Nov 03 20:30:04.530668 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:30:04.530646    5570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-9e1bcb-29895630-tgcnz_odh-ai-gateway-infra(3b745fba-4594-4621-81bd-e322d6dd1efa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-9e1bcb-29895630-tgcnz" podUID="3b745fba-4594-4621-81bd-e322d6dd1efa"
Nov 03 20:30:04.531019 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:04.530999    5570 scope.go:122] "RemoveContainer" containerID="11601267c1adb4f3d47827398cd66f607f981fe45f500a72a8c98be276b4e4a7"
Nov 03 20:30:04.546586 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:04.546568    5570 scope.go:122] "RemoveContainer" containerID="243b4d4d23db7ad36f7ca14b76e1573cbac46b724d52f137231f03e6177cae45"
Nov 03 20:30:04.564097 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:04.564072    5570 scope.go:122] "RemoveContainer" containerID="b6baf752e6a24af8613ca833bfd9367b19d880e8fcbf67373c5a69adcb8b2475"
Nov 03 20:30:04.581338 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:04.581308    5570 scope.go:122] "RemoveContainer" containerID="08645a3897f1706b5def167c003a4778ae412eaf2402b688d4b0ffefee9d50dd"
Nov 03 20:30:04.597523 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:04.597502    5570 scope.go:122] "RemoveContainer" containerID="a779706851882c9381f962b5b7459ef2a5a7fe0cbc59b1c65e380186d5b184ad"
Nov 03 20:30:04.612086 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:04.612063    5570 scope.go:122] "RemoveContainer" containerID="e480a298621964ac57cec51cfe2fb384bc6450e50ce11ce42a5cceb203562808"
Nov 03 20:30:04.918940 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:04.918845    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-eb87f429-7fbff5576b-89srx"]
Nov 03 20:30:04.949352 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:04.949319    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-eb87f429-6b49f7957f-bc4ph"]
Nov 03 20:30:05.539691 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:05.539664    5570 scope.go:122] "RemoveContainer" containerID="080916e31a6cc1fbff6fd197c71a326087e7cb9a497214125dcc3853c0ab2690"
Nov 03 20:30:05.540267 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:30:05.540179    5570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-9e1bcb-29895630-tgcnz_odh-ai-gateway-infra(3b745fba-4594-4621-81bd-e322d6dd1efa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-9e1bcb-29895630-tgcnz" podUID="3b745fba-4594-4621-81bd-e322d6dd1efa"
Nov 03 20:30:05.541512 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:05.541491    5570 scope.go:122] "RemoveContainer" containerID="9fd2123cca39d0925ffda113e83f55bc9e3d7572bae4164043482f80ac2d0442"
Nov 03 20:30:05.541914 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:30:05.541890    5570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-d1bfb8-29895630-m6rrt_odh-ai-gateway-infra(7e73f70d-ab6c-4cc8-aff1-4d9833b5b991)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d1bfb8-29895630-m6rrt" podUID="7e73f70d-ab6c-4cc8-aff1-4d9833b5b991"
Nov 03 20:30:05.543209 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:05.543189    5570 scope.go:122] "RemoveContainer" containerID="ed55a79301e63a765e0cc98f12f233acd532ceecc908376be464ed7a6b4d165c"
Nov 03 20:30:05.543535 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:30:05.543512    5570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-17b3d9-29895630-btts9_odh-ai-gateway-infra(4c8a1c8b-4798-435a-a859-f6d1bc27efda)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-17b3d9-29895630-btts9" podUID="4c8a1c8b-4798-435a-a859-f6d1bc27efda"
Nov 03 20:30:05.544818 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:05.544797    5570 scope.go:122] "RemoveContainer" containerID="c6630c2e624055a7b0e840ab8ca899d5a6999fa0e557fc8a1254e56ee3842823"
Nov 03 20:30:05.545131 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:30:05.545112    5570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-482722-29895630-9pjh8_odh-ai-gateway-infra(36bf8581-603b-4a8d-8d7b-6da231cf7161)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-482722-29895630-9pjh8" podUID="36bf8581-603b-4a8d-8d7b-6da231cf7161"
Nov 03 20:30:05.546352 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:05.546335    5570 scope.go:122] "RemoveContainer" containerID="d922ff14989c4195be88eb688dc97063f64fd3115e69e56ba7c305be4b542127"
Nov 03 20:30:05.546664 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:30:05.546644    5570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-addf1a-29895630-btmk6_odh-ai-gateway-infra(4b3cfadc-bda0-4d0a-abff-056991a17a37)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-addf1a-29895630-btmk6" podUID="4b3cfadc-bda0-4d0a-abff-056991a17a37"
Nov 03 20:30:05.548060 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:05.548042    5570 scope.go:122] "RemoveContainer" containerID="4da39c147cb49c604398fb6d33112df3e7cfc29d40943e7357a680dac09002e6"
Nov 03 20:30:05.548387 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:30:05.548362    5570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-e40816-29895630-4n472_odh-ai-gateway-infra(11f669a0-4617-4c3e-96cc-fc288ef2aea1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e40816-29895630-4n472" podUID="11f669a0-4617-4c3e-96cc-fc288ef2aea1"
Nov 03 20:30:05.549758 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:05.549741    5570 scope.go:122] "RemoveContainer" containerID="d7f0b476e96a84ba9a6c76c0b7f83997bec85cb3c3b6be593e69abe2b9a8d3bc"
Nov 03 20:30:05.550010 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:30:05.549993    5570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-b917c2-29895630-l49db_odh-ai-gateway-infra(c9cb6b5f-bf07-48ed-846a-eae8a3c19c1d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-b917c2-29895630-l49db" podUID="c9cb6b5f-bf07-48ed-846a-eae8a3c19c1d"
Nov 03 20:30:09.920306 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:09.920246    5570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-eb87f429-7fbff5576b-89srx" podUID="fc90ef38-bcf6-4df4-a07c-cb9726b731af" containerName="payload-processing" containerID="cri-o://0e5a3aee5347afddb2d3ee7cd1065d0920bc6a99ccab810d2b962744d9a11e49" gracePeriod=25
Nov 03 20:30:09.949763 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:09.949725    5570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-eb87f429-6b49f7957f-bc4ph" podUID="f578561f-2a3b-49cf-b7fa-3a22da0240b3" containerName="payload-pre-processing" containerID="cri-o://ec4d3c01e3b418bab5c1a403550ef75a801d9e4a3ec0caad7a7361b364f77f9c" gracePeriod=25
Nov 03 20:30:10.290101 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:10.290067    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-eb87f429-7fbff5576b-89srx"
Nov 03 20:30:10.295068 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:10.295040    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-eb87f429-6b49f7957f-bc4ph"
Nov 03 20:30:10.356613 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:10.356580    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f578561f-2a3b-49cf-b7fa-3a22da0240b3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f578561f-2a3b-49cf-b7fa-3a22da0240b3-plugins-config-volume\") pod \"f578561f-2a3b-49cf-b7fa-3a22da0240b3\" (UID: \"f578561f-2a3b-49cf-b7fa-3a22da0240b3\") "
Nov 03 20:30:10.356853 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:10.356769    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f578561f-2a3b-49cf-b7fa-3a22da0240b3-kube-api-access-bcc6z\" (UniqueName: \"kubernetes.io/projected/f578561f-2a3b-49cf-b7fa-3a22da0240b3-kube-api-access-bcc6z\") pod \"f578561f-2a3b-49cf-b7fa-3a22da0240b3\" (UID: \"f578561f-2a3b-49cf-b7fa-3a22da0240b3\") "
Nov 03 20:30:10.356989 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:10.356971    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc90ef38-bcf6-4df4-a07c-cb9726b731af-kube-api-access-frj95\" (UniqueName: \"kubernetes.io/projected/fc90ef38-bcf6-4df4-a07c-cb9726b731af-kube-api-access-frj95\") pod \"fc90ef38-bcf6-4df4-a07c-cb9726b731af\" (UID: \"fc90ef38-bcf6-4df4-a07c-cb9726b731af\") "
Nov 03 20:30:10.357110 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:10.357084    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f578561f-2a3b-49cf-b7fa-3a22da0240b3-plugins-config-volume" pod "f578561f-2a3b-49cf-b7fa-3a22da0240b3" (UID: "f578561f-2a3b-49cf-b7fa-3a22da0240b3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:30:10.357188 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:10.357170    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc90ef38-bcf6-4df4-a07c-cb9726b731af-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc90ef38-bcf6-4df4-a07c-cb9726b731af-plugins-config-volume\") pod \"fc90ef38-bcf6-4df4-a07c-cb9726b731af\" (UID: \"fc90ef38-bcf6-4df4-a07c-cb9726b731af\") "
Nov 03 20:30:10.357664 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:10.357640    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc90ef38-bcf6-4df4-a07c-cb9726b731af-plugins-config-volume" pod "fc90ef38-bcf6-4df4-a07c-cb9726b731af" (UID: "fc90ef38-bcf6-4df4-a07c-cb9726b731af"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:30:10.358051 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:10.358032    5570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f578561f-2a3b-49cf-b7fa-3a22da0240b3-plugins-config-volume\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:30:10.358138 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:10.358053    5570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc90ef38-bcf6-4df4-a07c-cb9726b731af-plugins-config-volume\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:30:10.359506 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:10.359481    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc90ef38-bcf6-4df4-a07c-cb9726b731af-kube-api-access-frj95" pod "fc90ef38-bcf6-4df4-a07c-cb9726b731af" (UID: "fc90ef38-bcf6-4df4-a07c-cb9726b731af"). InnerVolumeSpecName "kube-api-access-frj95". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:30:10.359886 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:10.359861    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f578561f-2a3b-49cf-b7fa-3a22da0240b3-kube-api-access-bcc6z" pod "f578561f-2a3b-49cf-b7fa-3a22da0240b3" (UID: "f578561f-2a3b-49cf-b7fa-3a22da0240b3"). InnerVolumeSpecName "kube-api-access-bcc6z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:30:10.460888 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:10.460852    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bcc6z\" (UniqueName: \"kubernetes.io/projected/f578561f-2a3b-49cf-b7fa-3a22da0240b3-kube-api-access-bcc6z\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:30:10.460888 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:10.460891    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frj95\" (UniqueName: \"kubernetes.io/projected/fc90ef38-bcf6-4df4-a07c-cb9726b731af-kube-api-access-frj95\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:30:10.674990 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:10.674897    5570 generic.go:356] "Generic (PLEG): container finished" podID="f578561f-2a3b-49cf-b7fa-3a22da0240b3" containerID="ec4d3c01e3b418bab5c1a403550ef75a801d9e4a3ec0caad7a7361b364f77f9c" exitCode=0
Nov 03 20:30:10.674990 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:10.674949    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-eb87f429-6b49f7957f-bc4ph"
Nov 03 20:30:10.674990 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:10.674974    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-eb87f429-6b49f7957f-bc4ph" event={"ID":"f578561f-2a3b-49cf-b7fa-3a22da0240b3","Type":"ContainerDied","Data":"ec4d3c01e3b418bab5c1a403550ef75a801d9e4a3ec0caad7a7361b364f77f9c"}
Nov 03 20:30:10.675264 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:10.675035    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-eb87f429-6b49f7957f-bc4ph" event={"ID":"f578561f-2a3b-49cf-b7fa-3a22da0240b3","Type":"ContainerDied","Data":"c7ad3a2e50b57f264536ccd3ee7d9c23b238355fadca444ed300b55439c0f95f"}
Nov 03 20:30:10.675264 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:10.675065    5570 scope.go:122] "RemoveContainer" containerID="ec4d3c01e3b418bab5c1a403550ef75a801d9e4a3ec0caad7a7361b364f77f9c"
Nov 03 20:30:10.676850 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:10.676825    5570 generic.go:356] "Generic (PLEG): container finished" podID="fc90ef38-bcf6-4df4-a07c-cb9726b731af" containerID="0e5a3aee5347afddb2d3ee7cd1065d0920bc6a99ccab810d2b962744d9a11e49" exitCode=0
Nov 03 20:30:10.676971 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:10.676872    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-eb87f429-7fbff5576b-89srx"
Nov 03 20:30:10.676971 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:10.676872    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-eb87f429-7fbff5576b-89srx" event={"ID":"fc90ef38-bcf6-4df4-a07c-cb9726b731af","Type":"ContainerDied","Data":"0e5a3aee5347afddb2d3ee7cd1065d0920bc6a99ccab810d2b962744d9a11e49"}
Nov 03 20:30:10.677091 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:10.677008    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-eb87f429-7fbff5576b-89srx" event={"ID":"fc90ef38-bcf6-4df4-a07c-cb9726b731af","Type":"ContainerDied","Data":"aa599f8f8e6611c1ad2f72df2a56964b147d3247d4e7088b7df25375a83d3cd0"}
Nov 03 20:30:10.687790 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:10.687726    5570 scope.go:122] "RemoveContainer" containerID="ec4d3c01e3b418bab5c1a403550ef75a801d9e4a3ec0caad7a7361b364f77f9c"
Nov 03 20:30:10.688245 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:30:10.688210    5570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec4d3c01e3b418bab5c1a403550ef75a801d9e4a3ec0caad7a7361b364f77f9c\": container with ID starting with ec4d3c01e3b418bab5c1a403550ef75a801d9e4a3ec0caad7a7361b364f77f9c not found: ID does not exist" containerID="ec4d3c01e3b418bab5c1a403550ef75a801d9e4a3ec0caad7a7361b364f77f9c"
Nov 03 20:30:10.688360 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:10.688248    5570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec4d3c01e3b418bab5c1a403550ef75a801d9e4a3ec0caad7a7361b364f77f9c"} err="failed to get container status \"ec4d3c01e3b418bab5c1a403550ef75a801d9e4a3ec0caad7a7361b364f77f9c\": rpc error: code = NotFound desc = could not find container \"ec4d3c01e3b418bab5c1a403550ef75a801d9e4a3ec0caad7a7361b364f77f9c\": container with ID starting with ec4d3c01e3b418bab5c1a403550ef75a801d9e4a3ec0caad7a7361b364f77f9c not found: ID does not exist"
Nov 03 20:30:10.688360 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:10.688273    5570 scope.go:122] "RemoveContainer" containerID="0e5a3aee5347afddb2d3ee7cd1065d0920bc6a99ccab810d2b962744d9a11e49"
Nov 03 20:30:10.699488 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:10.699465    5570 scope.go:122] "RemoveContainer" containerID="0e5a3aee5347afddb2d3ee7cd1065d0920bc6a99ccab810d2b962744d9a11e49"
Nov 03 20:30:10.699878 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:30:10.699851    5570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e5a3aee5347afddb2d3ee7cd1065d0920bc6a99ccab810d2b962744d9a11e49\": container with ID starting with 0e5a3aee5347afddb2d3ee7cd1065d0920bc6a99ccab810d2b962744d9a11e49 not found: ID does not exist" containerID="0e5a3aee5347afddb2d3ee7cd1065d0920bc6a99ccab810d2b962744d9a11e49"
Nov 03 20:30:10.699993 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:10.699884    5570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e5a3aee5347afddb2d3ee7cd1065d0920bc6a99ccab810d2b962744d9a11e49"} err="failed to get container status \"0e5a3aee5347afddb2d3ee7cd1065d0920bc6a99ccab810d2b962744d9a11e49\": rpc error: code = NotFound desc = could not find container \"0e5a3aee5347afddb2d3ee7cd1065d0920bc6a99ccab810d2b962744d9a11e49\": container with ID starting with 0e5a3aee5347afddb2d3ee7cd1065d0920bc6a99ccab810d2b962744d9a11e49 not found: ID does not exist"
Nov 03 20:30:10.731043 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:10.731003    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-eb87f429-6b49f7957f-bc4ph"]
Nov 03 20:30:10.744060 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:10.744030    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-eb87f429-6b49f7957f-bc4ph"]
Nov 03 20:30:10.780294 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:10.780257    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-eb87f429-7fbff5576b-89srx"]
Nov 03 20:30:10.785497 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:10.785455    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-eb87f429-7fbff5576b-89srx"]
Nov 03 20:30:11.969227 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:11.969192    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f578561f-2a3b-49cf-b7fa-3a22da0240b3" path="/var/lib/kubelet/pods/f578561f-2a3b-49cf-b7fa-3a22da0240b3/volumes"
Nov 03 20:30:11.969734 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:11.969719    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc90ef38-bcf6-4df4-a07c-cb9726b731af" path="/var/lib/kubelet/pods/fc90ef38-bcf6-4df4-a07c-cb9726b731af/volumes"
Nov 03 20:30:13.412798 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:13.412236    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g69f8"]
Nov 03 20:30:13.686688 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:13.686584    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a42b52f-j678v"]
Nov 03 20:30:13.687504 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:13.687480    5570 cpu_manager.go:414] "containerMap: removing container" podUID="fc90ef38-bcf6-4df4-a07c-cb9726b731af" containerName="payload-processing"
Nov 03 20:30:13.687504 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:13.687499    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc90ef38-bcf6-4df4-a07c-cb9726b731af" containerName="payload-processing"
Nov 03 20:30:13.687504 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:13.687511    5570 cpu_manager.go:414] "containerMap: removing container" podUID="f578561f-2a3b-49cf-b7fa-3a22da0240b3" containerName="payload-pre-processing"
Nov 03 20:30:13.687835 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:13.687517    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f578561f-2a3b-49cf-b7fa-3a22da0240b3" containerName="payload-pre-processing"
Nov 03 20:30:13.687835 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:13.687635    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc90ef38-bcf6-4df4-a07c-cb9726b731af" containerName="payload-processing"
Nov 03 20:30:13.687835 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:13.687645    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="f578561f-2a3b-49cf-b7fa-3a22da0240b3" containerName="payload-pre-processing"
Nov 03 20:30:13.693841 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:13.693815    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a42b52f-j678v"
Nov 03 20:30:13.708038 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:13.708006    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a42b52f-j678v"]
Nov 03 20:30:13.712729 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:13.712402    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:30:13.786756 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:13.786715    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/742793f0-4170-46dd-aa89-31df5f833c45-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0a42b52f-j678v\" (UID: \"742793f0-4170-46dd-aa89-31df5f833c45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a42b52f-j678v"
Nov 03 20:30:13.786944 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:13.786784    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndjwv\" (UniqueName: \"kubernetes.io/projected/742793f0-4170-46dd-aa89-31df5f833c45-kube-api-access-ndjwv\") pod \"maas-api-revoke-sub-0a42b52f-j678v\" (UID: \"742793f0-4170-46dd-aa89-31df5f833c45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a42b52f-j678v"
Nov 03 20:30:13.787275 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:13.787247    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/742793f0-4170-46dd-aa89-31df5f833c45-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0a42b52f-j678v\" (UID: \"742793f0-4170-46dd-aa89-31df5f833c45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a42b52f-j678v"
Nov 03 20:30:13.889143 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:13.889101    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/742793f0-4170-46dd-aa89-31df5f833c45-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0a42b52f-j678v\" (UID: \"742793f0-4170-46dd-aa89-31df5f833c45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a42b52f-j678v"
Nov 03 20:30:13.889309 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:13.889212    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/742793f0-4170-46dd-aa89-31df5f833c45-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0a42b52f-j678v\" (UID: \"742793f0-4170-46dd-aa89-31df5f833c45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a42b52f-j678v"
Nov 03 20:30:13.889309 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:13.889248    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndjwv\" (UniqueName: \"kubernetes.io/projected/742793f0-4170-46dd-aa89-31df5f833c45-kube-api-access-ndjwv\") pod \"maas-api-revoke-sub-0a42b52f-j678v\" (UID: \"742793f0-4170-46dd-aa89-31df5f833c45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a42b52f-j678v"
Nov 03 20:30:13.889975 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:13.889948    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/742793f0-4170-46dd-aa89-31df5f833c45-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0a42b52f-j678v\" (UID: \"742793f0-4170-46dd-aa89-31df5f833c45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a42b52f-j678v"
Nov 03 20:30:13.891567 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:13.891543    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/742793f0-4170-46dd-aa89-31df5f833c45-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0a42b52f-j678v\" (UID: \"742793f0-4170-46dd-aa89-31df5f833c45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a42b52f-j678v"
Nov 03 20:30:13.912655 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:13.912620    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndjwv\" (UniqueName: \"kubernetes.io/projected/742793f0-4170-46dd-aa89-31df5f833c45-kube-api-access-ndjwv\") pod \"maas-api-revoke-sub-0a42b52f-j678v\" (UID: \"742793f0-4170-46dd-aa89-31df5f833c45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a42b52f-j678v"
Nov 03 20:30:14.015359 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:14.015324    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a42b52f-j678v"
Nov 03 20:30:14.327253 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:14.327204    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a42b52f-j678v"]
Nov 03 20:30:14.327958 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:30:14.327926    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod742793f0_4170_46dd_aa89_31df5f833c45.slice/crio-2f1de6aade9a069c1ec5e25cea752d7cb1c51bf2189822b79fcefbbe88d7204e WatchSource:0}: Error finding container 2f1de6aade9a069c1ec5e25cea752d7cb1c51bf2189822b79fcefbbe88d7204e: Status 404 returned error can't find the container with id 2f1de6aade9a069c1ec5e25cea752d7cb1c51bf2189822b79fcefbbe88d7204e
Nov 03 20:30:14.851066 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:14.850967    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a42b52f-j678v" event={"ID":"742793f0-4170-46dd-aa89-31df5f833c45","Type":"ContainerStarted","Data":"2f1de6aade9a069c1ec5e25cea752d7cb1c51bf2189822b79fcefbbe88d7204e"}
Nov 03 20:30:14.963109 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:14.962858    5570 scope.go:122] "RemoveContainer" containerID="080916e31a6cc1fbff6fd197c71a326087e7cb9a497214125dcc3853c0ab2690"
Nov 03 20:30:15.868905 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:15.868805    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-9e1bcb-29895630-tgcnz" event={"ID":"3b745fba-4594-4621-81bd-e322d6dd1efa","Type":"ContainerStarted","Data":"d7b5cac572b05acb38461d99e96243c8cb7e79f2cca56ae1cb31adf85e26e864"}
Nov 03 20:30:15.870524 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:15.870495    5570 generic.go:356] "Generic (PLEG): container finished" podID="742793f0-4170-46dd-aa89-31df5f833c45" containerID="47143fc84098b81ae0a30abe3c762f33ac146636ea15729c1d08e8282b2558a4" exitCode=0
Nov 03 20:30:15.870609 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:15.870555    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a42b52f-j678v" event={"ID":"742793f0-4170-46dd-aa89-31df5f833c45","Type":"ContainerDied","Data":"47143fc84098b81ae0a30abe3c762f33ac146636ea15729c1d08e8282b2558a4"}
Nov 03 20:30:15.966846 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:15.966755    5570 scope.go:122] "RemoveContainer" containerID="9fd2123cca39d0925ffda113e83f55bc9e3d7572bae4164043482f80ac2d0442"
Nov 03 20:30:15.967990 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:15.967961    5570 scope.go:122] "RemoveContainer" containerID="d922ff14989c4195be88eb688dc97063f64fd3115e69e56ba7c305be4b542127"
Nov 03 20:30:15.968880 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:15.968856    5570 scope.go:122] "RemoveContainer" containerID="ed55a79301e63a765e0cc98f12f233acd532ceecc908376be464ed7a6b4d165c"
Nov 03 20:30:15.976493 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:15.976461    5570 scope.go:122] "RemoveContainer" containerID="d7f0b476e96a84ba9a6c76c0b7f83997bec85cb3c3b6be593e69abe2b9a8d3bc"
Nov 03 20:30:16.880302 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:16.880270    5570 generic.go:356] "Generic (PLEG): container finished" podID="3b745fba-4594-4621-81bd-e322d6dd1efa" containerID="d7b5cac572b05acb38461d99e96243c8cb7e79f2cca56ae1cb31adf85e26e864" exitCode=60
Nov 03 20:30:16.880672 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:16.880306    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-9e1bcb-29895630-tgcnz" event={"ID":"3b745fba-4594-4621-81bd-e322d6dd1efa","Type":"ContainerDied","Data":"d7b5cac572b05acb38461d99e96243c8cb7e79f2cca56ae1cb31adf85e26e864"}
Nov 03 20:30:16.880672 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:16.880366    5570 scope.go:122] "RemoveContainer" containerID="080916e31a6cc1fbff6fd197c71a326087e7cb9a497214125dcc3853c0ab2690"
Nov 03 20:30:16.880931 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:16.880914    5570 scope.go:122] "RemoveContainer" containerID="d7b5cac572b05acb38461d99e96243c8cb7e79f2cca56ae1cb31adf85e26e864"
Nov 03 20:30:16.881345 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:30:16.881325    5570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-9e1bcb-29895630-tgcnz_odh-ai-gateway-infra(3b745fba-4594-4621-81bd-e322d6dd1efa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-9e1bcb-29895630-tgcnz" podUID="3b745fba-4594-4621-81bd-e322d6dd1efa"
Nov 03 20:30:16.882599 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:16.882577    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d1bfb8-29895630-m6rrt" event={"ID":"7e73f70d-ab6c-4cc8-aff1-4d9833b5b991","Type":"ContainerStarted","Data":"e0c98815eed47d2c5819e36eb57fc8703007c406866ceba9016c1813f93fb8c0"}
Nov 03 20:30:16.885194 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:16.885174    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-17b3d9-29895630-btts9" event={"ID":"4c8a1c8b-4798-435a-a859-f6d1bc27efda","Type":"ContainerStarted","Data":"201d3509707f3b556f48395fb10fada6dd99b0a44f07afb4b86ddbf88804f86a"}
Nov 03 20:30:16.887409 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:16.887386    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-addf1a-29895630-btmk6" event={"ID":"4b3cfadc-bda0-4d0a-abff-056991a17a37","Type":"ContainerStarted","Data":"69b46b24b14e545ae26dd35ee7a13fd6b2c9060c039f7aaeb50cb7118e77406b"}
Nov 03 20:30:16.889738 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:16.889708    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-b917c2-29895630-l49db" event={"ID":"c9cb6b5f-bf07-48ed-846a-eae8a3c19c1d","Type":"ContainerStarted","Data":"2c50dc448abbffab7eb23791b6561791e14cf57af6c46b37bdf3d5a6e7d476e9"}
Nov 03 20:30:16.922981 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:16.922897    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-9e1bcb-29895630-tgcnz"]
Nov 03 20:30:16.969958 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:16.969914    5570 scope.go:122] "RemoveContainer" containerID="4da39c147cb49c604398fb6d33112df3e7cfc29d40943e7357a680dac09002e6"
Nov 03 20:30:17.199977 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:17.199948    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a42b52f-j678v"
Nov 03 20:30:17.305529 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:17.305493    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/742793f0-4170-46dd-aa89-31df5f833c45-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/742793f0-4170-46dd-aa89-31df5f833c45-maas-api-openshift-ca\") pod \"742793f0-4170-46dd-aa89-31df5f833c45\" (UID: \"742793f0-4170-46dd-aa89-31df5f833c45\") "
Nov 03 20:30:17.305733 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:17.305705    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/742793f0-4170-46dd-aa89-31df5f833c45-kube-api-access-ndjwv\" (UniqueName: \"kubernetes.io/projected/742793f0-4170-46dd-aa89-31df5f833c45-kube-api-access-ndjwv\") pod \"742793f0-4170-46dd-aa89-31df5f833c45\" (UID: \"742793f0-4170-46dd-aa89-31df5f833c45\") "
Nov 03 20:30:17.306157 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:17.306132    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/742793f0-4170-46dd-aa89-31df5f833c45-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/742793f0-4170-46dd-aa89-31df5f833c45-maas-api-xks-ca\") pod \"742793f0-4170-46dd-aa89-31df5f833c45\" (UID: \"742793f0-4170-46dd-aa89-31df5f833c45\") "
Nov 03 20:30:17.306505 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:17.306391    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/742793f0-4170-46dd-aa89-31df5f833c45-maas-api-openshift-ca" pod "742793f0-4170-46dd-aa89-31df5f833c45" (UID: "742793f0-4170-46dd-aa89-31df5f833c45"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:30:17.306978 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:17.306958    5570 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/742793f0-4170-46dd-aa89-31df5f833c45-maas-api-openshift-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:30:17.362813 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:17.360135    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/742793f0-4170-46dd-aa89-31df5f833c45-kube-api-access-ndjwv" pod "742793f0-4170-46dd-aa89-31df5f833c45" (UID: "742793f0-4170-46dd-aa89-31df5f833c45"). InnerVolumeSpecName "kube-api-access-ndjwv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:30:17.372169 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:17.372036    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/742793f0-4170-46dd-aa89-31df5f833c45-maas-api-xks-ca" pod "742793f0-4170-46dd-aa89-31df5f833c45" (UID: "742793f0-4170-46dd-aa89-31df5f833c45"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:30:17.408763 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:17.408711    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ndjwv\" (UniqueName: \"kubernetes.io/projected/742793f0-4170-46dd-aa89-31df5f833c45-kube-api-access-ndjwv\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:30:17.408763 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:17.408740    5570 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/742793f0-4170-46dd-aa89-31df5f833c45-maas-api-xks-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:30:17.501695 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:17.501657    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-eb87f429-59cb4f857f-xkbqm"]
Nov 03 20:30:17.502529 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:17.501960    5570 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-eb87f429-59cb4f857f-xkbqm" podUID="135caa8e-c038-4c19-8876-07c5fd69a4ae" containerName="maas-api" containerID="cri-o://222ea5a0c94594722028fb848dbba5a2c3839f4b32b00a775bae04ce4be7307b" gracePeriod=30
Nov 03 20:30:18.115796 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:18.110504    5570 scope.go:122] "RemoveContainer" containerID="c6630c2e624055a7b0e840ab8ca899d5a6999fa0e557fc8a1254e56ee3842823"
Nov 03 20:30:18.525539 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:18.525499    5570 generic.go:356] "Generic (PLEG): container finished" podID="4c8a1c8b-4798-435a-a859-f6d1bc27efda" containerID="201d3509707f3b556f48395fb10fada6dd99b0a44f07afb4b86ddbf88804f86a" exitCode=60
Nov 03 20:30:18.526735 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:18.526711    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-eb87f429-59cb4f857f-xkbqm"
Nov 03 20:30:18.539854 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:18.539818    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a42b52f-j678v"
Nov 03 20:30:18.564169 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:18.564126    5570 generic.go:356] "Generic (PLEG): container finished" podID="4b3cfadc-bda0-4d0a-abff-056991a17a37" containerID="69b46b24b14e545ae26dd35ee7a13fd6b2c9060c039f7aaeb50cb7118e77406b" exitCode=60
Nov 03 20:30:18.836194 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:18.836161    5570 generic.go:356] "Generic (PLEG): container finished" podID="c9cb6b5f-bf07-48ed-846a-eae8a3c19c1d" containerID="2c50dc448abbffab7eb23791b6561791e14cf57af6c46b37bdf3d5a6e7d476e9" exitCode=60
Nov 03 20:30:18.844164 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:18.844134    5570 generic.go:356] "Generic (PLEG): container finished" podID="135caa8e-c038-4c19-8876-07c5fd69a4ae" containerID="222ea5a0c94594722028fb848dbba5a2c3839f4b32b00a775bae04ce4be7307b" exitCode=0
Nov 03 20:30:18.844448 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:18.844430    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-eb87f429-59cb4f857f-xkbqm"
Nov 03 20:30:18.846676 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:18.846494    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d1bfb8-29895630-m6rrt"]
Nov 03 20:30:18.847620 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:18.846759    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-17b3d9-29895630-btts9" event={"ID":"4c8a1c8b-4798-435a-a859-f6d1bc27efda","Type":"ContainerDied","Data":"201d3509707f3b556f48395fb10fada6dd99b0a44f07afb4b86ddbf88804f86a"}
Nov 03 20:30:18.847728 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:18.847694    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a42b52f-j678v" event={"ID":"742793f0-4170-46dd-aa89-31df5f833c45","Type":"ContainerDied","Data":"2f1de6aade9a069c1ec5e25cea752d7cb1c51bf2189822b79fcefbbe88d7204e"}
Nov 03 20:30:18.847823 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:18.847806    5570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2f1de6aade9a069c1ec5e25cea752d7cb1c51bf2189822b79fcefbbe88d7204e"
Nov 03 20:30:18.847902 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:18.847835    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-addf1a-29895630-btmk6" event={"ID":"4b3cfadc-bda0-4d0a-abff-056991a17a37","Type":"ContainerDied","Data":"69b46b24b14e545ae26dd35ee7a13fd6b2c9060c039f7aaeb50cb7118e77406b"}
Nov 03 20:30:18.847970 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:18.847956    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e40816-29895630-4n472" event={"ID":"11f669a0-4617-4c3e-96cc-fc288ef2aea1","Type":"ContainerStarted","Data":"0b3ba4b58fceb90ea9991c905c2180bd4ce9e733ea05c51ab43880f768bd289f"}
Nov 03 20:30:18.848035 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:18.847986    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-b917c2-29895630-l49db" event={"ID":"c9cb6b5f-bf07-48ed-846a-eae8a3c19c1d","Type":"ContainerDied","Data":"2c50dc448abbffab7eb23791b6561791e14cf57af6c46b37bdf3d5a6e7d476e9"}
Nov 03 20:30:18.848122 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:18.848107    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-eb87f429-59cb4f857f-xkbqm" event={"ID":"135caa8e-c038-4c19-8876-07c5fd69a4ae","Type":"ContainerDied","Data":"222ea5a0c94594722028fb848dbba5a2c3839f4b32b00a775bae04ce4be7307b"}
Nov 03 20:30:18.851129 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:18.848978    5570 generic.go:356] "Generic (PLEG): container finished" podID="7e73f70d-ab6c-4cc8-aff1-4d9833b5b991" containerID="e0c98815eed47d2c5819e36eb57fc8703007c406866ceba9016c1813f93fb8c0" exitCode=60
Nov 03 20:30:18.851129 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:18.849049    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d1bfb8-29895630-m6rrt" event={"ID":"7e73f70d-ab6c-4cc8-aff1-4d9833b5b991","Type":"ContainerDied","Data":"e0c98815eed47d2c5819e36eb57fc8703007c406866ceba9016c1813f93fb8c0"}
Nov 03 20:30:18.851129 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:18.849432    5570 scope.go:122] "RemoveContainer" containerID="ed55a79301e63a765e0cc98f12f233acd532ceecc908376be464ed7a6b4d165c"
Nov 03 20:30:18.851129 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:18.849912    5570 scope.go:122] "RemoveContainer" containerID="201d3509707f3b556f48395fb10fada6dd99b0a44f07afb4b86ddbf88804f86a"
Nov 03 20:30:18.851129 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:30:18.850277    5570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-17b3d9-29895630-btts9_odh-ai-gateway-infra(4c8a1c8b-4798-435a-a859-f6d1bc27efda)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-17b3d9-29895630-btts9" podUID="4c8a1c8b-4798-435a-a859-f6d1bc27efda"
Nov 03 20:30:18.853513 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:18.852363    5570 scope.go:122] "RemoveContainer" containerID="69b46b24b14e545ae26dd35ee7a13fd6b2c9060c039f7aaeb50cb7118e77406b"
Nov 03 20:30:18.853513 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:30:18.852707    5570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-addf1a-29895630-btmk6_odh-ai-gateway-infra(4b3cfadc-bda0-4d0a-abff-056991a17a37)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-addf1a-29895630-btmk6" podUID="4b3cfadc-bda0-4d0a-abff-056991a17a37"
Nov 03 20:30:18.862388 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:18.862360    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-b917c2-29895630-l49db"]
Nov 03 20:30:18.877260 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:18.877224    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-addf1a-29895630-btmk6"]
Nov 03 20:30:18.885845 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:18.885818    5570 scope.go:122] "RemoveContainer" containerID="2c50dc448abbffab7eb23791b6561791e14cf57af6c46b37bdf3d5a6e7d476e9"
Nov 03 20:30:18.886534 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:30:18.886499    5570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-b917c2-29895630-l49db_odh-ai-gateway-infra(c9cb6b5f-bf07-48ed-846a-eae8a3c19c1d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-b917c2-29895630-l49db" podUID="c9cb6b5f-bf07-48ed-846a-eae8a3c19c1d"
Nov 03 20:30:18.890444 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:18.890351    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-17b3d9-29895630-btts9"]
Nov 03 20:30:18.903931 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:18.903840    5570 scope.go:122] "RemoveContainer" containerID="d922ff14989c4195be88eb688dc97063f64fd3115e69e56ba7c305be4b542127"
Nov 03 20:30:18.927065 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:18.927036    5570 scope.go:122] "RemoveContainer" containerID="d7f0b476e96a84ba9a6c76c0b7f83997bec85cb3c3b6be593e69abe2b9a8d3bc"
Nov 03 20:30:18.951817 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:18.949241    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/135caa8e-c038-4c19-8876-07c5fd69a4ae-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/135caa8e-c038-4c19-8876-07c5fd69a4ae-openshift-service-ca\") pod \"135caa8e-c038-4c19-8876-07c5fd69a4ae\" (UID: \"135caa8e-c038-4c19-8876-07c5fd69a4ae\") "
Nov 03 20:30:18.951817 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:18.949321    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/135caa8e-c038-4c19-8876-07c5fd69a4ae-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/135caa8e-c038-4c19-8876-07c5fd69a4ae-maas-api-tls\") pod \"135caa8e-c038-4c19-8876-07c5fd69a4ae\" (UID: \"135caa8e-c038-4c19-8876-07c5fd69a4ae\") "
Nov 03 20:30:18.951817 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:18.949372    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/135caa8e-c038-4c19-8876-07c5fd69a4ae-metrics-certs\" (UniqueName: \"kubernetes.io/secret/135caa8e-c038-4c19-8876-07c5fd69a4ae-metrics-certs\") pod \"135caa8e-c038-4c19-8876-07c5fd69a4ae\" (UID: \"135caa8e-c038-4c19-8876-07c5fd69a4ae\") "
Nov 03 20:30:18.951817 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:18.949558    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/135caa8e-c038-4c19-8876-07c5fd69a4ae-kube-api-access-m6f8t\" (UniqueName: \"kubernetes.io/projected/135caa8e-c038-4c19-8876-07c5fd69a4ae-kube-api-access-m6f8t\") pod \"135caa8e-c038-4c19-8876-07c5fd69a4ae\" (UID: \"135caa8e-c038-4c19-8876-07c5fd69a4ae\") "
Nov 03 20:30:18.952193 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:18.952015    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/135caa8e-c038-4c19-8876-07c5fd69a4ae-openshift-service-ca" pod "135caa8e-c038-4c19-8876-07c5fd69a4ae" (UID: "135caa8e-c038-4c19-8876-07c5fd69a4ae"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:30:18.965049 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:18.964996    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/135caa8e-c038-4c19-8876-07c5fd69a4ae-maas-api-tls" pod "135caa8e-c038-4c19-8876-07c5fd69a4ae" (UID: "135caa8e-c038-4c19-8876-07c5fd69a4ae"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:30:18.966259 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:18.966222    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/135caa8e-c038-4c19-8876-07c5fd69a4ae-metrics-certs" pod "135caa8e-c038-4c19-8876-07c5fd69a4ae" (UID: "135caa8e-c038-4c19-8876-07c5fd69a4ae"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:30:18.966730 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:18.966705    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/135caa8e-c038-4c19-8876-07c5fd69a4ae-kube-api-access-m6f8t" pod "135caa8e-c038-4c19-8876-07c5fd69a4ae" (UID: "135caa8e-c038-4c19-8876-07c5fd69a4ae"). InnerVolumeSpecName "kube-api-access-m6f8t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:30:19.018793 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:19.018637    5570 scope.go:122] "RemoveContainer" containerID="222ea5a0c94594722028fb848dbba5a2c3839f4b32b00a775bae04ce4be7307b"
Nov 03 20:30:19.073617 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:19.073585    5570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/135caa8e-c038-4c19-8876-07c5fd69a4ae-openshift-service-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:30:19.073916 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:19.073895    5570 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/135caa8e-c038-4c19-8876-07c5fd69a4ae-maas-api-tls\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:30:19.074046 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:19.074033    5570 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/135caa8e-c038-4c19-8876-07c5fd69a4ae-metrics-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:30:19.074140 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:19.074128    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6f8t\" (UniqueName: \"kubernetes.io/projected/135caa8e-c038-4c19-8876-07c5fd69a4ae-kube-api-access-m6f8t\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:30:19.076505 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:19.074559    5570 scope.go:122] "RemoveContainer" containerID="9fd2123cca39d0925ffda113e83f55bc9e3d7572bae4164043482f80ac2d0442"
Nov 03 20:30:19.122356 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:19.118689    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d1bfb8-29895630-m6rrt"
Nov 03 20:30:19.182765 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:19.175401    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e73f70d-ab6c-4cc8-aff1-4d9833b5b991-kube-api-access-4cfvx\" (UniqueName: \"kubernetes.io/projected/7e73f70d-ab6c-4cc8-aff1-4d9833b5b991-kube-api-access-4cfvx\") pod \"7e73f70d-ab6c-4cc8-aff1-4d9833b5b991\" (UID: \"7e73f70d-ab6c-4cc8-aff1-4d9833b5b991\") "
Nov 03 20:30:19.182765 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:19.175534    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e73f70d-ab6c-4cc8-aff1-4d9833b5b991-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e73f70d-ab6c-4cc8-aff1-4d9833b5b991-maas-api-service-ca\") pod \"7e73f70d-ab6c-4cc8-aff1-4d9833b5b991\" (UID: \"7e73f70d-ab6c-4cc8-aff1-4d9833b5b991\") "
Nov 03 20:30:19.182765 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:19.176723    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e73f70d-ab6c-4cc8-aff1-4d9833b5b991-maas-api-service-ca" pod "7e73f70d-ab6c-4cc8-aff1-4d9833b5b991" (UID: "7e73f70d-ab6c-4cc8-aff1-4d9833b5b991"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:30:19.196231 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:19.196195    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e73f70d-ab6c-4cc8-aff1-4d9833b5b991-kube-api-access-4cfvx" pod "7e73f70d-ab6c-4cc8-aff1-4d9833b5b991" (UID: "7e73f70d-ab6c-4cc8-aff1-4d9833b5b991"). InnerVolumeSpecName "kube-api-access-4cfvx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:30:19.276266 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:19.276233    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-9e1bcb-29895630-tgcnz"
Nov 03 20:30:19.278766 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:19.278740    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4cfvx\" (UniqueName: \"kubernetes.io/projected/7e73f70d-ab6c-4cc8-aff1-4d9833b5b991-kube-api-access-4cfvx\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:30:19.278863 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:19.278789    5570 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e73f70d-ab6c-4cc8-aff1-4d9833b5b991-maas-api-service-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:30:19.379695 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:19.379601    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b745fba-4594-4621-81bd-e322d6dd1efa-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b745fba-4594-4621-81bd-e322d6dd1efa-maas-api-service-ca\") pod \"3b745fba-4594-4621-81bd-e322d6dd1efa\" (UID: \"3b745fba-4594-4621-81bd-e322d6dd1efa\") "
Nov 03 20:30:19.379909 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:19.379697    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b745fba-4594-4621-81bd-e322d6dd1efa-kube-api-access-5hp9d\" (UniqueName: \"kubernetes.io/projected/3b745fba-4594-4621-81bd-e322d6dd1efa-kube-api-access-5hp9d\") pod \"3b745fba-4594-4621-81bd-e322d6dd1efa\" (UID: \"3b745fba-4594-4621-81bd-e322d6dd1efa\") "
Nov 03 20:30:19.381062 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:19.381029    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b745fba-4594-4621-81bd-e322d6dd1efa-maas-api-service-ca" pod "3b745fba-4594-4621-81bd-e322d6dd1efa" (UID: "3b745fba-4594-4621-81bd-e322d6dd1efa"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:30:19.384368 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:19.384280    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b745fba-4594-4621-81bd-e322d6dd1efa-kube-api-access-5hp9d" pod "3b745fba-4594-4621-81bd-e322d6dd1efa" (UID: "3b745fba-4594-4621-81bd-e322d6dd1efa"). InnerVolumeSpecName "kube-api-access-5hp9d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:30:19.406999 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:19.406963    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-eb87f429-59cb4f857f-xkbqm"]
Nov 03 20:30:19.411509 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:19.411481    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-eb87f429-59cb4f857f-xkbqm"]
Nov 03 20:30:19.481961 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:19.481929    5570 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b745fba-4594-4621-81bd-e322d6dd1efa-maas-api-service-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:30:19.481961 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:19.481957    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5hp9d\" (UniqueName: \"kubernetes.io/projected/3b745fba-4594-4621-81bd-e322d6dd1efa-kube-api-access-5hp9d\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:30:19.873837 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:19.873798    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-482722-29895630-9pjh8" event={"ID":"36bf8581-603b-4a8d-8d7b-6da231cf7161","Type":"ContainerStarted","Data":"548f0fdb61a0ab8a4f32c4239311e31a7264da7b7aab24053dcd3208682d1b5e"}
Nov 03 20:30:19.877738 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:19.877712    5570 generic.go:356] "Generic (PLEG): container finished" podID="11f669a0-4617-4c3e-96cc-fc288ef2aea1" containerID="0b3ba4b58fceb90ea9991c905c2180bd4ce9e733ea05c51ab43880f768bd289f" exitCode=60
Nov 03 20:30:19.877874 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:19.877734    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e40816-29895630-4n472" event={"ID":"11f669a0-4617-4c3e-96cc-fc288ef2aea1","Type":"ContainerDied","Data":"0b3ba4b58fceb90ea9991c905c2180bd4ce9e733ea05c51ab43880f768bd289f"}
Nov 03 20:30:19.877874 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:19.877818    5570 scope.go:122] "RemoveContainer" containerID="4da39c147cb49c604398fb6d33112df3e7cfc29d40943e7357a680dac09002e6"
Nov 03 20:30:19.878190 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:19.878176    5570 scope.go:122] "RemoveContainer" containerID="0b3ba4b58fceb90ea9991c905c2180bd4ce9e733ea05c51ab43880f768bd289f"
Nov 03 20:30:19.878626 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:30:19.878605    5570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-e40816-29895630-4n472_odh-ai-gateway-infra(11f669a0-4617-4c3e-96cc-fc288ef2aea1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e40816-29895630-4n472" podUID="11f669a0-4617-4c3e-96cc-fc288ef2aea1"
Nov 03 20:30:19.883855 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:19.883817    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-9e1bcb-29895630-tgcnz"
Nov 03 20:30:19.884034 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:19.884010    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-9e1bcb-29895630-tgcnz" event={"ID":"3b745fba-4594-4621-81bd-e322d6dd1efa","Type":"ContainerDied","Data":"51dac6a9c0ca4ae1694e5011176f43f90724a342109cb1a4dc0c2faa57635002"}
Nov 03 20:30:19.895627 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:19.893575    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d1bfb8-29895630-m6rrt" event={"ID":"7e73f70d-ab6c-4cc8-aff1-4d9833b5b991","Type":"ContainerDied","Data":"29513b3174e93d4d40415b3f04ea40422e13d3c6cceff46642ba6de36e27c611"}
Nov 03 20:30:19.895627 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:19.893682    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d1bfb8-29895630-m6rrt"
Nov 03 20:30:19.900292 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:19.900260    5570 scope.go:122] "RemoveContainer" containerID="d7b5cac572b05acb38461d99e96243c8cb7e79f2cca56ae1cb31adf85e26e864"
Nov 03 20:30:19.923104 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:19.923080    5570 scope.go:122] "RemoveContainer" containerID="e0c98815eed47d2c5819e36eb57fc8703007c406866ceba9016c1813f93fb8c0"
Nov 03 20:30:19.999838 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:19.997715    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="135caa8e-c038-4c19-8876-07c5fd69a4ae" path="/var/lib/kubelet/pods/135caa8e-c038-4c19-8876-07c5fd69a4ae/volumes"
Nov 03 20:30:20.084898 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:20.084856    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d1bfb8-29895630-m6rrt"]
Nov 03 20:30:20.085148 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:20.085135    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d1bfb8-29895630-m6rrt"]
Nov 03 20:30:20.099841 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:20.099100    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-9e1bcb-29895630-tgcnz"]
Nov 03 20:30:20.106848 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:20.106427    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-9e1bcb-29895630-tgcnz"]
Nov 03 20:30:20.133257 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:20.133226    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e40816-29895630-4n472"]
Nov 03 20:30:20.293717 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:20.293690    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-17b3d9-29895630-btts9"
Nov 03 20:30:20.297603 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:20.297581    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-b917c2-29895630-l49db"
Nov 03 20:30:20.301804 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:20.301783    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-addf1a-29895630-btmk6"
Nov 03 20:30:20.474797 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:20.474458    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c8a1c8b-4798-435a-a859-f6d1bc27efda-kube-api-access-228pp\" (UniqueName: \"kubernetes.io/projected/4c8a1c8b-4798-435a-a859-f6d1bc27efda-kube-api-access-228pp\") pod \"4c8a1c8b-4798-435a-a859-f6d1bc27efda\" (UID: \"4c8a1c8b-4798-435a-a859-f6d1bc27efda\") "
Nov 03 20:30:20.487794 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:20.475735    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b3cfadc-bda0-4d0a-abff-056991a17a37-kube-api-access-fkbjf\" (UniqueName: \"kubernetes.io/projected/4b3cfadc-bda0-4d0a-abff-056991a17a37-kube-api-access-fkbjf\") pod \"4b3cfadc-bda0-4d0a-abff-056991a17a37\" (UID: \"4b3cfadc-bda0-4d0a-abff-056991a17a37\") "
Nov 03 20:30:20.487794 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:20.475957    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9cb6b5f-bf07-48ed-846a-eae8a3c19c1d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9cb6b5f-bf07-48ed-846a-eae8a3c19c1d-maas-api-service-ca\") pod \"c9cb6b5f-bf07-48ed-846a-eae8a3c19c1d\" (UID: \"c9cb6b5f-bf07-48ed-846a-eae8a3c19c1d\") "
Nov 03 20:30:20.487794 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:20.476032    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9cb6b5f-bf07-48ed-846a-eae8a3c19c1d-kube-api-access-zst2g\" (UniqueName: \"kubernetes.io/projected/c9cb6b5f-bf07-48ed-846a-eae8a3c19c1d-kube-api-access-zst2g\") pod \"c9cb6b5f-bf07-48ed-846a-eae8a3c19c1d\" (UID: \"c9cb6b5f-bf07-48ed-846a-eae8a3c19c1d\") "
Nov 03 20:30:20.487794 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:20.476270    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b3cfadc-bda0-4d0a-abff-056991a17a37-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b3cfadc-bda0-4d0a-abff-056991a17a37-maas-api-service-ca\") pod \"4b3cfadc-bda0-4d0a-abff-056991a17a37\" (UID: \"4b3cfadc-bda0-4d0a-abff-056991a17a37\") "
Nov 03 20:30:20.487794 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:20.476313    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c8a1c8b-4798-435a-a859-f6d1bc27efda-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c8a1c8b-4798-435a-a859-f6d1bc27efda-maas-api-service-ca\") pod \"4c8a1c8b-4798-435a-a859-f6d1bc27efda\" (UID: \"4c8a1c8b-4798-435a-a859-f6d1bc27efda\") "
Nov 03 20:30:20.502973 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:20.499379    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c8a1c8b-4798-435a-a859-f6d1bc27efda-kube-api-access-228pp" pod "4c8a1c8b-4798-435a-a859-f6d1bc27efda" (UID: "4c8a1c8b-4798-435a-a859-f6d1bc27efda"). InnerVolumeSpecName "kube-api-access-228pp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:30:20.505837 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:20.504201    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b3cfadc-bda0-4d0a-abff-056991a17a37-kube-api-access-fkbjf" pod "4b3cfadc-bda0-4d0a-abff-056991a17a37" (UID: "4b3cfadc-bda0-4d0a-abff-056991a17a37"). InnerVolumeSpecName "kube-api-access-fkbjf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:30:20.522077 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:20.522039    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9cb6b5f-bf07-48ed-846a-eae8a3c19c1d-maas-api-service-ca" pod "c9cb6b5f-bf07-48ed-846a-eae8a3c19c1d" (UID: "c9cb6b5f-bf07-48ed-846a-eae8a3c19c1d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:30:20.529242 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:20.529202    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b3cfadc-bda0-4d0a-abff-056991a17a37-maas-api-service-ca" pod "4b3cfadc-bda0-4d0a-abff-056991a17a37" (UID: "4b3cfadc-bda0-4d0a-abff-056991a17a37"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:30:20.529607 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:20.529572    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c8a1c8b-4798-435a-a859-f6d1bc27efda-maas-api-service-ca" pod "4c8a1c8b-4798-435a-a859-f6d1bc27efda" (UID: "4c8a1c8b-4798-435a-a859-f6d1bc27efda"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:30:20.532394 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:20.532366    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9cb6b5f-bf07-48ed-846a-eae8a3c19c1d-kube-api-access-zst2g" pod "c9cb6b5f-bf07-48ed-846a-eae8a3c19c1d" (UID: "c9cb6b5f-bf07-48ed-846a-eae8a3c19c1d"). InnerVolumeSpecName "kube-api-access-zst2g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:30:20.624018 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:20.623980    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zst2g\" (UniqueName: \"kubernetes.io/projected/c9cb6b5f-bf07-48ed-846a-eae8a3c19c1d-kube-api-access-zst2g\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:30:20.624018 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:20.624011    5570 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b3cfadc-bda0-4d0a-abff-056991a17a37-maas-api-service-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:30:20.624018 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:20.624020    5570 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c8a1c8b-4798-435a-a859-f6d1bc27efda-maas-api-service-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:30:20.624018 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:20.624029    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-228pp\" (UniqueName: \"kubernetes.io/projected/4c8a1c8b-4798-435a-a859-f6d1bc27efda-kube-api-access-228pp\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:30:20.624303 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:20.624039    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fkbjf\" (UniqueName: \"kubernetes.io/projected/4b3cfadc-bda0-4d0a-abff-056991a17a37-kube-api-access-fkbjf\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:30:20.624303 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:20.624047    5570 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9cb6b5f-bf07-48ed-846a-eae8a3c19c1d-maas-api-service-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:30:20.905707 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:20.905252    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-addf1a-29895630-btmk6"
Nov 03 20:30:20.905707 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:20.905274    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-addf1a-29895630-btmk6" event={"ID":"4b3cfadc-bda0-4d0a-abff-056991a17a37","Type":"ContainerDied","Data":"88d15c9836555cd744dcd834adad28c9a90935230bd1e881218441127539e057"}
Nov 03 20:30:20.905707 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:20.905379    5570 scope.go:122] "RemoveContainer" containerID="69b46b24b14e545ae26dd35ee7a13fd6b2c9060c039f7aaeb50cb7118e77406b"
Nov 03 20:30:20.911286 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:20.911258    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-b917c2-29895630-l49db"
Nov 03 20:30:20.911286 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:20.911267    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-b917c2-29895630-l49db" event={"ID":"c9cb6b5f-bf07-48ed-846a-eae8a3c19c1d","Type":"ContainerDied","Data":"70dde93fc27bc8a4f513203ac8518673f6102a906660dbf048590a1ee3b2bd3a"}
Nov 03 20:30:20.917368 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:20.917341    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-17b3d9-29895630-btts9"
Nov 03 20:30:20.918298 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:20.917337    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-17b3d9-29895630-btts9" event={"ID":"4c8a1c8b-4798-435a-a859-f6d1bc27efda","Type":"ContainerDied","Data":"c36cb2ad9ee7211d545ca4aa6db77573d85573eae18b6a7cfca98a89e0b628d2"}
Nov 03 20:30:20.921046 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:20.921012    5570 generic.go:356] "Generic (PLEG): container finished" podID="36bf8581-603b-4a8d-8d7b-6da231cf7161" containerID="548f0fdb61a0ab8a4f32c4239311e31a7264da7b7aab24053dcd3208682d1b5e" exitCode=60
Nov 03 20:30:20.921203 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:20.921091    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-482722-29895630-9pjh8" event={"ID":"36bf8581-603b-4a8d-8d7b-6da231cf7161","Type":"ContainerDied","Data":"548f0fdb61a0ab8a4f32c4239311e31a7264da7b7aab24053dcd3208682d1b5e"}
Nov 03 20:30:20.921664 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:20.921640    5570 scope.go:122] "RemoveContainer" containerID="548f0fdb61a0ab8a4f32c4239311e31a7264da7b7aab24053dcd3208682d1b5e"
Nov 03 20:30:20.922196 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:30:20.922164    5570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-482722-29895630-9pjh8_odh-ai-gateway-infra(36bf8581-603b-4a8d-8d7b-6da231cf7161)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-482722-29895630-9pjh8" podUID="36bf8581-603b-4a8d-8d7b-6da231cf7161"
Nov 03 20:30:20.924465 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:20.924429    5570 scope.go:122] "RemoveContainer" containerID="2c50dc448abbffab7eb23791b6561791e14cf57af6c46b37bdf3d5a6e7d476e9"
Nov 03 20:30:20.945843 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:20.945818    5570 scope.go:122] "RemoveContainer" containerID="201d3509707f3b556f48395fb10fada6dd99b0a44f07afb4b86ddbf88804f86a"
Nov 03 20:30:20.962230 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:20.961964    5570 scope.go:122] "RemoveContainer" containerID="c6630c2e624055a7b0e840ab8ca899d5a6999fa0e557fc8a1254e56ee3842823"
Nov 03 20:30:20.974054 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:20.974027    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-addf1a-29895630-btmk6"]
Nov 03 20:30:20.980545 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:20.980515    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-addf1a-29895630-btmk6"]
Nov 03 20:30:21.100578 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:21.100539    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e40816-29895630-4n472"
Nov 03 20:30:21.137728 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:21.137693    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-b917c2-29895630-l49db"]
Nov 03 20:30:21.145257 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:21.145143    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-b917c2-29895630-l49db"]
Nov 03 20:30:21.186823 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:21.186756    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-17b3d9-29895630-btts9"]
Nov 03 20:30:21.192627 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:21.192590    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-17b3d9-29895630-btts9"]
Nov 03 20:30:21.233982 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:21.233946    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-482722-29895630-9pjh8"]
Nov 03 20:30:21.236373 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:21.236336    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11f669a0-4617-4c3e-96cc-fc288ef2aea1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/11f669a0-4617-4c3e-96cc-fc288ef2aea1-maas-api-service-ca\") pod \"11f669a0-4617-4c3e-96cc-fc288ef2aea1\" (UID: \"11f669a0-4617-4c3e-96cc-fc288ef2aea1\") "
Nov 03 20:30:21.236840 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:21.236814    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11f669a0-4617-4c3e-96cc-fc288ef2aea1-kube-api-access-8ksn9\" (UniqueName: \"kubernetes.io/projected/11f669a0-4617-4c3e-96cc-fc288ef2aea1-kube-api-access-8ksn9\") pod \"11f669a0-4617-4c3e-96cc-fc288ef2aea1\" (UID: \"11f669a0-4617-4c3e-96cc-fc288ef2aea1\") "
Nov 03 20:30:21.241692 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:21.241199    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11f669a0-4617-4c3e-96cc-fc288ef2aea1-maas-api-service-ca" pod "11f669a0-4617-4c3e-96cc-fc288ef2aea1" (UID: "11f669a0-4617-4c3e-96cc-fc288ef2aea1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:30:21.254932 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:21.254885    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11f669a0-4617-4c3e-96cc-fc288ef2aea1-kube-api-access-8ksn9" pod "11f669a0-4617-4c3e-96cc-fc288ef2aea1" (UID: "11f669a0-4617-4c3e-96cc-fc288ef2aea1"). InnerVolumeSpecName "kube-api-access-8ksn9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:30:21.338930 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:21.338899    5570 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/11f669a0-4617-4c3e-96cc-fc288ef2aea1-maas-api-service-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:30:21.338930 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:21.338929    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ksn9\" (UniqueName: \"kubernetes.io/projected/11f669a0-4617-4c3e-96cc-fc288ef2aea1-kube-api-access-8ksn9\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:30:21.489656 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:21.489604    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-mt-eb87f429-59cb4f857f-xkbqm" podUID="135caa8e-c038-4c19-8876-07c5fd69a4ae" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.135:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:30:21.936139 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:21.936114    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e40816-29895630-4n472"
Nov 03 20:30:21.938094 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:21.937852    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e40816-29895630-4n472" event={"ID":"11f669a0-4617-4c3e-96cc-fc288ef2aea1","Type":"ContainerDied","Data":"eadbc6fed59cd16b42e9ed22f65dd7157da701ea6580ff1693f4a2765f5f607f"}
Nov 03 20:30:21.938094 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:21.937935    5570 scope.go:122] "RemoveContainer" containerID="0b3ba4b58fceb90ea9991c905c2180bd4ce9e733ea05c51ab43880f768bd289f"
Nov 03 20:30:21.968607 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:21.968549    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b745fba-4594-4621-81bd-e322d6dd1efa" path="/var/lib/kubelet/pods/3b745fba-4594-4621-81bd-e322d6dd1efa/volumes"
Nov 03 20:30:21.969291 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:21.969270    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b3cfadc-bda0-4d0a-abff-056991a17a37" path="/var/lib/kubelet/pods/4b3cfadc-bda0-4d0a-abff-056991a17a37/volumes"
Nov 03 20:30:21.970053 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:21.969888    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c8a1c8b-4798-435a-a859-f6d1bc27efda" path="/var/lib/kubelet/pods/4c8a1c8b-4798-435a-a859-f6d1bc27efda/volumes"
Nov 03 20:30:21.970707 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:21.970689    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e73f70d-ab6c-4cc8-aff1-4d9833b5b991" path="/var/lib/kubelet/pods/7e73f70d-ab6c-4cc8-aff1-4d9833b5b991/volumes"
Nov 03 20:30:21.971205 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:21.971188    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9cb6b5f-bf07-48ed-846a-eae8a3c19c1d" path="/var/lib/kubelet/pods/c9cb6b5f-bf07-48ed-846a-eae8a3c19c1d/volumes"
Nov 03 20:30:22.023840 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:22.022810    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e40816-29895630-4n472"]
Nov 03 20:30:22.046575 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:22.046524    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e40816-29895630-4n472"]
Nov 03 20:30:22.316087 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:22.316052    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-482722-29895630-9pjh8"
Nov 03 20:30:22.406406 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:22.406362    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36bf8581-603b-4a8d-8d7b-6da231cf7161-kube-api-access-spsw6\" (UniqueName: \"kubernetes.io/projected/36bf8581-603b-4a8d-8d7b-6da231cf7161-kube-api-access-spsw6\") pod \"36bf8581-603b-4a8d-8d7b-6da231cf7161\" (UID: \"36bf8581-603b-4a8d-8d7b-6da231cf7161\") "
Nov 03 20:30:22.406618 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:22.406548    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36bf8581-603b-4a8d-8d7b-6da231cf7161-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/36bf8581-603b-4a8d-8d7b-6da231cf7161-maas-api-service-ca\") pod \"36bf8581-603b-4a8d-8d7b-6da231cf7161\" (UID: \"36bf8581-603b-4a8d-8d7b-6da231cf7161\") "
Nov 03 20:30:22.408420 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:22.408375    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36bf8581-603b-4a8d-8d7b-6da231cf7161-maas-api-service-ca" pod "36bf8581-603b-4a8d-8d7b-6da231cf7161" (UID: "36bf8581-603b-4a8d-8d7b-6da231cf7161"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:30:22.414314 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:22.414271    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36bf8581-603b-4a8d-8d7b-6da231cf7161-kube-api-access-spsw6" pod "36bf8581-603b-4a8d-8d7b-6da231cf7161" (UID: "36bf8581-603b-4a8d-8d7b-6da231cf7161"). InnerVolumeSpecName "kube-api-access-spsw6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:30:22.509815 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:22.509707    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-spsw6\" (UniqueName: \"kubernetes.io/projected/36bf8581-603b-4a8d-8d7b-6da231cf7161-kube-api-access-spsw6\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:30:22.509815 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:22.509748    5570 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/36bf8581-603b-4a8d-8d7b-6da231cf7161-maas-api-service-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:30:22.951071 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:22.951037    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-482722-29895630-9pjh8"
Nov 03 20:30:22.951259 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:22.951044    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-482722-29895630-9pjh8" event={"ID":"36bf8581-603b-4a8d-8d7b-6da231cf7161","Type":"ContainerDied","Data":"4e0344c18c145edeedad4b156e5c97da12b8ef432d60d52f49b3252dbfc6416c"}
Nov 03 20:30:22.951259 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:22.951219    5570 scope.go:122] "RemoveContainer" containerID="548f0fdb61a0ab8a4f32c4239311e31a7264da7b7aab24053dcd3208682d1b5e"
Nov 03 20:30:23.008592 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:23.008554    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-482722-29895630-9pjh8"]
Nov 03 20:30:23.022293 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:23.022251    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-482722-29895630-9pjh8"]
Nov 03 20:30:24.010054 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:24.010019    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11f669a0-4617-4c3e-96cc-fc288ef2aea1" path="/var/lib/kubelet/pods/11f669a0-4617-4c3e-96cc-fc288ef2aea1/volumes"
Nov 03 20:30:24.010564 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:24.010548    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36bf8581-603b-4a8d-8d7b-6da231cf7161" path="/var/lib/kubelet/pods/36bf8581-603b-4a8d-8d7b-6da231cf7161/volumes"
Nov 03 20:30:25.997233 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:25.997197    5570 generic.go:356] "Generic (PLEG): container finished" podID="46af9c86-b058-4323-97a2-af8d89d3dc8f" containerID="3fef9e5c8d3757cb3a7f4684106e6b30c1b1f324ad8bb942240b0cf6e9a436d8" exitCode=0
Nov 03 20:30:25.997679 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:25.997269    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-fab871-openshift-default-698b94dc5f-l6ggn" event={"ID":"46af9c86-b058-4323-97a2-af8d89d3dc8f","Type":"ContainerDied","Data":"3fef9e5c8d3757cb3a7f4684106e6b30c1b1f324ad8bb942240b0cf6e9a436d8"}
Nov 03 20:30:25.997755 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:25.997740    5570 scope.go:122] "RemoveContainer" containerID="3fef9e5c8d3757cb3a7f4684106e6b30c1b1f324ad8bb942240b0cf6e9a436d8"
Nov 03 20:30:25.999545 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:25.999518    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:30:25.999620 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:25.999588    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:30:25.999658 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:25.999619    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:30:27.008111 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:27.008072    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-fab871-openshift-default-698b94dc5f-l6ggn" event={"ID":"46af9c86-b058-4323-97a2-af8d89d3dc8f","Type":"ContainerStarted","Data":"a80861982cfdef69f6febbf07cdbb95484719e8d26e334dfd35ae52132567687"}
Nov 03 20:30:27.255429 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:27.255392    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-fab871-openshift-default-698b94dc5f-l6ggn"
Nov 03 20:30:27.258272 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:27.257834    5570 patch_prober.go:29] interesting pod/e2e-api-a-fab871-openshift-default-698b94dc5f-l6ggn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body=
Nov 03 20:30:27.258272 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:27.257914    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-fab871-openshift-default-698b94dc5f-l6ggn" podUID="46af9c86-b058-4323-97a2-af8d89d3dc8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused"
Nov 03 20:30:27.275202 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:30:27.275167    5570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8bwhz" containerName="extractor"
Nov 03 20:30:28.262451 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:28.262403    5570 patch_prober.go:29] interesting pod/e2e-api-a-fab871-openshift-default-698b94dc5f-l6ggn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body=
Nov 03 20:30:28.263038 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:28.262474    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-fab871-openshift-default-698b94dc5f-l6ggn" podUID="46af9c86-b058-4323-97a2-af8d89d3dc8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused"
Nov 03 20:30:29.263206 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:29.262527    5570 patch_prober.go:29] interesting pod/e2e-api-a-fab871-openshift-default-698b94dc5f-l6ggn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body=
Nov 03 20:30:29.263206 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:29.262592    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-fab871-openshift-default-698b94dc5f-l6ggn" podUID="46af9c86-b058-4323-97a2-af8d89d3dc8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused"
Nov 03 20:30:30.420680 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:30.420523    5570 patch_prober.go:29] interesting pod/e2e-api-a-fab871-openshift-default-698b94dc5f-l6ggn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body=
Nov 03 20:30:30.420680 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:30.420619    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-fab871-openshift-default-698b94dc5f-l6ggn" podUID="46af9c86-b058-4323-97a2-af8d89d3dc8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused"
Nov 03 20:30:31.259800 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:31.258932    5570 patch_prober.go:29] interesting pod/e2e-api-a-fab871-openshift-default-698b94dc5f-l6ggn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body=
Nov 03 20:30:31.259800 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:31.259012    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-fab871-openshift-default-698b94dc5f-l6ggn" podUID="46af9c86-b058-4323-97a2-af8d89d3dc8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused"
Nov 03 20:30:31.522865 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:31.517231    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-fab871-openshift-default-698b94dc5f-l6ggn"
Nov 03 20:30:32.270797 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:32.270728    5570 patch_prober.go:29] interesting pod/e2e-api-a-fab871-openshift-default-698b94dc5f-l6ggn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body=
Nov 03 20:30:32.271021 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:32.270873    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-fab871-openshift-default-698b94dc5f-l6ggn" podUID="46af9c86-b058-4323-97a2-af8d89d3dc8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused"
Nov 03 20:30:33.369082 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:33.369041    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-fab871-openshift-default-698b94dc5f-l6ggn"
Nov 03 20:30:33.370583 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:33.370555    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-fab871-openshift-default-698b94dc5f-l6ggn"
Nov 03 20:30:35.506828 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:35.506743    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-d05604-openshift-default-6bdffb549c-w62g5"]
Nov 03 20:30:35.507630 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:35.507610    5570 cpu_manager.go:414] "containerMap: removing container" podUID="c9cb6b5f-bf07-48ed-846a-eae8a3c19c1d" containerName="cleanup"
Nov 03 20:30:35.507630 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:35.507630    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9cb6b5f-bf07-48ed-846a-eae8a3c19c1d" containerName="cleanup"
Nov 03 20:30:35.507892 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:35.507645    5570 cpu_manager.go:414] "containerMap: removing container" podUID="4c8a1c8b-4798-435a-a859-f6d1bc27efda" containerName="cleanup"
Nov 03 20:30:35.507892 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:35.507654    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c8a1c8b-4798-435a-a859-f6d1bc27efda" containerName="cleanup"
Nov 03 20:30:35.507892 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:35.507666    5570 cpu_manager.go:414] "containerMap: removing container" podUID="4c8a1c8b-4798-435a-a859-f6d1bc27efda" containerName="cleanup"
Nov 03 20:30:35.507892 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:35.507671    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c8a1c8b-4798-435a-a859-f6d1bc27efda" containerName="cleanup"
Nov 03 20:30:35.507892 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:35.507680    5570 cpu_manager.go:414] "containerMap: removing container" podUID="11f669a0-4617-4c3e-96cc-fc288ef2aea1" containerName="cleanup"
Nov 03 20:30:35.507892 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:35.507684    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11f669a0-4617-4c3e-96cc-fc288ef2aea1" containerName="cleanup"
Nov 03 20:30:35.507892 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:35.507693    5570 cpu_manager.go:414] "containerMap: removing container" podUID="4b3cfadc-bda0-4d0a-abff-056991a17a37" containerName="cleanup"
Nov 03 20:30:35.507892 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:35.507698    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b3cfadc-bda0-4d0a-abff-056991a17a37" containerName="cleanup"
Nov 03 20:30:35.507892 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:35.507706    5570 cpu_manager.go:414] "containerMap: removing container" podUID="4c8a1c8b-4798-435a-a859-f6d1bc27efda" containerName="cleanup"
Nov 03 20:30:35.507892 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:35.507711    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c8a1c8b-4798-435a-a859-f6d1bc27efda" containerName="cleanup"
Nov 03 20:30:35.507892 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:35.507727    5570 cpu_manager.go:414] "containerMap: removing container" podUID="742793f0-4170-46dd-aa89-31df5f833c45" containerName="revoke-keys"
Nov 03 20:30:35.507892 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:35.507732    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="742793f0-4170-46dd-aa89-31df5f833c45" containerName="revoke-keys"
Nov 03 20:30:35.507892 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:35.507740    5570 cpu_manager.go:414] "containerMap: removing container" podUID="4b3cfadc-bda0-4d0a-abff-056991a17a37" containerName="cleanup"
Nov 03 20:30:35.507892 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:35.507746    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b3cfadc-bda0-4d0a-abff-056991a17a37" containerName="cleanup"
Nov 03 20:30:35.507892 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:35.507753    5570 cpu_manager.go:414] "containerMap: removing container" podUID="135caa8e-c038-4c19-8876-07c5fd69a4ae" containerName="maas-api"
Nov 03 20:30:35.507892 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:35.507757    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="135caa8e-c038-4c19-8876-07c5fd69a4ae" containerName="maas-api"
Nov 03 20:30:35.507892 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:35.507765    5570 cpu_manager.go:414] "containerMap: removing container" podUID="c9cb6b5f-bf07-48ed-846a-eae8a3c19c1d" containerName="cleanup"
Nov 03 20:30:35.507892 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:35.507785    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9cb6b5f-bf07-48ed-846a-eae8a3c19c1d" containerName="cleanup"
Nov 03 20:30:35.507892 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:35.507799    5570 cpu_manager.go:414] "containerMap: removing container" podUID="3b745fba-4594-4621-81bd-e322d6dd1efa" containerName="cleanup"
Nov 03 20:30:35.507892 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:35.507804    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b745fba-4594-4621-81bd-e322d6dd1efa" containerName="cleanup"
Nov 03 20:30:35.507892 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:35.507819    5570 cpu_manager.go:414] "containerMap: removing container" podUID="36bf8581-603b-4a8d-8d7b-6da231cf7161" containerName="cleanup"
Nov 03 20:30:35.507892 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:35.507823    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36bf8581-603b-4a8d-8d7b-6da231cf7161" containerName="cleanup"
Nov 03 20:30:35.507892 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:35.507840    5570 cpu_manager.go:414] "containerMap: removing container" podUID="7e73f70d-ab6c-4cc8-aff1-4d9833b5b991" containerName="cleanup"
Nov 03 20:30:35.507892 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:35.507846    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e73f70d-ab6c-4cc8-aff1-4d9833b5b991" containerName="cleanup"
Nov 03 20:30:35.507892 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:35.507852    5570 cpu_manager.go:414] "containerMap: removing container" podUID="3b745fba-4594-4621-81bd-e322d6dd1efa" containerName="cleanup"
Nov 03 20:30:35.507892 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:35.507859    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b745fba-4594-4621-81bd-e322d6dd1efa" containerName="cleanup"
Nov 03 20:30:35.507892 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:35.507875    5570 cpu_manager.go:414] "containerMap: removing container" podUID="36bf8581-603b-4a8d-8d7b-6da231cf7161" containerName="cleanup"
Nov 03 20:30:35.507892 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:35.507880    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36bf8581-603b-4a8d-8d7b-6da231cf7161" containerName="cleanup"
Nov 03 20:30:35.507892 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:35.507888    5570 cpu_manager.go:414] "containerMap: removing container" podUID="c9cb6b5f-bf07-48ed-846a-eae8a3c19c1d" containerName="cleanup"
Nov 03 20:30:35.507892 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:35.507893    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9cb6b5f-bf07-48ed-846a-eae8a3c19c1d" containerName="cleanup"
Nov 03 20:30:35.507892 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:35.507905    5570 cpu_manager.go:414] "containerMap: removing container" podUID="7e73f70d-ab6c-4cc8-aff1-4d9833b5b991" containerName="cleanup"
Nov 03 20:30:35.509523 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:35.507952    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e73f70d-ab6c-4cc8-aff1-4d9833b5b991" containerName="cleanup"
Nov 03 20:30:35.509523 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:35.507964    5570 cpu_manager.go:414] "containerMap: removing container" podUID="11f669a0-4617-4c3e-96cc-fc288ef2aea1" containerName="cleanup"
Nov 03 20:30:35.509523 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:35.507970    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11f669a0-4617-4c3e-96cc-fc288ef2aea1" containerName="cleanup"
Nov 03 20:30:35.509523 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:35.508175    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9cb6b5f-bf07-48ed-846a-eae8a3c19c1d" containerName="cleanup"
Nov 03 20:30:35.509523 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:35.508190    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c8a1c8b-4798-435a-a859-f6d1bc27efda" containerName="cleanup"
Nov 03 20:30:35.509523 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:35.508199    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c8a1c8b-4798-435a-a859-f6d1bc27efda" containerName="cleanup"
Nov 03 20:30:35.509523 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:35.508212    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="11f669a0-4617-4c3e-96cc-fc288ef2aea1" containerName="cleanup"
Nov 03 20:30:35.509523 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:35.508221    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b3cfadc-bda0-4d0a-abff-056991a17a37" containerName="cleanup"
Nov 03 20:30:35.509523 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:35.508232    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c8a1c8b-4798-435a-a859-f6d1bc27efda" containerName="cleanup"
Nov 03 20:30:35.509523 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:35.508244    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="742793f0-4170-46dd-aa89-31df5f833c45" containerName="revoke-keys"
Nov 03 20:30:35.509523 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:35.508252    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b3cfadc-bda0-4d0a-abff-056991a17a37" containerName="cleanup"
Nov 03 20:30:35.509523 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:35.508260    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="135caa8e-c038-4c19-8876-07c5fd69a4ae" containerName="maas-api"
Nov 03 20:30:35.509523 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:35.508268    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9cb6b5f-bf07-48ed-846a-eae8a3c19c1d" containerName="cleanup"
Nov 03 20:30:35.509523 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:35.508277    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b745fba-4594-4621-81bd-e322d6dd1efa" containerName="cleanup"
Nov 03 20:30:35.509523 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:35.508286    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="36bf8581-603b-4a8d-8d7b-6da231cf7161" containerName="cleanup"
Nov 03 20:30:35.509523 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:35.508297    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e73f70d-ab6c-4cc8-aff1-4d9833b5b991" containerName="cleanup"
Nov 03 20:30:35.509523 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:35.508305    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b745fba-4594-4621-81bd-e322d6dd1efa" containerName="cleanup"
Nov 03 20:30:35.509523 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:35.508316    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="36bf8581-603b-4a8d-8d7b-6da231cf7161" containerName="cleanup"
Nov 03 20:30:35.509523 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:35.508326    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e73f70d-ab6c-4cc8-aff1-4d9833b5b991" containerName="cleanup"
Nov 03 20:30:35.509523 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:35.508333    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e73f70d-ab6c-4cc8-aff1-4d9833b5b991" containerName="cleanup"
Nov 03 20:30:35.509523 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:35.508344    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="36bf8581-603b-4a8d-8d7b-6da231cf7161" containerName="cleanup"
Nov 03 20:30:35.509523 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:35.508353    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="11f669a0-4617-4c3e-96cc-fc288ef2aea1" containerName="cleanup"
Nov 03 20:30:35.513015 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:35.512987    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d05604-openshift-default-6bdffb549c-w62g5"
Nov 03 20:30:35.529915 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:35.529881    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-d05604-openshift-default-6bdffb549c-w62g5"]
Nov 03 20:30:35.539202 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:35.539138    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-d05604-openshift-default-dockercfg-hc8rz\""
Nov 03 20:30:35.692828 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:35.688966    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9ec153a-ac81-4e70-8524-c3121c31c41f-workload-socket\") pod \"e2e-api-b-d05604-openshift-default-6bdffb549c-w62g5\" (UID: \"b9ec153a-ac81-4e70-8524-c3121c31c41f\") " pod="openshift-ingress/e2e-api-b-d05604-openshift-default-6bdffb549c-w62g5"
Nov 03 20:30:35.692828 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:35.689143    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzvxp\" (UniqueName: \"kubernetes.io/projected/b9ec153a-ac81-4e70-8524-c3121c31c41f-kube-api-access-jzvxp\") pod \"e2e-api-b-d05604-openshift-default-6bdffb549c-w62g5\" (UID: \"b9ec153a-ac81-4e70-8524-c3121c31c41f\") " pod="openshift-ingress/e2e-api-b-d05604-openshift-default-6bdffb549c-w62g5"
Nov 03 20:30:35.692828 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:35.689311    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9ec153a-ac81-4e70-8524-c3121c31c41f-istiod-ca-cert\") pod \"e2e-api-b-d05604-openshift-default-6bdffb549c-w62g5\" (UID: \"b9ec153a-ac81-4e70-8524-c3121c31c41f\") " pod="openshift-ingress/e2e-api-b-d05604-openshift-default-6bdffb549c-w62g5"
Nov 03 20:30:35.692828 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:35.689426    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9ec153a-ac81-4e70-8524-c3121c31c41f-istio-token\") pod \"e2e-api-b-d05604-openshift-default-6bdffb549c-w62g5\" (UID: \"b9ec153a-ac81-4e70-8524-c3121c31c41f\") " pod="openshift-ingress/e2e-api-b-d05604-openshift-default-6bdffb549c-w62g5"
Nov 03 20:30:35.692828 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:35.689481    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9ec153a-ac81-4e70-8524-c3121c31c41f-istio-podinfo\") pod \"e2e-api-b-d05604-openshift-default-6bdffb549c-w62g5\" (UID: \"b9ec153a-ac81-4e70-8524-c3121c31c41f\") " pod="openshift-ingress/e2e-api-b-d05604-openshift-default-6bdffb549c-w62g5"
Nov 03 20:30:35.692828 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:35.689586    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9ec153a-ac81-4e70-8524-c3121c31c41f-workload-certs\") pod \"e2e-api-b-d05604-openshift-default-6bdffb549c-w62g5\" (UID: \"b9ec153a-ac81-4e70-8524-c3121c31c41f\") " pod="openshift-ingress/e2e-api-b-d05604-openshift-default-6bdffb549c-w62g5"
Nov 03 20:30:35.692828 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:35.689637    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9ec153a-ac81-4e70-8524-c3121c31c41f-istio-envoy\") pod \"e2e-api-b-d05604-openshift-default-6bdffb549c-w62g5\" (UID: \"b9ec153a-ac81-4e70-8524-c3121c31c41f\") " pod="openshift-ingress/e2e-api-b-d05604-openshift-default-6bdffb549c-w62g5"
Nov 03 20:30:35.692828 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:35.689720    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9ec153a-ac81-4e70-8524-c3121c31c41f-istio-data\") pod \"e2e-api-b-d05604-openshift-default-6bdffb549c-w62g5\" (UID: \"b9ec153a-ac81-4e70-8524-c3121c31c41f\") " pod="openshift-ingress/e2e-api-b-d05604-openshift-default-6bdffb549c-w62g5"
Nov 03 20:30:35.692828 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:35.689750    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9ec153a-ac81-4e70-8524-c3121c31c41f-credential-socket\") pod \"e2e-api-b-d05604-openshift-default-6bdffb549c-w62g5\" (UID: \"b9ec153a-ac81-4e70-8524-c3121c31c41f\") " pod="openshift-ingress/e2e-api-b-d05604-openshift-default-6bdffb549c-w62g5"
Nov 03 20:30:35.802912 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:35.793342    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9ec153a-ac81-4e70-8524-c3121c31c41f-workload-socket\") pod \"e2e-api-b-d05604-openshift-default-6bdffb549c-w62g5\" (UID: \"b9ec153a-ac81-4e70-8524-c3121c31c41f\") " pod="openshift-ingress/e2e-api-b-d05604-openshift-default-6bdffb549c-w62g5"
Nov 03 20:30:35.802912 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:35.793460    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzvxp\" (UniqueName: \"kubernetes.io/projected/b9ec153a-ac81-4e70-8524-c3121c31c41f-kube-api-access-jzvxp\") pod \"e2e-api-b-d05604-openshift-default-6bdffb549c-w62g5\" (UID: \"b9ec153a-ac81-4e70-8524-c3121c31c41f\") " pod="openshift-ingress/e2e-api-b-d05604-openshift-default-6bdffb549c-w62g5"
Nov 03 20:30:35.802912 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:35.793560    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9ec153a-ac81-4e70-8524-c3121c31c41f-istiod-ca-cert\") pod \"e2e-api-b-d05604-openshift-default-6bdffb549c-w62g5\" (UID: \"b9ec153a-ac81-4e70-8524-c3121c31c41f\") " pod="openshift-ingress/e2e-api-b-d05604-openshift-default-6bdffb549c-w62g5"
Nov 03 20:30:35.802912 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:35.793648    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9ec153a-ac81-4e70-8524-c3121c31c41f-istio-token\") pod \"e2e-api-b-d05604-openshift-default-6bdffb549c-w62g5\" (UID: \"b9ec153a-ac81-4e70-8524-c3121c31c41f\") " pod="openshift-ingress/e2e-api-b-d05604-openshift-default-6bdffb549c-w62g5"
Nov 03 20:30:35.802912 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:35.793716    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9ec153a-ac81-4e70-8524-c3121c31c41f-istio-podinfo\") pod \"e2e-api-b-d05604-openshift-default-6bdffb549c-w62g5\" (UID: \"b9ec153a-ac81-4e70-8524-c3121c31c41f\") " pod="openshift-ingress/e2e-api-b-d05604-openshift-default-6bdffb549c-w62g5"
Nov 03 20:30:35.802912 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:35.793843    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9ec153a-ac81-4e70-8524-c3121c31c41f-workload-certs\") pod \"e2e-api-b-d05604-openshift-default-6bdffb549c-w62g5\" (UID: \"b9ec153a-ac81-4e70-8524-c3121c31c41f\") " pod="openshift-ingress/e2e-api-b-d05604-openshift-default-6bdffb549c-w62g5"
Nov 03 20:30:35.802912 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:35.793888    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9ec153a-ac81-4e70-8524-c3121c31c41f-istio-envoy\") pod \"e2e-api-b-d05604-openshift-default-6bdffb549c-w62g5\" (UID: \"b9ec153a-ac81-4e70-8524-c3121c31c41f\") " pod="openshift-ingress/e2e-api-b-d05604-openshift-default-6bdffb549c-w62g5"
Nov 03 20:30:35.802912 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:35.793962    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9ec153a-ac81-4e70-8524-c3121c31c41f-istio-data\") pod \"e2e-api-b-d05604-openshift-default-6bdffb549c-w62g5\" (UID: \"b9ec153a-ac81-4e70-8524-c3121c31c41f\") " pod="openshift-ingress/e2e-api-b-d05604-openshift-default-6bdffb549c-w62g5"
Nov 03 20:30:35.802912 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:35.793994    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9ec153a-ac81-4e70-8524-c3121c31c41f-credential-socket\") pod \"e2e-api-b-d05604-openshift-default-6bdffb549c-w62g5\" (UID: \"b9ec153a-ac81-4e70-8524-c3121c31c41f\") " pod="openshift-ingress/e2e-api-b-d05604-openshift-default-6bdffb549c-w62g5"
Nov 03 20:30:35.802912 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:35.794530    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9ec153a-ac81-4e70-8524-c3121c31c41f-credential-socket\") pod \"e2e-api-b-d05604-openshift-default-6bdffb549c-w62g5\" (UID: \"b9ec153a-ac81-4e70-8524-c3121c31c41f\") " pod="openshift-ingress/e2e-api-b-d05604-openshift-default-6bdffb549c-w62g5"
Nov 03 20:30:35.802912 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:35.794810    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9ec153a-ac81-4e70-8524-c3121c31c41f-workload-socket\") pod \"e2e-api-b-d05604-openshift-default-6bdffb549c-w62g5\" (UID: \"b9ec153a-ac81-4e70-8524-c3121c31c41f\") " pod="openshift-ingress/e2e-api-b-d05604-openshift-default-6bdffb549c-w62g5"
Nov 03 20:30:35.839145 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:35.829864    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9ec153a-ac81-4e70-8524-c3121c31c41f-workload-certs\") pod \"e2e-api-b-d05604-openshift-default-6bdffb549c-w62g5\" (UID: \"b9ec153a-ac81-4e70-8524-c3121c31c41f\") " pod="openshift-ingress/e2e-api-b-d05604-openshift-default-6bdffb549c-w62g5"
Nov 03 20:30:35.839145 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:35.834478    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9ec153a-ac81-4e70-8524-c3121c31c41f-istio-data\") pod \"e2e-api-b-d05604-openshift-default-6bdffb549c-w62g5\" (UID: \"b9ec153a-ac81-4e70-8524-c3121c31c41f\") " pod="openshift-ingress/e2e-api-b-d05604-openshift-default-6bdffb549c-w62g5"
Nov 03 20:30:35.839145 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:35.835018    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9ec153a-ac81-4e70-8524-c3121c31c41f-istio-podinfo\") pod \"e2e-api-b-d05604-openshift-default-6bdffb549c-w62g5\" (UID: \"b9ec153a-ac81-4e70-8524-c3121c31c41f\") " pod="openshift-ingress/e2e-api-b-d05604-openshift-default-6bdffb549c-w62g5"
Nov 03 20:30:35.839145 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:35.836805    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9ec153a-ac81-4e70-8524-c3121c31c41f-istiod-ca-cert\") pod \"e2e-api-b-d05604-openshift-default-6bdffb549c-w62g5\" (UID: \"b9ec153a-ac81-4e70-8524-c3121c31c41f\") " pod="openshift-ingress/e2e-api-b-d05604-openshift-default-6bdffb549c-w62g5"
Nov 03 20:30:35.854802 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:35.846425    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9ec153a-ac81-4e70-8524-c3121c31c41f-istio-token\") pod \"e2e-api-b-d05604-openshift-default-6bdffb549c-w62g5\" (UID: \"b9ec153a-ac81-4e70-8524-c3121c31c41f\") " pod="openshift-ingress/e2e-api-b-d05604-openshift-default-6bdffb549c-w62g5"
Nov 03 20:30:35.854802 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:35.851311    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9ec153a-ac81-4e70-8524-c3121c31c41f-istio-envoy\") pod \"e2e-api-b-d05604-openshift-default-6bdffb549c-w62g5\" (UID: \"b9ec153a-ac81-4e70-8524-c3121c31c41f\") " pod="openshift-ingress/e2e-api-b-d05604-openshift-default-6bdffb549c-w62g5"
Nov 03 20:30:35.887002 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:35.867046    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzvxp\" (UniqueName: \"kubernetes.io/projected/b9ec153a-ac81-4e70-8524-c3121c31c41f-kube-api-access-jzvxp\") pod \"e2e-api-b-d05604-openshift-default-6bdffb549c-w62g5\" (UID: \"b9ec153a-ac81-4e70-8524-c3121c31c41f\") " pod="openshift-ingress/e2e-api-b-d05604-openshift-default-6bdffb549c-w62g5"
Nov 03 20:30:36.132995 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:36.132886    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d05604-openshift-default-6bdffb549c-w62g5"
Nov 03 20:30:36.462696 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:36.462664    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-d05604-openshift-default-6bdffb549c-w62g5"]
Nov 03 20:30:36.479519 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:36.479470    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:30:36.479692 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:36.479559    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:30:36.479692 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:36.479587    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:30:37.401117 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:37.400279    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d05604-openshift-default-6bdffb549c-w62g5" event={"ID":"b9ec153a-ac81-4e70-8524-c3121c31c41f","Type":"ContainerStarted","Data":"ee06ea841d618a90e18d8c5fa9a485cb9edcb998c251762a4960ceceeef12d2b"}
Nov 03 20:30:37.401117 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:37.400405    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d05604-openshift-default-6bdffb549c-w62g5" event={"ID":"b9ec153a-ac81-4e70-8524-c3121c31c41f","Type":"ContainerStarted","Data":"5d9aec411ccf53ae23211129b4d1f200eb4318a7c86e3c851d379201e1925059"}
Nov 03 20:30:37.478018 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:37.477957    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-d05604-openshift-default-6bdffb549c-w62g5" podStartSLOduration=2.477939911 podStartE2EDuration="2.477939911s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:30:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:30:37.473649432 +0000 UTC m=+1571.626311332" watchObservedRunningTime="2026-11-03 20:30:37.477939911 +0000 UTC m=+1571.630601797"
Nov 03 20:30:38.134090 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:38.134049    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-d05604-openshift-default-6bdffb549c-w62g5"
Nov 03 20:30:38.140959 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:38.140927    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-d05604-openshift-default-6bdffb549c-w62g5"
Nov 03 20:30:38.408572 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:38.408486    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-d05604-openshift-default-6bdffb549c-w62g5"
Nov 03 20:30:38.409501 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:38.409481    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-d05604-openshift-default-6bdffb549c-w62g5"
Nov 03 20:30:41.723948 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:41.723906    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d05604-7799fccfc9-m4xqb"]
Nov 03 20:30:41.724707 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:41.724690    5570 cpu_manager.go:414] "containerMap: removing container" podUID="4b3cfadc-bda0-4d0a-abff-056991a17a37" containerName="cleanup"
Nov 03 20:30:41.724707 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:41.724708    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b3cfadc-bda0-4d0a-abff-056991a17a37" containerName="cleanup"
Nov 03 20:30:41.724838 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:41.724726    5570 cpu_manager.go:414] "containerMap: removing container" podUID="3b745fba-4594-4621-81bd-e322d6dd1efa" containerName="cleanup"
Nov 03 20:30:41.724838 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:41.724731    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b745fba-4594-4621-81bd-e322d6dd1efa" containerName="cleanup"
Nov 03 20:30:41.724838 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:41.724757    5570 cpu_manager.go:414] "containerMap: removing container" podUID="36bf8581-603b-4a8d-8d7b-6da231cf7161" containerName="cleanup"
Nov 03 20:30:41.724838 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:41.724765    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36bf8581-603b-4a8d-8d7b-6da231cf7161" containerName="cleanup"
Nov 03 20:30:41.724838 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:41.724789    5570 cpu_manager.go:414] "containerMap: removing container" podUID="7e73f70d-ab6c-4cc8-aff1-4d9833b5b991" containerName="cleanup"
Nov 03 20:30:41.724838 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:41.724796    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e73f70d-ab6c-4cc8-aff1-4d9833b5b991" containerName="cleanup"
Nov 03 20:30:41.724838 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:41.724821    5570 cpu_manager.go:414] "containerMap: removing container" podUID="11f669a0-4617-4c3e-96cc-fc288ef2aea1" containerName="cleanup"
Nov 03 20:30:41.724838 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:41.724826    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11f669a0-4617-4c3e-96cc-fc288ef2aea1" containerName="cleanup"
Nov 03 20:30:41.725077 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:41.724946    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="11f669a0-4617-4c3e-96cc-fc288ef2aea1" containerName="cleanup"
Nov 03 20:30:41.725077 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:41.724959    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b3cfadc-bda0-4d0a-abff-056991a17a37" containerName="cleanup"
Nov 03 20:30:41.725077 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:41.724966    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9cb6b5f-bf07-48ed-846a-eae8a3c19c1d" containerName="cleanup"
Nov 03 20:30:41.725077 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:41.724976    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b745fba-4594-4621-81bd-e322d6dd1efa" containerName="cleanup"
Nov 03 20:30:41.727864 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:41.727840    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d05604-7799fccfc9-m4xqb"
Nov 03 20:30:41.745672 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:41.745640    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d05604-7799fccfc9-m4xqb"]
Nov 03 20:30:41.753155 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:41.753107    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-d05604-dockercfg-6fcfg\""
Nov 03 20:30:41.753372 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:41.753143    5570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-d05604\""
Nov 03 20:30:42.117858 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:42.117566    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwvwb\" (UniqueName: \"kubernetes.io/projected/f9114ccd-117b-4fb8-9314-434a65aeabe6-kube-api-access-pwvwb\") pod \"payload-pre-processing-e2e-api-b-d05604-7799fccfc9-m4xqb\" (UID: \"f9114ccd-117b-4fb8-9314-434a65aeabe6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d05604-7799fccfc9-m4xqb"
Nov 03 20:30:42.117858 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:42.117649    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9114ccd-117b-4fb8-9314-434a65aeabe6-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-d05604-7799fccfc9-m4xqb\" (UID: \"f9114ccd-117b-4fb8-9314-434a65aeabe6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d05604-7799fccfc9-m4xqb"
Nov 03 20:30:42.222003 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:42.221960    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwvwb\" (UniqueName: \"kubernetes.io/projected/f9114ccd-117b-4fb8-9314-434a65aeabe6-kube-api-access-pwvwb\") pod \"payload-pre-processing-e2e-api-b-d05604-7799fccfc9-m4xqb\" (UID: \"f9114ccd-117b-4fb8-9314-434a65aeabe6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d05604-7799fccfc9-m4xqb"
Nov 03 20:30:42.222311 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:42.222293    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9114ccd-117b-4fb8-9314-434a65aeabe6-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-d05604-7799fccfc9-m4xqb\" (UID: \"f9114ccd-117b-4fb8-9314-434a65aeabe6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d05604-7799fccfc9-m4xqb"
Nov 03 20:30:42.223697 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:42.223616    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9114ccd-117b-4fb8-9314-434a65aeabe6-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-d05604-7799fccfc9-m4xqb\" (UID: \"f9114ccd-117b-4fb8-9314-434a65aeabe6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d05604-7799fccfc9-m4xqb"
Nov 03 20:30:42.241246 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:42.241213    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d05604-5fb56668c4-24x95"]
Nov 03 20:30:42.247175 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:42.246500    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-d05604-5fb56668c4-24x95"
Nov 03 20:30:42.247734 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:42.247704    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwvwb\" (UniqueName: \"kubernetes.io/projected/f9114ccd-117b-4fb8-9314-434a65aeabe6-kube-api-access-pwvwb\") pod \"payload-pre-processing-e2e-api-b-d05604-7799fccfc9-m4xqb\" (UID: \"f9114ccd-117b-4fb8-9314-434a65aeabe6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d05604-7799fccfc9-m4xqb"
Nov 03 20:30:42.249336 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:42.249309    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d05604-5fb56668c4-24x95"]
Nov 03 20:30:42.324835 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:42.324794    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2clp9\" (UniqueName: \"kubernetes.io/projected/6216548e-093f-4c34-bb8a-01fe1ad91aa8-kube-api-access-2clp9\") pod \"payload-processing-e2e-api-b-d05604-5fb56668c4-24x95\" (UID: \"6216548e-093f-4c34-bb8a-01fe1ad91aa8\") " pod="openshift-ingress/payload-processing-e2e-api-b-d05604-5fb56668c4-24x95"
Nov 03 20:30:42.325157 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:42.325129    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6216548e-093f-4c34-bb8a-01fe1ad91aa8-plugins-config-volume\") pod \"payload-processing-e2e-api-b-d05604-5fb56668c4-24x95\" (UID: \"6216548e-093f-4c34-bb8a-01fe1ad91aa8\") " pod="openshift-ingress/payload-processing-e2e-api-b-d05604-5fb56668c4-24x95"
Nov 03 20:30:42.345227 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:42.345181    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d05604-7799fccfc9-m4xqb"
Nov 03 20:30:42.451798 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:42.439345    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2clp9\" (UniqueName: \"kubernetes.io/projected/6216548e-093f-4c34-bb8a-01fe1ad91aa8-kube-api-access-2clp9\") pod \"payload-processing-e2e-api-b-d05604-5fb56668c4-24x95\" (UID: \"6216548e-093f-4c34-bb8a-01fe1ad91aa8\") " pod="openshift-ingress/payload-processing-e2e-api-b-d05604-5fb56668c4-24x95"
Nov 03 20:30:42.451798 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:42.439477    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6216548e-093f-4c34-bb8a-01fe1ad91aa8-plugins-config-volume\") pod \"payload-processing-e2e-api-b-d05604-5fb56668c4-24x95\" (UID: \"6216548e-093f-4c34-bb8a-01fe1ad91aa8\") " pod="openshift-ingress/payload-processing-e2e-api-b-d05604-5fb56668c4-24x95"
Nov 03 20:30:42.451798 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:42.440740    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6216548e-093f-4c34-bb8a-01fe1ad91aa8-plugins-config-volume\") pod \"payload-processing-e2e-api-b-d05604-5fb56668c4-24x95\" (UID: \"6216548e-093f-4c34-bb8a-01fe1ad91aa8\") " pod="openshift-ingress/payload-processing-e2e-api-b-d05604-5fb56668c4-24x95"
Nov 03 20:30:42.496875 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:42.489021    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2clp9\" (UniqueName: \"kubernetes.io/projected/6216548e-093f-4c34-bb8a-01fe1ad91aa8-kube-api-access-2clp9\") pod \"payload-processing-e2e-api-b-d05604-5fb56668c4-24x95\" (UID: \"6216548e-093f-4c34-bb8a-01fe1ad91aa8\") " pod="openshift-ingress/payload-processing-e2e-api-b-d05604-5fb56668c4-24x95"
Nov 03 20:30:42.595500 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:42.595442    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-d05604-5fb56668c4-24x95"
Nov 03 20:30:43.021481 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:43.021449    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d05604-7799fccfc9-m4xqb"]
Nov 03 20:30:43.021894 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:30:43.021818    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9114ccd_117b_4fb8_9314_434a65aeabe6.slice/crio-fcb8c1cc1a67e43785dcf1b2d2129218b5f68f8fda7cf3386774915c0f7eb728 WatchSource:0}: Error finding container fcb8c1cc1a67e43785dcf1b2d2129218b5f68f8fda7cf3386774915c0f7eb728: Status 404 returned error can't find the container with id fcb8c1cc1a67e43785dcf1b2d2129218b5f68f8fda7cf3386774915c0f7eb728
Nov 03 20:30:43.051662 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:43.051619    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d05604-5fb56668c4-24x95"]
Nov 03 20:30:43.071559 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:30:43.071524    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6216548e_093f_4c34_bb8a_01fe1ad91aa8.slice/crio-4a9dab3bf88c4a0e6955db31b2485b4a977aea3e89c5659ceb8e60b51a203970 WatchSource:0}: Error finding container 4a9dab3bf88c4a0e6955db31b2485b4a977aea3e89c5659ceb8e60b51a203970: Status 404 returned error can't find the container with id 4a9dab3bf88c4a0e6955db31b2485b4a977aea3e89c5659ceb8e60b51a203970
Nov 03 20:30:43.532836 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:43.532745    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-d05604-5fb56668c4-24x95" event={"ID":"6216548e-093f-4c34-bb8a-01fe1ad91aa8","Type":"ContainerStarted","Data":"26aad60548fb9ae9dc303f7839e88dc85f6e3efae94f71132f80941b40cc45ff"}
Nov 03 20:30:43.533031 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:43.532857    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-d05604-5fb56668c4-24x95" event={"ID":"6216548e-093f-4c34-bb8a-01fe1ad91aa8","Type":"ContainerStarted","Data":"4a9dab3bf88c4a0e6955db31b2485b4a977aea3e89c5659ceb8e60b51a203970"}
Nov 03 20:30:43.533031 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:43.532886    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-d05604-5fb56668c4-24x95"
Nov 03 20:30:43.534862 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:43.534836    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d05604-7799fccfc9-m4xqb" event={"ID":"f9114ccd-117b-4fb8-9314-434a65aeabe6","Type":"ContainerStarted","Data":"596bfe0429d65cc33f5a9a1b08e2d9f0fa5b38e1da423abdc5f62b353dc81499"}
Nov 03 20:30:43.535014 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:43.534881    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d05604-7799fccfc9-m4xqb" event={"ID":"f9114ccd-117b-4fb8-9314-434a65aeabe6","Type":"ContainerStarted","Data":"fcb8c1cc1a67e43785dcf1b2d2129218b5f68f8fda7cf3386774915c0f7eb728"}
Nov 03 20:30:43.535089 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:43.535020    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d05604-7799fccfc9-m4xqb"
Nov 03 20:30:43.574632 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:43.574566    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-d05604-5fb56668c4-24x95" podStartSLOduration=2.574545525 podStartE2EDuration="2.574545525s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:30:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:30:43.573538779 +0000 UTC m=+1577.726200663" watchObservedRunningTime="2026-11-03 20:30:43.574545525 +0000 UTC m=+1577.727207415"
Nov 03 20:30:43.617866 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:43.617792    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d05604-7799fccfc9-m4xqb" podStartSLOduration=2.617755228 podStartE2EDuration="2.617755228s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:30:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:30:43.614308208 +0000 UTC m=+1577.766970095" watchObservedRunningTime="2026-11-03 20:30:43.617755228 +0000 UTC m=+1577.770417115"
Nov 03 20:30:49.079570 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:49.079508    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g69f8"]
Nov 03 20:30:52.116882 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:52.116762    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/premium-w4-e80203-kserve-5f9f445456-bms9d" podUID="6ad76efc-20c2-4951-b22a-c5a39cd39ed4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Nov 03 20:30:52.272987 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:52.272926    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/distinct-2-w0-e40816-kserve-79fb46654d-s242x" podUID="4ad9b048-925c-4d01-a839-a5f25d4dd02d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:30:52.326805 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:52.324884    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-2-w6-9e1bcb-kserve-7d96d9f6f6-4f7fl" podUID="56353b0f-08ee-4ada-9dc0-ac5afd5a59c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:30:52.645288 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:52.645237    5570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-e40816/distinct-2-w0-e40816-kserve-79fb46654d-s242x" podUID="4ad9b048-925c-4d01-a839-a5f25d4dd02d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:30:53.168800 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:53.168190    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/premium-w4-e80203-kserve-5f9f445456-bms9d" podUID="6ad76efc-20c2-4951-b22a-c5a39cd39ed4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:30:53.169353 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:53.169099    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-e80203/premium-w4-e80203-kserve-5f9f445456-bms9d"
Nov 03 20:30:53.282798 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:53.276890    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/distinct-2-w0-e40816-kserve-79fb46654d-s242x" podUID="4ad9b048-925c-4d01-a839-a5f25d4dd02d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:30:53.282798 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:53.277569    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e40816/distinct-2-w0-e40816-kserve-79fb46654d-s242x"
Nov 03 20:30:53.314801 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:53.301272    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz" podUID="3a715ef1-38e9-4386-9eec-e10e8df11a10" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:30:53.387956 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:53.377594    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/distinct-2-w3-482722-kserve-74bf58cf7d-v9cjw" podUID="9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded"
Nov 03 20:30:53.387956 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:53.378035    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/distinct-2-w4-e80203-kserve-756f9b9bc6-69gxj" podUID="ccf5dc2f-f116-4917-bb1a-b136d8ed17f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:30:53.456797 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:53.453276    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-2-w6-9e1bcb-kserve-7d96d9f6f6-4f7fl" podUID="56353b0f-08ee-4ada-9dc0-ac5afd5a59c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded"
Nov 03 20:30:53.456797 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:53.453681    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-2-w6-9e1bcb-kserve-7d96d9f6f6-4f7fl"
Nov 03 20:30:53.550942 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:53.550876    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh" podUID="2d3647f2-89e9-4908-a8ee-d84e527abeb0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:30:53.788861 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:53.785299    5570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-e80203/premium-w4-e80203-kserve-5f9f445456-bms9d" podUID="6ad76efc-20c2-4951-b22a-c5a39cd39ed4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": context deadline exceeded"
Nov 03 20:30:54.059230 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:53.917990    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/premium-w2-9973d9-kserve-5c6dd4669c-5gv8x" podUID="cefc89ad-7ae1-433b-979d-6601678a6d98" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Nov 03 20:30:54.118858 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:54.091065    5570 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-vdrfs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:30:54.118858 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:54.091147    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-vdrfs" podUID="1b70a87b-76a6-40e6-9ced-037c3ceee5cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:30:54.118858 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:54.091810    5570 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-657vf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:30:54.118858 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:54.091888    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-657vf" podUID="f089dd20-d6a6-4c63-88a5-3fbe9a2ff315" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:30:54.326894 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:54.293709    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/premium-w4-e80203-kserve-5f9f445456-bms9d" podUID="6ad76efc-20c2-4951-b22a-c5a39cd39ed4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:30:54.537394 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:54.364485    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/distinct-2-w0-e40816-kserve-79fb46654d-s242x" podUID="4ad9b048-925c-4d01-a839-a5f25d4dd02d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded"
Nov 03 20:30:54.538066 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:54.381330    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/distinct-2-w4-e80203-kserve-756f9b9bc6-69gxj" podUID="ccf5dc2f-f116-4917-bb1a-b136d8ed17f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:30:54.538495 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:54.382555    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/distinct-2-w3-482722-kserve-74bf58cf7d-v9cjw" podUID="9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded"
Nov 03 20:30:54.538889 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:54.448110    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4cdbeb-6b867f8fcd-rw72k"]
Nov 03 20:30:54.539279 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:54.539227    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-e80203/distinct-2-w4-e80203-kserve-756f9b9bc6-69gxj"
Nov 03 20:30:54.574796 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:54.539492    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-482722/distinct-2-w3-482722-kserve-74bf58cf7d-v9cjw"
Nov 03 20:30:54.645956 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:54.588836    5570 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4cdbeb-6b867f8fcd-rw72k" podUID="2c6d9548-c1f6-4efb-969b-260308e4ac3a" containerName="maas-api" containerID="cri-o://dd981c84133739e32d79464a1bfd7e6f5091642f5f731090459e003ef5dffcfc" gracePeriod=30
Nov 03 20:30:54.722474 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:54.656897    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-2-w6-9e1bcb-kserve-7d96d9f6f6-4f7fl" podUID="56353b0f-08ee-4ada-9dc0-ac5afd5a59c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded"
Nov 03 20:30:54.873847 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:54.711686    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-d05604-5fb56668c4-24x95"
Nov 03 20:30:54.933005 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:54.932884    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/premium-w2-9973d9-kserve-5c6dd4669c-5gv8x" podUID="cefc89ad-7ae1-433b-979d-6601678a6d98" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:30:54.962448 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:54.962409    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9973d9/premium-w2-9973d9-kserve-5c6dd4669c-5gv8x"
Nov 03 20:30:54.978802 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:54.973030    5570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-9973d9/premium-w2-9973d9-kserve-5c6dd4669c-5gv8x" podUID="cefc89ad-7ae1-433b-979d-6601678a6d98" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:30:55.044796 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:55.012166    5570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-482722/distinct-w3-482722-kserve-d876967c-kd488" podUID="9a97f9cc-2261-471d-9053-51ee965850bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:30:55.170796 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:55.152554    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/distinct-w3-482722-kserve-d876967c-kd488" podUID="9a97f9cc-2261-471d-9053-51ee965850bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded"
Nov 03 20:30:55.668100 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:55.667988    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/distinct-2-w3-482722-kserve-74bf58cf7d-v9cjw" podUID="9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:30:55.669529 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:55.669492    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/premium-w3-482722-kserve-689dc7bd5d-x8rjp" podUID="369045c3-4d6a-49a7-9bc6-f2939883bee8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Nov 03 20:30:55.670752 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:55.670727    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/simulator-w0-e40816-kserve-688f79796f-2tvw5" podUID="c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Nov 03 20:30:55.712013 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:55.711960    5570 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-482722/premium-w3-482722-kserve-689dc7bd5d-x8rjp" podUID="369045c3-4d6a-49a7-9bc6-f2939883bee8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:30:55.755800 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:55.747757    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh" podUID="2d3647f2-89e9-4908-a8ee-d84e527abeb0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Nov 03 20:30:55.776828 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:55.765203    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/premium-w4-e80203-kserve-5f9f445456-bms9d" podUID="6ad76efc-20c2-4951-b22a-c5a39cd39ed4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Nov 03 20:30:55.776828 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:55.765952    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/distinct-2-w0-e40816-kserve-79fb46654d-s242x" podUID="4ad9b048-925c-4d01-a839-a5f25d4dd02d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded"
Nov 03 20:30:55.776828 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:55.766625    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/distinct-2-w4-e80203-kserve-756f9b9bc6-69gxj" podUID="ccf5dc2f-f116-4917-bb1a-b136d8ed17f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded"
Nov 03 20:30:55.822336 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:55.795988    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-9e1bcb/premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh"
Nov 03 20:30:55.857000 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:55.856948    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-2-w6-9e1bcb-kserve-7d96d9f6f6-4f7fl" podUID="56353b0f-08ee-4ada-9dc0-ac5afd5a59c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:30:56.105820 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:56.098103    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/premium-w2-9973d9-kserve-5c6dd4669c-5gv8x" podUID="cefc89ad-7ae1-433b-979d-6601678a6d98" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:30:56.293902 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:56.289587    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/distinct-w3-482722-kserve-d876967c-kd488" podUID="9a97f9cc-2261-471d-9053-51ee965850bd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:30:56.530192 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:56.530151    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz" podUID="3a715ef1-38e9-4386-9eec-e10e8df11a10" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": context deadline exceeded"
Nov 03 20:30:56.633083 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:56.625024    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-482722/distinct-w3-482722-kserve-d876967c-kd488"
Nov 03 20:30:56.633430 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:56.633406    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz"
Nov 03 20:30:56.656319 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:56.655580    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4cdbeb-6b867f8fcd-rw72k" podUID="2c6d9548-c1f6-4efb-969b-260308e4ac3a" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.124:8443/health\": dial tcp 10.134.0.124:8443: connect: connection refused"
Nov 03 20:30:56.677584 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:56.673892    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/simulator-w0-e40816-kserve-688f79796f-2tvw5" podUID="c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:30:56.691472 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:56.691432    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/premium-w3-482722-kserve-689dc7bd5d-x8rjp" podUID="369045c3-4d6a-49a7-9bc6-f2939883bee8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Nov 03 20:30:56.718997 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:56.718950    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/distinct-2-w3-482722-kserve-74bf58cf7d-v9cjw" podUID="9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:30:56.812797 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:56.794017    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/distinct-2-w4-e80203-kserve-756f9b9bc6-69gxj" podUID="ccf5dc2f-f116-4917-bb1a-b136d8ed17f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:30:56.812797 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:56.794468    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh" podUID="2d3647f2-89e9-4908-a8ee-d84e527abeb0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Nov 03 20:30:56.812797 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:56.794792    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/premium-w4-e80203-kserve-5f9f445456-bms9d" podUID="6ad76efc-20c2-4951-b22a-c5a39cd39ed4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Nov 03 20:30:56.812797 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:56.795077    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/distinct-2-w0-e40816-kserve-79fb46654d-s242x" podUID="4ad9b048-925c-4d01-a839-a5f25d4dd02d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded"
Nov 03 20:30:56.933853 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:56.922729    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-2-w6-9e1bcb-kserve-7d96d9f6f6-4f7fl" podUID="56353b0f-08ee-4ada-9dc0-ac5afd5a59c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:30:57.008793 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:57.006310    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e40816/simulator-w0-e40816-kserve-688f79796f-2tvw5"
Nov 03 20:30:57.008793 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:57.006394    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-482722/premium-w3-482722-kserve-689dc7bd5d-x8rjp"
Nov 03 20:30:57.008793 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:57.006440    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e40816/distinct-2-w0-e40816-kserve-79fb46654d-s242x"
Nov 03 20:30:57.008793 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:57.006525    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-e80203/premium-w4-e80203-kserve-5f9f445456-bms9d"
Nov 03 20:30:57.008793 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:57.006557    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e40816/simulator-w0-e40816-kserve-688f79796f-2tvw5"
Nov 03 20:30:57.008793 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:57.006590    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz"
Nov 03 20:30:57.008793 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:57.006621    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-9e1bcb/premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh"
Nov 03 20:30:57.008793 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:57.006650    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-482722/premium-w3-482722-kserve-689dc7bd5d-x8rjp"
Nov 03 20:30:57.008793 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:57.006682    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-e80203/distinct-2-w4-e80203-kserve-756f9b9bc6-69gxj"
Nov 03 20:30:57.008793 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:57.006714    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-482722/distinct-w3-482722-kserve-d876967c-kd488"
Nov 03 20:30:57.024482 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:57.009522    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9973d9/premium-w2-9973d9-kserve-5c6dd4669c-5gv8x"
Nov 03 20:30:57.028291 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:57.027607    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d05604-7799fccfc9-m4xqb"
Nov 03 20:30:57.031397 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:57.031366    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-482722/distinct-2-w3-482722-kserve-74bf58cf7d-v9cjw"
Nov 03 20:30:57.046828 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:57.036977    5570 generic.go:356] "Generic (PLEG): container finished" podID="2c6d9548-c1f6-4efb-969b-260308e4ac3a" containerID="dd981c84133739e32d79464a1bfd7e6f5091642f5f731090459e003ef5dffcfc" exitCode=0
Nov 03 20:30:57.046828 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:57.037065    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-2-w6-9e1bcb-kserve-7d96d9f6f6-4f7fl"
Nov 03 20:30:57.046828 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:57.037102    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4cdbeb-6b867f8fcd-rw72k" event={"ID":"2c6d9548-c1f6-4efb-969b-260308e4ac3a","Type":"ContainerDied","Data":"dd981c84133739e32d79464a1bfd7e6f5091642f5f731090459e003ef5dffcfc"}
Nov 03 20:30:57.106557 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:57.101606    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4cdbeb-6b867f8fcd-rw72k"
Nov 03 20:30:57.231315 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:57.231277    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c6d9548-c1f6-4efb-969b-260308e4ac3a-kube-api-access-72pxf\" (UniqueName: \"kubernetes.io/projected/2c6d9548-c1f6-4efb-969b-260308e4ac3a-kube-api-access-72pxf\") pod \"2c6d9548-c1f6-4efb-969b-260308e4ac3a\" (UID: \"2c6d9548-c1f6-4efb-969b-260308e4ac3a\") "
Nov 03 20:30:57.231520 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:57.231500    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c6d9548-c1f6-4efb-969b-260308e4ac3a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c6d9548-c1f6-4efb-969b-260308e4ac3a-maas-api-tls\") pod \"2c6d9548-c1f6-4efb-969b-260308e4ac3a\" (UID: \"2c6d9548-c1f6-4efb-969b-260308e4ac3a\") "
Nov 03 20:30:57.231592 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:57.231559    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c6d9548-c1f6-4efb-969b-260308e4ac3a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c6d9548-c1f6-4efb-969b-260308e4ac3a-metrics-certs\") pod \"2c6d9548-c1f6-4efb-969b-260308e4ac3a\" (UID: \"2c6d9548-c1f6-4efb-969b-260308e4ac3a\") "
Nov 03 20:30:57.231740 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:57.231723    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c6d9548-c1f6-4efb-969b-260308e4ac3a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c6d9548-c1f6-4efb-969b-260308e4ac3a-openshift-service-ca\") pod \"2c6d9548-c1f6-4efb-969b-260308e4ac3a\" (UID: \"2c6d9548-c1f6-4efb-969b-260308e4ac3a\") "
Nov 03 20:30:57.232679 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:57.232652    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c6d9548-c1f6-4efb-969b-260308e4ac3a-openshift-service-ca" pod "2c6d9548-c1f6-4efb-969b-260308e4ac3a" (UID: "2c6d9548-c1f6-4efb-969b-260308e4ac3a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:30:57.232912 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:57.232887    5570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c6d9548-c1f6-4efb-969b-260308e4ac3a-openshift-service-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:30:57.235386 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:57.235359    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c6d9548-c1f6-4efb-969b-260308e4ac3a-metrics-certs" pod "2c6d9548-c1f6-4efb-969b-260308e4ac3a" (UID: "2c6d9548-c1f6-4efb-969b-260308e4ac3a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:30:57.236017 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:57.235967    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c6d9548-c1f6-4efb-969b-260308e4ac3a-maas-api-tls" pod "2c6d9548-c1f6-4efb-969b-260308e4ac3a" (UID: "2c6d9548-c1f6-4efb-969b-260308e4ac3a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:30:57.321834 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:57.316812    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-addf1a-5584bbb5cb-tv2db"]
Nov 03 20:30:57.324641 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:30:57.324588    5570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8bwhz" containerName="extractor"
Nov 03 20:30:57.326117 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:57.326089    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c6d9548-c1f6-4efb-969b-260308e4ac3a-kube-api-access-72pxf" pod "2c6d9548-c1f6-4efb-969b-260308e4ac3a" (UID: "2c6d9548-c1f6-4efb-969b-260308e4ac3a"). InnerVolumeSpecName "kube-api-access-72pxf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:30:57.328653 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:57.328598    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-addf1a-857995db79-q4sn9"]
Nov 03 20:30:57.334651 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:57.334617    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72pxf\" (UniqueName: \"kubernetes.io/projected/2c6d9548-c1f6-4efb-969b-260308e4ac3a-kube-api-access-72pxf\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:30:57.334651 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:57.334654    5570 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c6d9548-c1f6-4efb-969b-260308e4ac3a-maas-api-tls\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:30:57.334858 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:57.334668    5570 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c6d9548-c1f6-4efb-969b-260308e4ac3a-metrics-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:30:58.428863 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:58.428117    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4cdbeb-6b867f8fcd-rw72k"
Nov 03 20:30:58.499811 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:58.499270    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4cdbeb-6b867f8fcd-rw72k" event={"ID":"2c6d9548-c1f6-4efb-969b-260308e4ac3a","Type":"ContainerDied","Data":"bfb84b25cc95ad52751e51bad85aef9c2e842dcdf06a9138b7040cbcfc4d45cf"}
Nov 03 20:30:58.499811 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:58.499358    5570 scope.go:122] "RemoveContainer" containerID="dd981c84133739e32d79464a1bfd7e6f5091642f5f731090459e003ef5dffcfc"
Nov 03 20:30:58.831346 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:58.831311    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4cdbeb-6b867f8fcd-rw72k"]
Nov 03 20:30:58.835255 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:58.835215    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4cdbeb-6b867f8fcd-rw72k"]
Nov 03 20:30:59.990699 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:30:59.990657    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c6d9548-c1f6-4efb-969b-260308e4ac3a" path="/var/lib/kubelet/pods/2c6d9548-c1f6-4efb-969b-260308e4ac3a/volumes"
Nov 03 20:31:00.901157 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:00.901112    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xhr9f"]
Nov 03 20:31:00.901392 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:00.901367    5570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xhr9f" podUID="5668dee7-8a03-4a0c-ab66-303fa052ea77" containerName="discovery" containerID="cri-o://1f19f2144b102cc1956626a8bb852b8709eee1ead55e6bac6ab748e23cc1fe79" gracePeriod=30
Nov 03 20:31:01.500205 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:01.500171    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xhr9f"
Nov 03 20:31:01.615869 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:01.614534    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5668dee7-8a03-4a0c-ab66-303fa052ea77-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5668dee7-8a03-4a0c-ab66-303fa052ea77-local-certs\") pod \"5668dee7-8a03-4a0c-ab66-303fa052ea77\" (UID: \"5668dee7-8a03-4a0c-ab66-303fa052ea77\") "
Nov 03 20:31:01.615869 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:01.614613    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5668dee7-8a03-4a0c-ab66-303fa052ea77-cacerts\" (UniqueName: \"kubernetes.io/secret/5668dee7-8a03-4a0c-ab66-303fa052ea77-cacerts\") pod \"5668dee7-8a03-4a0c-ab66-303fa052ea77\" (UID: \"5668dee7-8a03-4a0c-ab66-303fa052ea77\") "
Nov 03 20:31:01.615869 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:01.614723    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5668dee7-8a03-4a0c-ab66-303fa052ea77-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5668dee7-8a03-4a0c-ab66-303fa052ea77-istio-csr-dns-cert\") pod \"5668dee7-8a03-4a0c-ab66-303fa052ea77\" (UID: \"5668dee7-8a03-4a0c-ab66-303fa052ea77\") "
Nov 03 20:31:01.615869 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:01.614883    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5668dee7-8a03-4a0c-ab66-303fa052ea77-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5668dee7-8a03-4a0c-ab66-303fa052ea77-istio-kubeconfig\") pod \"5668dee7-8a03-4a0c-ab66-303fa052ea77\" (UID: \"5668dee7-8a03-4a0c-ab66-303fa052ea77\") "
Nov 03 20:31:01.615869 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:01.614959    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5668dee7-8a03-4a0c-ab66-303fa052ea77-istio-token\" (UniqueName: \"kubernetes.io/projected/5668dee7-8a03-4a0c-ab66-303fa052ea77-istio-token\") pod \"5668dee7-8a03-4a0c-ab66-303fa052ea77\" (UID: \"5668dee7-8a03-4a0c-ab66-303fa052ea77\") "
Nov 03 20:31:01.615869 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:01.615102    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5668dee7-8a03-4a0c-ab66-303fa052ea77-kube-api-access-xzx8m\" (UniqueName: \"kubernetes.io/projected/5668dee7-8a03-4a0c-ab66-303fa052ea77-kube-api-access-xzx8m\") pod \"5668dee7-8a03-4a0c-ab66-303fa052ea77\" (UID: \"5668dee7-8a03-4a0c-ab66-303fa052ea77\") "
Nov 03 20:31:01.615869 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:01.615178    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5668dee7-8a03-4a0c-ab66-303fa052ea77-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5668dee7-8a03-4a0c-ab66-303fa052ea77-istio-csr-ca-configmap\") pod \"5668dee7-8a03-4a0c-ab66-303fa052ea77\" (UID: \"5668dee7-8a03-4a0c-ab66-303fa052ea77\") "
Nov 03 20:31:01.634247 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:01.632843    5570 generic.go:356] "Generic (PLEG): container finished" podID="5668dee7-8a03-4a0c-ab66-303fa052ea77" containerID="1f19f2144b102cc1956626a8bb852b8709eee1ead55e6bac6ab748e23cc1fe79" exitCode=0
Nov 03 20:31:01.634247 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:01.633157    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xhr9f" event={"ID":"5668dee7-8a03-4a0c-ab66-303fa052ea77","Type":"ContainerDied","Data":"1f19f2144b102cc1956626a8bb852b8709eee1ead55e6bac6ab748e23cc1fe79"}
Nov 03 20:31:01.634247 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:01.633247    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xhr9f" event={"ID":"5668dee7-8a03-4a0c-ab66-303fa052ea77","Type":"ContainerDied","Data":"418742f11838292768a3113bd777cd3ca270581e010416f91c64e86e9fa8a6ba"}
Nov 03 20:31:01.634247 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:01.633277    5570 scope.go:122] "RemoveContainer" containerID="1f19f2144b102cc1956626a8bb852b8709eee1ead55e6bac6ab748e23cc1fe79"
Nov 03 20:31:01.634247 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:01.633468    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xhr9f"
Nov 03 20:31:01.645855 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:01.642013    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5668dee7-8a03-4a0c-ab66-303fa052ea77-istio-csr-ca-configmap" pod "5668dee7-8a03-4a0c-ab66-303fa052ea77" (UID: "5668dee7-8a03-4a0c-ab66-303fa052ea77"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:31:01.746130 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:01.733253    5570 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5668dee7-8a03-4a0c-ab66-303fa052ea77-istio-csr-ca-configmap\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:31:01.746130 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:01.736664    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5668dee7-8a03-4a0c-ab66-303fa052ea77-local-certs" pod "5668dee7-8a03-4a0c-ab66-303fa052ea77" (UID: "5668dee7-8a03-4a0c-ab66-303fa052ea77"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:31:01.746130 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:01.736675    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5668dee7-8a03-4a0c-ab66-303fa052ea77-cacerts" pod "5668dee7-8a03-4a0c-ab66-303fa052ea77" (UID: "5668dee7-8a03-4a0c-ab66-303fa052ea77"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:31:01.746130 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:01.736813    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5668dee7-8a03-4a0c-ab66-303fa052ea77-kube-api-access-xzx8m" pod "5668dee7-8a03-4a0c-ab66-303fa052ea77" (UID: "5668dee7-8a03-4a0c-ab66-303fa052ea77"). InnerVolumeSpecName "kube-api-access-xzx8m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:31:01.746130 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:01.737283    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5668dee7-8a03-4a0c-ab66-303fa052ea77-istio-token" pod "5668dee7-8a03-4a0c-ab66-303fa052ea77" (UID: "5668dee7-8a03-4a0c-ab66-303fa052ea77"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:31:01.746130 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:01.737382    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5668dee7-8a03-4a0c-ab66-303fa052ea77-istio-kubeconfig" pod "5668dee7-8a03-4a0c-ab66-303fa052ea77" (UID: "5668dee7-8a03-4a0c-ab66-303fa052ea77"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:31:01.746130 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:01.738316    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5668dee7-8a03-4a0c-ab66-303fa052ea77-istio-csr-dns-cert" pod "5668dee7-8a03-4a0c-ab66-303fa052ea77" (UID: "5668dee7-8a03-4a0c-ab66-303fa052ea77"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:31:01.762509 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:01.762140    5570 scope.go:122] "RemoveContainer" containerID="1f19f2144b102cc1956626a8bb852b8709eee1ead55e6bac6ab748e23cc1fe79"
Nov 03 20:31:01.762868 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:31:01.762753    5570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f19f2144b102cc1956626a8bb852b8709eee1ead55e6bac6ab748e23cc1fe79\": container with ID starting with 1f19f2144b102cc1956626a8bb852b8709eee1ead55e6bac6ab748e23cc1fe79 not found: ID does not exist" containerID="1f19f2144b102cc1956626a8bb852b8709eee1ead55e6bac6ab748e23cc1fe79"
Nov 03 20:31:01.762868 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:01.762817    5570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f19f2144b102cc1956626a8bb852b8709eee1ead55e6bac6ab748e23cc1fe79"} err="failed to get container status \"1f19f2144b102cc1956626a8bb852b8709eee1ead55e6bac6ab748e23cc1fe79\": rpc error: code = NotFound desc = could not find container \"1f19f2144b102cc1956626a8bb852b8709eee1ead55e6bac6ab748e23cc1fe79\": container with ID starting with 1f19f2144b102cc1956626a8bb852b8709eee1ead55e6bac6ab748e23cc1fe79 not found: ID does not exist"
Nov 03 20:31:01.835764 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:01.835718    5570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5668dee7-8a03-4a0c-ab66-303fa052ea77-istio-token\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:31:01.835764 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:01.835760    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xzx8m\" (UniqueName: \"kubernetes.io/projected/5668dee7-8a03-4a0c-ab66-303fa052ea77-kube-api-access-xzx8m\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:31:01.836072 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:01.835816    5570 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5668dee7-8a03-4a0c-ab66-303fa052ea77-local-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:31:01.836072 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:01.835829    5570 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5668dee7-8a03-4a0c-ab66-303fa052ea77-cacerts\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:31:01.836072 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:01.835847    5570 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5668dee7-8a03-4a0c-ab66-303fa052ea77-istio-csr-dns-cert\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:31:01.836072 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:01.835858    5570 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5668dee7-8a03-4a0c-ab66-303fa052ea77-istio-kubeconfig\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:31:02.074760 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:02.074710    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xhr9f"]
Nov 03 20:31:02.080899 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:02.080861    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xhr9f"]
Nov 03 20:31:02.323185 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:02.323087    5570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-addf1a-5584bbb5cb-tv2db" podUID="16b1d5c2-f657-4041-b4e2-582d074725bd" containerName="payload-processing" containerID="cri-o://1ee01d63ad2321edbf20229db31c8f2d0858af09c650d151f1adaa61b5494ce0" gracePeriod=25
Nov 03 20:31:02.330092 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:02.329836    5570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-addf1a-857995db79-q4sn9" podUID="87cae337-afa6-4e03-99b6-2f0ff53c985a" containerName="payload-pre-processing" containerID="cri-o://aac616e6892e874ea3c74677d75ab45da1b4eedf93284aa5bd0fdcf17c3d996c" gracePeriod=25
Nov 03 20:31:02.797296 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:02.797264    5570 generic.go:356] "Generic (PLEG): container finished" podID="16b1d5c2-f657-4041-b4e2-582d074725bd" containerID="1ee01d63ad2321edbf20229db31c8f2d0858af09c650d151f1adaa61b5494ce0" exitCode=0
Nov 03 20:31:02.807330 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:02.807132    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-addf1a-5584bbb5cb-tv2db" event={"ID":"16b1d5c2-f657-4041-b4e2-582d074725bd","Type":"ContainerDied","Data":"1ee01d63ad2321edbf20229db31c8f2d0858af09c650d151f1adaa61b5494ce0"}
Nov 03 20:31:02.817296 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:02.817251    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-addf1a-857995db79-q4sn9" event={"ID":"87cae337-afa6-4e03-99b6-2f0ff53c985a","Type":"ContainerDied","Data":"aac616e6892e874ea3c74677d75ab45da1b4eedf93284aa5bd0fdcf17c3d996c"}
Nov 03 20:31:02.832412 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:02.816965    5570 generic.go:356] "Generic (PLEG): container finished" podID="87cae337-afa6-4e03-99b6-2f0ff53c985a" containerID="aac616e6892e874ea3c74677d75ab45da1b4eedf93284aa5bd0fdcf17c3d996c" exitCode=0
Nov 03 20:31:03.152363 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:03.152328    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-addf1a-5584bbb5cb-tv2db"
Nov 03 20:31:03.163602 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:03.163483    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-addf1a-857995db79-q4sn9"
Nov 03 20:31:03.275353 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:03.274230    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16b1d5c2-f657-4041-b4e2-582d074725bd-kube-api-access-nbqbl\" (UniqueName: \"kubernetes.io/projected/16b1d5c2-f657-4041-b4e2-582d074725bd-kube-api-access-nbqbl\") pod \"16b1d5c2-f657-4041-b4e2-582d074725bd\" (UID: \"16b1d5c2-f657-4041-b4e2-582d074725bd\") "
Nov 03 20:31:03.275353 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:03.274360    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87cae337-afa6-4e03-99b6-2f0ff53c985a-kube-api-access-nx6lz\" (UniqueName: \"kubernetes.io/projected/87cae337-afa6-4e03-99b6-2f0ff53c985a-kube-api-access-nx6lz\") pod \"87cae337-afa6-4e03-99b6-2f0ff53c985a\" (UID: \"87cae337-afa6-4e03-99b6-2f0ff53c985a\") "
Nov 03 20:31:03.275353 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:03.274559    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87cae337-afa6-4e03-99b6-2f0ff53c985a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87cae337-afa6-4e03-99b6-2f0ff53c985a-plugins-config-volume\") pod \"87cae337-afa6-4e03-99b6-2f0ff53c985a\" (UID: \"87cae337-afa6-4e03-99b6-2f0ff53c985a\") "
Nov 03 20:31:03.275353 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:03.274663    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16b1d5c2-f657-4041-b4e2-582d074725bd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16b1d5c2-f657-4041-b4e2-582d074725bd-plugins-config-volume\") pod \"16b1d5c2-f657-4041-b4e2-582d074725bd\" (UID: \"16b1d5c2-f657-4041-b4e2-582d074725bd\") "
Nov 03 20:31:03.278956 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:03.276517    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16b1d5c2-f657-4041-b4e2-582d074725bd-plugins-config-volume" pod "16b1d5c2-f657-4041-b4e2-582d074725bd" (UID: "16b1d5c2-f657-4041-b4e2-582d074725bd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:31:03.278956 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:03.276695    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87cae337-afa6-4e03-99b6-2f0ff53c985a-plugins-config-volume" pod "87cae337-afa6-4e03-99b6-2f0ff53c985a" (UID: "87cae337-afa6-4e03-99b6-2f0ff53c985a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:31:03.284469 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:03.284425    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87cae337-afa6-4e03-99b6-2f0ff53c985a-kube-api-access-nx6lz" pod "87cae337-afa6-4e03-99b6-2f0ff53c985a" (UID: "87cae337-afa6-4e03-99b6-2f0ff53c985a"). InnerVolumeSpecName "kube-api-access-nx6lz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:31:03.285126 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:03.285054    5570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16b1d5c2-f657-4041-b4e2-582d074725bd-plugins-config-volume\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:31:03.285126 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:03.285083    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nx6lz\" (UniqueName: \"kubernetes.io/projected/87cae337-afa6-4e03-99b6-2f0ff53c985a-kube-api-access-nx6lz\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:31:03.285126 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:03.285101    5570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87cae337-afa6-4e03-99b6-2f0ff53c985a-plugins-config-volume\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:31:03.323184 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:03.321621    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16b1d5c2-f657-4041-b4e2-582d074725bd-kube-api-access-nbqbl" pod "16b1d5c2-f657-4041-b4e2-582d074725bd" (UID: "16b1d5c2-f657-4041-b4e2-582d074725bd"). InnerVolumeSpecName "kube-api-access-nbqbl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:31:03.388406 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:03.388366    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nbqbl\" (UniqueName: \"kubernetes.io/projected/16b1d5c2-f657-4041-b4e2-582d074725bd-kube-api-access-nbqbl\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:31:03.846860 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:03.844796    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-addf1a-857995db79-q4sn9"
Nov 03 20:31:03.846860 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:03.844804    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-addf1a-857995db79-q4sn9" event={"ID":"87cae337-afa6-4e03-99b6-2f0ff53c985a","Type":"ContainerDied","Data":"75af2602a7363e8868dd4c52eae28a369b313c49f2348a848f2a0fa8c6bd87c4"}
Nov 03 20:31:03.846860 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:03.844961    5570 scope.go:122] "RemoveContainer" containerID="aac616e6892e874ea3c74677d75ab45da1b4eedf93284aa5bd0fdcf17c3d996c"
Nov 03 20:31:03.847635 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:03.847006    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-addf1a-5584bbb5cb-tv2db"
Nov 03 20:31:03.847635 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:03.847089    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-addf1a-5584bbb5cb-tv2db" event={"ID":"16b1d5c2-f657-4041-b4e2-582d074725bd","Type":"ContainerDied","Data":"98c4d23a5c7a6918078c7ae1b09f8cfc8e02181235903c9df710f3037bf0fc51"}
Nov 03 20:31:03.875193 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:03.875166    5570 scope.go:122] "RemoveContainer" containerID="1ee01d63ad2321edbf20229db31c8f2d0858af09c650d151f1adaa61b5494ce0"
Nov 03 20:31:03.924470 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:03.924381    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-addf1a-857995db79-q4sn9"]
Nov 03 20:31:03.936228 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:03.936189    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-addf1a-857995db79-q4sn9"]
Nov 03 20:31:03.970584 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:03.970547    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5668dee7-8a03-4a0c-ab66-303fa052ea77" path="/var/lib/kubelet/pods/5668dee7-8a03-4a0c-ab66-303fa052ea77/volumes"
Nov 03 20:31:03.971120 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:03.971104    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87cae337-afa6-4e03-99b6-2f0ff53c985a" path="/var/lib/kubelet/pods/87cae337-afa6-4e03-99b6-2f0ff53c985a/volumes"
Nov 03 20:31:03.971470 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:03.971455    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-addf1a-5584bbb5cb-tv2db"]
Nov 03 20:31:03.971514 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:03.971489    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-addf1a-5584bbb5cb-tv2db"]
Nov 03 20:31:04.479464 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:04.479429    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-60efdf-openshift-default-5dfc484849-k6445"]
Nov 03 20:31:04.480342 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:04.480316    5570 cpu_manager.go:414] "containerMap: removing container" podUID="5668dee7-8a03-4a0c-ab66-303fa052ea77" containerName="discovery"
Nov 03 20:31:04.480342 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:04.480342    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5668dee7-8a03-4a0c-ab66-303fa052ea77" containerName="discovery"
Nov 03 20:31:04.480543 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:04.480361    5570 cpu_manager.go:414] "containerMap: removing container" podUID="16b1d5c2-f657-4041-b4e2-582d074725bd" containerName="payload-processing"
Nov 03 20:31:04.480543 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:04.480369    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16b1d5c2-f657-4041-b4e2-582d074725bd" containerName="payload-processing"
Nov 03 20:31:04.480543 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:04.480392    5570 cpu_manager.go:414] "containerMap: removing container" podUID="87cae337-afa6-4e03-99b6-2f0ff53c985a" containerName="payload-pre-processing"
Nov 03 20:31:04.480543 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:04.480399    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87cae337-afa6-4e03-99b6-2f0ff53c985a" containerName="payload-pre-processing"
Nov 03 20:31:04.480543 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:04.480414    5570 cpu_manager.go:414] "containerMap: removing container" podUID="2c6d9548-c1f6-4efb-969b-260308e4ac3a" containerName="maas-api"
Nov 03 20:31:04.480543 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:04.480420    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c6d9548-c1f6-4efb-969b-260308e4ac3a" containerName="maas-api"
Nov 03 20:31:04.480884 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:04.480650    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="5668dee7-8a03-4a0c-ab66-303fa052ea77" containerName="discovery"
Nov 03 20:31:04.480884 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:04.480664    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="16b1d5c2-f657-4041-b4e2-582d074725bd" containerName="payload-processing"
Nov 03 20:31:04.480884 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:04.480677    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="87cae337-afa6-4e03-99b6-2f0ff53c985a" containerName="payload-pre-processing"
Nov 03 20:31:04.480884 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:04.480685    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c6d9548-c1f6-4efb-969b-260308e4ac3a" containerName="maas-api"
Nov 03 20:31:04.576140 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:04.576099    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-60efdf-openshift-default-5dfc484849-k6445"]
Nov 03 20:31:04.576322 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:04.576240    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-60efdf-openshift-default-5dfc484849-k6445"
Nov 03 20:31:04.592551 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:04.592524    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-60efdf-openshift-default-dockercfg-289b9\""
Nov 03 20:31:04.745889 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:04.723993    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6fb7442-714a-4e4a-a6cf-19326b50a929-istio-data\") pod \"e2e-inf-a-60efdf-openshift-default-5dfc484849-k6445\" (UID: \"c6fb7442-714a-4e4a-a6cf-19326b50a929\") " pod="openshift-ingress/e2e-inf-a-60efdf-openshift-default-5dfc484849-k6445"
Nov 03 20:31:04.745889 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:04.724055    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6fb7442-714a-4e4a-a6cf-19326b50a929-istio-podinfo\") pod \"e2e-inf-a-60efdf-openshift-default-5dfc484849-k6445\" (UID: \"c6fb7442-714a-4e4a-a6cf-19326b50a929\") " pod="openshift-ingress/e2e-inf-a-60efdf-openshift-default-5dfc484849-k6445"
Nov 03 20:31:04.745889 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:04.724089    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6fb7442-714a-4e4a-a6cf-19326b50a929-istio-token\") pod \"e2e-inf-a-60efdf-openshift-default-5dfc484849-k6445\" (UID: \"c6fb7442-714a-4e4a-a6cf-19326b50a929\") " pod="openshift-ingress/e2e-inf-a-60efdf-openshift-default-5dfc484849-k6445"
Nov 03 20:31:04.745889 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:04.724113    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6fb7442-714a-4e4a-a6cf-19326b50a929-workload-socket\") pod \"e2e-inf-a-60efdf-openshift-default-5dfc484849-k6445\" (UID: \"c6fb7442-714a-4e4a-a6cf-19326b50a929\") " pod="openshift-ingress/e2e-inf-a-60efdf-openshift-default-5dfc484849-k6445"
Nov 03 20:31:04.745889 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:04.724260    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6fb7442-714a-4e4a-a6cf-19326b50a929-credential-socket\") pod \"e2e-inf-a-60efdf-openshift-default-5dfc484849-k6445\" (UID: \"c6fb7442-714a-4e4a-a6cf-19326b50a929\") " pod="openshift-ingress/e2e-inf-a-60efdf-openshift-default-5dfc484849-k6445"
Nov 03 20:31:04.745889 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:04.724380    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbmj4\" (UniqueName: \"kubernetes.io/projected/c6fb7442-714a-4e4a-a6cf-19326b50a929-kube-api-access-lbmj4\") pod \"e2e-inf-a-60efdf-openshift-default-5dfc484849-k6445\" (UID: \"c6fb7442-714a-4e4a-a6cf-19326b50a929\") " pod="openshift-ingress/e2e-inf-a-60efdf-openshift-default-5dfc484849-k6445"
Nov 03 20:31:04.745889 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:04.724570    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6fb7442-714a-4e4a-a6cf-19326b50a929-istio-envoy\") pod \"e2e-inf-a-60efdf-openshift-default-5dfc484849-k6445\" (UID: \"c6fb7442-714a-4e4a-a6cf-19326b50a929\") " pod="openshift-ingress/e2e-inf-a-60efdf-openshift-default-5dfc484849-k6445"
Nov 03 20:31:04.745889 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:04.724664    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6fb7442-714a-4e4a-a6cf-19326b50a929-istiod-ca-cert\") pod \"e2e-inf-a-60efdf-openshift-default-5dfc484849-k6445\" (UID: \"c6fb7442-714a-4e4a-a6cf-19326b50a929\") " pod="openshift-ingress/e2e-inf-a-60efdf-openshift-default-5dfc484849-k6445"
Nov 03 20:31:04.745889 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:04.724724    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6fb7442-714a-4e4a-a6cf-19326b50a929-workload-certs\") pod \"e2e-inf-a-60efdf-openshift-default-5dfc484849-k6445\" (UID: \"c6fb7442-714a-4e4a-a6cf-19326b50a929\") " pod="openshift-ingress/e2e-inf-a-60efdf-openshift-default-5dfc484849-k6445"
Nov 03 20:31:04.828800 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:04.827562    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6fb7442-714a-4e4a-a6cf-19326b50a929-credential-socket\") pod \"e2e-inf-a-60efdf-openshift-default-5dfc484849-k6445\" (UID: \"c6fb7442-714a-4e4a-a6cf-19326b50a929\") " pod="openshift-ingress/e2e-inf-a-60efdf-openshift-default-5dfc484849-k6445"
Nov 03 20:31:04.828800 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:04.827711    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbmj4\" (UniqueName: \"kubernetes.io/projected/c6fb7442-714a-4e4a-a6cf-19326b50a929-kube-api-access-lbmj4\") pod \"e2e-inf-a-60efdf-openshift-default-5dfc484849-k6445\" (UID: \"c6fb7442-714a-4e4a-a6cf-19326b50a929\") " pod="openshift-ingress/e2e-inf-a-60efdf-openshift-default-5dfc484849-k6445"
Nov 03 20:31:04.828800 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:04.827869    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6fb7442-714a-4e4a-a6cf-19326b50a929-istio-envoy\") pod \"e2e-inf-a-60efdf-openshift-default-5dfc484849-k6445\" (UID: \"c6fb7442-714a-4e4a-a6cf-19326b50a929\") " pod="openshift-ingress/e2e-inf-a-60efdf-openshift-default-5dfc484849-k6445"
Nov 03 20:31:04.828800 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:04.827952    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6fb7442-714a-4e4a-a6cf-19326b50a929-istiod-ca-cert\") pod \"e2e-inf-a-60efdf-openshift-default-5dfc484849-k6445\" (UID: \"c6fb7442-714a-4e4a-a6cf-19326b50a929\") " pod="openshift-ingress/e2e-inf-a-60efdf-openshift-default-5dfc484849-k6445"
Nov 03 20:31:04.828800 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:04.828006    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6fb7442-714a-4e4a-a6cf-19326b50a929-workload-certs\") pod \"e2e-inf-a-60efdf-openshift-default-5dfc484849-k6445\" (UID: \"c6fb7442-714a-4e4a-a6cf-19326b50a929\") " pod="openshift-ingress/e2e-inf-a-60efdf-openshift-default-5dfc484849-k6445"
Nov 03 20:31:04.828800 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:04.828096    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6fb7442-714a-4e4a-a6cf-19326b50a929-istio-data\") pod \"e2e-inf-a-60efdf-openshift-default-5dfc484849-k6445\" (UID: \"c6fb7442-714a-4e4a-a6cf-19326b50a929\") " pod="openshift-ingress/e2e-inf-a-60efdf-openshift-default-5dfc484849-k6445"
Nov 03 20:31:04.828800 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:04.828124    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6fb7442-714a-4e4a-a6cf-19326b50a929-istio-podinfo\") pod \"e2e-inf-a-60efdf-openshift-default-5dfc484849-k6445\" (UID: \"c6fb7442-714a-4e4a-a6cf-19326b50a929\") " pod="openshift-ingress/e2e-inf-a-60efdf-openshift-default-5dfc484849-k6445"
Nov 03 20:31:04.828800 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:04.828160    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6fb7442-714a-4e4a-a6cf-19326b50a929-istio-token\") pod \"e2e-inf-a-60efdf-openshift-default-5dfc484849-k6445\" (UID: \"c6fb7442-714a-4e4a-a6cf-19326b50a929\") " pod="openshift-ingress/e2e-inf-a-60efdf-openshift-default-5dfc484849-k6445"
Nov 03 20:31:04.828800 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:04.828183    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6fb7442-714a-4e4a-a6cf-19326b50a929-workload-socket\") pod \"e2e-inf-a-60efdf-openshift-default-5dfc484849-k6445\" (UID: \"c6fb7442-714a-4e4a-a6cf-19326b50a929\") " pod="openshift-ingress/e2e-inf-a-60efdf-openshift-default-5dfc484849-k6445"
Nov 03 20:31:04.829594 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:04.828909    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6fb7442-714a-4e4a-a6cf-19326b50a929-workload-socket\") pod \"e2e-inf-a-60efdf-openshift-default-5dfc484849-k6445\" (UID: \"c6fb7442-714a-4e4a-a6cf-19326b50a929\") " pod="openshift-ingress/e2e-inf-a-60efdf-openshift-default-5dfc484849-k6445"
Nov 03 20:31:04.829594 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:04.829168    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6fb7442-714a-4e4a-a6cf-19326b50a929-credential-socket\") pod \"e2e-inf-a-60efdf-openshift-default-5dfc484849-k6445\" (UID: \"c6fb7442-714a-4e4a-a6cf-19326b50a929\") " pod="openshift-ingress/e2e-inf-a-60efdf-openshift-default-5dfc484849-k6445"
Nov 03 20:31:04.830344 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:04.830054    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6fb7442-714a-4e4a-a6cf-19326b50a929-istio-data\") pod \"e2e-inf-a-60efdf-openshift-default-5dfc484849-k6445\" (UID: \"c6fb7442-714a-4e4a-a6cf-19326b50a929\") " pod="openshift-ingress/e2e-inf-a-60efdf-openshift-default-5dfc484849-k6445"
Nov 03 20:31:04.834806 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:04.830687    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6fb7442-714a-4e4a-a6cf-19326b50a929-istiod-ca-cert\") pod \"e2e-inf-a-60efdf-openshift-default-5dfc484849-k6445\" (UID: \"c6fb7442-714a-4e4a-a6cf-19326b50a929\") " pod="openshift-ingress/e2e-inf-a-60efdf-openshift-default-5dfc484849-k6445"
Nov 03 20:31:04.853798 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:04.851736    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6fb7442-714a-4e4a-a6cf-19326b50a929-workload-certs\") pod \"e2e-inf-a-60efdf-openshift-default-5dfc484849-k6445\" (UID: \"c6fb7442-714a-4e4a-a6cf-19326b50a929\") " pod="openshift-ingress/e2e-inf-a-60efdf-openshift-default-5dfc484849-k6445"
Nov 03 20:31:04.941797 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:04.901355    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6fb7442-714a-4e4a-a6cf-19326b50a929-istio-podinfo\") pod \"e2e-inf-a-60efdf-openshift-default-5dfc484849-k6445\" (UID: \"c6fb7442-714a-4e4a-a6cf-19326b50a929\") " pod="openshift-ingress/e2e-inf-a-60efdf-openshift-default-5dfc484849-k6445"
Nov 03 20:31:04.941797 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:04.906439    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6fb7442-714a-4e4a-a6cf-19326b50a929-istio-envoy\") pod \"e2e-inf-a-60efdf-openshift-default-5dfc484849-k6445\" (UID: \"c6fb7442-714a-4e4a-a6cf-19326b50a929\") " pod="openshift-ingress/e2e-inf-a-60efdf-openshift-default-5dfc484849-k6445"
Nov 03 20:31:04.941797 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:04.908233    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6fb7442-714a-4e4a-a6cf-19326b50a929-istio-token\") pod \"e2e-inf-a-60efdf-openshift-default-5dfc484849-k6445\" (UID: \"c6fb7442-714a-4e4a-a6cf-19326b50a929\") " pod="openshift-ingress/e2e-inf-a-60efdf-openshift-default-5dfc484849-k6445"
Nov 03 20:31:04.975078 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:04.975004    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbmj4\" (UniqueName: \"kubernetes.io/projected/c6fb7442-714a-4e4a-a6cf-19326b50a929-kube-api-access-lbmj4\") pod \"e2e-inf-a-60efdf-openshift-default-5dfc484849-k6445\" (UID: \"c6fb7442-714a-4e4a-a6cf-19326b50a929\") " pod="openshift-ingress/e2e-inf-a-60efdf-openshift-default-5dfc484849-k6445"
Nov 03 20:31:05.229791 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:05.229731    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-60efdf-openshift-default-5dfc484849-k6445"
Nov 03 20:31:05.615819 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:31:05.615759    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6fb7442_714a_4e4a_a6cf_19326b50a929.slice/crio-3142a43f29671e032e8ff70ba749cd8c00c7f3930d96979dbe7456d9b8ca9461 WatchSource:0}: Error finding container 3142a43f29671e032e8ff70ba749cd8c00c7f3930d96979dbe7456d9b8ca9461: Status 404 returned error can't find the container with id 3142a43f29671e032e8ff70ba749cd8c00c7f3930d96979dbe7456d9b8ca9461
Nov 03 20:31:05.617626 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:05.617519    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-60efdf-openshift-default-5dfc484849-k6445"]
Nov 03 20:31:05.619015 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:05.618982    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:31:05.619095 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:05.619048    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:31:05.619095 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:05.619082    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:31:05.956584 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:05.956536    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-60efdf-openshift-default-5dfc484849-k6445" event={"ID":"c6fb7442-714a-4e4a-a6cf-19326b50a929","Type":"ContainerStarted","Data":"4537e19896b788980331af4346ff30e28e6227d9410b794ab6558c526b1fc121"}
Nov 03 20:31:05.957189 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:05.956616    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-60efdf-openshift-default-5dfc484849-k6445" event={"ID":"c6fb7442-714a-4e4a-a6cf-19326b50a929","Type":"ContainerStarted","Data":"3142a43f29671e032e8ff70ba749cd8c00c7f3930d96979dbe7456d9b8ca9461"}
Nov 03 20:31:05.970330 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:05.970291    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16b1d5c2-f657-4041-b4e2-582d074725bd" path="/var/lib/kubelet/pods/16b1d5c2-f657-4041-b4e2-582d074725bd/volumes"
Nov 03 20:31:06.011357 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:06.011284    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-60efdf-openshift-default-5dfc484849-k6445" podStartSLOduration=2.011262245 podStartE2EDuration="2.011262245s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:31:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:31:06.004876523 +0000 UTC m=+1600.157538413" watchObservedRunningTime="2026-11-03 20:31:06.011262245 +0000 UTC m=+1600.163924133"
Nov 03 20:31:06.253879 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:06.239721    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-60efdf-openshift-default-5dfc484849-k6445"
Nov 03 20:31:06.253879 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:06.239931    5570 patch_prober.go:29] interesting pod/e2e-inf-a-60efdf-openshift-default-5dfc484849-k6445 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body=
Nov 03 20:31:06.253879 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:06.240010    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-60efdf-openshift-default-5dfc484849-k6445" podUID="c6fb7442-714a-4e4a-a6cf-19326b50a929" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused"
Nov 03 20:31:07.277737 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:07.277656    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-60efdf-openshift-default-5dfc484849-k6445"
Nov 03 20:31:07.974061 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:07.974003    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-60efdf-openshift-default-5dfc484849-k6445"
Nov 03 20:31:07.975751 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:07.975719    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-60efdf-openshift-default-5dfc484849-k6445"
Nov 03 20:31:09.078864 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:09.061214    5570 generic.go:356] "Generic (PLEG): container finished" podID="1f5aeb82-01f6-483f-8273-10bd0714e03c" containerID="17e044fcfdd8d20d3cc3fa7ec5897d1ad22ba6fcf7a0f47f100303c23ff48798" exitCode=0
Nov 03 20:31:09.078864 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:09.061397    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-7e4389-openshift-default-5b5bd7f55-kpk7n" event={"ID":"1f5aeb82-01f6-483f-8273-10bd0714e03c","Type":"ContainerDied","Data":"17e044fcfdd8d20d3cc3fa7ec5897d1ad22ba6fcf7a0f47f100303c23ff48798"}
Nov 03 20:31:09.078864 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:09.062119    5570 scope.go:122] "RemoveContainer" containerID="17e044fcfdd8d20d3cc3fa7ec5897d1ad22ba6fcf7a0f47f100303c23ff48798"
Nov 03 20:31:09.078864 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:09.072383    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:31:09.078864 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:09.072486    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:31:09.078864 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:09.072536    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:31:09.138877 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:09.138833    5570 generic.go:356] "Generic (PLEG): container finished" podID="5a701695-c67d-4cda-a46d-7834b22750ed" containerID="8aaebcb8af28523aa8fb9da47a8dd3c1e0d18b4d1ea123656dcb34aa2d36d88c" exitCode=0
Nov 03 20:31:09.139130 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:09.138824    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d1aa66-openshift-default-8457c96657-lt2kw" event={"ID":"5a701695-c67d-4cda-a46d-7834b22750ed","Type":"ContainerDied","Data":"8aaebcb8af28523aa8fb9da47a8dd3c1e0d18b4d1ea123656dcb34aa2d36d88c"}
Nov 03 20:31:09.140368 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:09.140339    5570 scope.go:122] "RemoveContainer" containerID="8aaebcb8af28523aa8fb9da47a8dd3c1e0d18b4d1ea123656dcb34aa2d36d88c"
Nov 03 20:31:09.146340 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:09.146300    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:31:09.146497 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:09.146400    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:31:09.146497 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:09.146450    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:31:10.192419 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:10.192378    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-7e4389-openshift-default-5b5bd7f55-kpk7n" event={"ID":"1f5aeb82-01f6-483f-8273-10bd0714e03c","Type":"ContainerStarted","Data":"b24a6ddb0a915a87947f09e2e5141c213ffbeec08769fa4c0a030faf8f68be69"}
Nov 03 20:31:10.195260 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:10.195230    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d1aa66-openshift-default-8457c96657-lt2kw" event={"ID":"5a701695-c67d-4cda-a46d-7834b22750ed","Type":"ContainerStarted","Data":"7d6407f3a09e7889871e501625f155ce4ccdfbd5bf75f845f25ed94d35a2e75f"}
Nov 03 20:31:10.209207 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:10.209166    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-d1aa66-openshift-default-8457c96657-lt2kw"
Nov 03 20:31:10.210703 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:10.210676    5570 patch_prober.go:29] interesting pod/e2e-api-b-d1aa66-openshift-default-8457c96657-lt2kw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body=
Nov 03 20:31:10.210884 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:10.210723    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d1aa66-openshift-default-8457c96657-lt2kw" podUID="5a701695-c67d-4cda-a46d-7834b22750ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused"
Nov 03 20:31:10.222718 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:10.222685    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-7e4389-openshift-default-5b5bd7f55-kpk7n"
Nov 03 20:31:10.223909 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:10.223884    5570 patch_prober.go:29] interesting pod/e2e-api-b-7e4389-openshift-default-5b5bd7f55-kpk7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body=
Nov 03 20:31:10.224040 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:10.223937    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-7e4389-openshift-default-5b5bd7f55-kpk7n" podUID="1f5aeb82-01f6-483f-8273-10bd0714e03c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused"
Nov 03 20:31:11.192885 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:11.190799    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-60efdf-7bfdc67bd7-j6ctn"]
Nov 03 20:31:11.198966 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:11.198632    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-60efdf-7bfdc67bd7-j6ctn"
Nov 03 20:31:11.208517 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:11.203340    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-60efdf-7bfdc67bd7-j6ctn"]
Nov 03 20:31:11.220273 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:11.213271    5570 patch_prober.go:29] interesting pod/e2e-api-b-d1aa66-openshift-default-8457c96657-lt2kw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body=
Nov 03 20:31:11.220273 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:11.213333    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d1aa66-openshift-default-8457c96657-lt2kw" podUID="5a701695-c67d-4cda-a46d-7834b22750ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused"
Nov 03 20:31:11.223970 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:11.223920    5570 patch_prober.go:29] interesting pod/e2e-api-b-7e4389-openshift-default-5b5bd7f55-kpk7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body=
Nov 03 20:31:11.224117 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:11.223976    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-7e4389-openshift-default-5b5bd7f55-kpk7n" podUID="1f5aeb82-01f6-483f-8273-10bd0714e03c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused"
Nov 03 20:31:11.228407 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:11.228374    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-60efdf-dockercfg-xzmz6\""
Nov 03 20:31:11.228733 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:11.228708    5570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-60efdf\""
Nov 03 20:31:11.272801 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:11.257672    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-7e4389-openshift-default-5b5bd7f55-kpk7n"
Nov 03 20:31:11.299086 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:11.292839    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-gd5bg"]
Nov 03 20:31:11.299086 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:11.293345    5570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-gd5bg" podUID="0037c462-11bc-4f13-9d5a-b4caaec482f5" containerName="istio-proxy" containerID="cri-o://dfbabe837991c35498bcfdd8e6a2b1187ffc2d56f090498d2294563f36c5f235" gracePeriod=30
Nov 03 20:31:11.317124 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:11.317101    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zdsqp"]
Nov 03 20:31:11.318102 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:11.318019    5570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zdsqp" podUID="f747f534-4e45-446f-948e-c66ce316db5d" containerName="istio-proxy" containerID="cri-o://1b29b47590450ba9dbd2e2bf29919ee55c3af4cce616436f3a421e68c2808b25" gracePeriod=30
Nov 03 20:31:11.320325 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:11.320287    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/effc03b7-0b9e-4239-baa0-5c6cd120f479-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-60efdf-7bfdc67bd7-j6ctn\" (UID: \"effc03b7-0b9e-4239-baa0-5c6cd120f479\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-60efdf-7bfdc67bd7-j6ctn"
Nov 03 20:31:11.320569 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:11.320547    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkvrq\" (UniqueName: \"kubernetes.io/projected/effc03b7-0b9e-4239-baa0-5c6cd120f479-kube-api-access-jkvrq\") pod \"payload-pre-processing-e2e-inf-a-60efdf-7bfdc67bd7-j6ctn\" (UID: \"effc03b7-0b9e-4239-baa0-5c6cd120f479\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-60efdf-7bfdc67bd7-j6ctn"
Nov 03 20:31:11.342821 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:11.324921    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-60efdf-7cb6d9fbc8-fwx62"]
Nov 03 20:31:11.446898 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:11.427388    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkvrq\" (UniqueName: \"kubernetes.io/projected/effc03b7-0b9e-4239-baa0-5c6cd120f479-kube-api-access-jkvrq\") pod \"payload-pre-processing-e2e-inf-a-60efdf-7bfdc67bd7-j6ctn\" (UID: \"effc03b7-0b9e-4239-baa0-5c6cd120f479\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-60efdf-7bfdc67bd7-j6ctn"
Nov 03 20:31:11.446898 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:11.427754    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/effc03b7-0b9e-4239-baa0-5c6cd120f479-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-60efdf-7bfdc67bd7-j6ctn\" (UID: \"effc03b7-0b9e-4239-baa0-5c6cd120f479\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-60efdf-7bfdc67bd7-j6ctn"
Nov 03 20:31:11.554095 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:11.554051    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-nsjcd"]
Nov 03 20:31:11.613797 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:11.611545    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/effc03b7-0b9e-4239-baa0-5c6cd120f479-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-60efdf-7bfdc67bd7-j6ctn\" (UID: \"effc03b7-0b9e-4239-baa0-5c6cd120f479\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-60efdf-7bfdc67bd7-j6ctn"
Nov 03 20:31:11.613797 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:11.611805    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-60efdf-7cb6d9fbc8-fwx62"
Nov 03 20:31:11.652798 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:11.636874    5570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-nsjcd" podUID="7b57a22a-cf28-4099-8883-c04c61807dfa" containerName="istio-proxy" containerID="cri-o://77b59920db5fd3798fd079759ead749d52749bd7e92d88348b87940208da84d0" gracePeriod=30
Nov 03 20:31:11.883903 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:11.883802    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c03d73cd-af2e-4af4-88cb-bca5f26dda01-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-60efdf-7cb6d9fbc8-fwx62\" (UID: \"c03d73cd-af2e-4af4-88cb-bca5f26dda01\") " pod="openshift-ingress/payload-processing-e2e-inf-a-60efdf-7cb6d9fbc8-fwx62"
Nov 03 20:31:11.884393 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:11.884367    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmnqz\" (UniqueName: \"kubernetes.io/projected/c03d73cd-af2e-4af4-88cb-bca5f26dda01-kube-api-access-dmnqz\") pod \"payload-processing-e2e-inf-a-60efdf-7cb6d9fbc8-fwx62\" (UID: \"c03d73cd-af2e-4af4-88cb-bca5f26dda01\") " pod="openshift-ingress/payload-processing-e2e-inf-a-60efdf-7cb6d9fbc8-fwx62"
Nov 03 20:31:11.936803 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:11.931469    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkvrq\" (UniqueName: \"kubernetes.io/projected/effc03b7-0b9e-4239-baa0-5c6cd120f479-kube-api-access-jkvrq\") pod \"payload-pre-processing-e2e-inf-a-60efdf-7bfdc67bd7-j6ctn\" (UID: \"effc03b7-0b9e-4239-baa0-5c6cd120f479\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-60efdf-7bfdc67bd7-j6ctn"
Nov 03 20:31:11.976230 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:11.976189    5570 patch_prober.go:29] interesting pod/e2e-mt-eb87f429-openshift-default-64649dfdb5-nsjcd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 20:31:11.976521 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:11.976488    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-nsjcd" podUID="7b57a22a-cf28-4099-8883-c04c61807dfa" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 20:31:12.002792 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:12.001972    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c03d73cd-af2e-4af4-88cb-bca5f26dda01-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-60efdf-7cb6d9fbc8-fwx62\" (UID: \"c03d73cd-af2e-4af4-88cb-bca5f26dda01\") " pod="openshift-ingress/payload-processing-e2e-inf-a-60efdf-7cb6d9fbc8-fwx62"
Nov 03 20:31:12.002792 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:12.002216    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmnqz\" (UniqueName: \"kubernetes.io/projected/c03d73cd-af2e-4af4-88cb-bca5f26dda01-kube-api-access-dmnqz\") pod \"payload-processing-e2e-inf-a-60efdf-7cb6d9fbc8-fwx62\" (UID: \"c03d73cd-af2e-4af4-88cb-bca5f26dda01\") " pod="openshift-ingress/payload-processing-e2e-inf-a-60efdf-7cb6d9fbc8-fwx62"
Nov 03 20:31:12.008797 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:12.003326    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c03d73cd-af2e-4af4-88cb-bca5f26dda01-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-60efdf-7cb6d9fbc8-fwx62\" (UID: \"c03d73cd-af2e-4af4-88cb-bca5f26dda01\") " pod="openshift-ingress/payload-processing-e2e-inf-a-60efdf-7cb6d9fbc8-fwx62"
Nov 03 20:31:12.020038 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:12.020004    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-60efdf-7cb6d9fbc8-fwx62"]
Nov 03 20:31:12.055100 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:12.044884    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-d1aa66-openshift-default-8457c96657-lt2kw"
Nov 03 20:31:12.099821 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:12.098588    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmnqz\" (UniqueName: \"kubernetes.io/projected/c03d73cd-af2e-4af4-88cb-bca5f26dda01-kube-api-access-dmnqz\") pod \"payload-processing-e2e-inf-a-60efdf-7cb6d9fbc8-fwx62\" (UID: \"c03d73cd-af2e-4af4-88cb-bca5f26dda01\") " pod="openshift-ingress/payload-processing-e2e-inf-a-60efdf-7cb6d9fbc8-fwx62"
Nov 03 20:31:12.125601 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:12.125568    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-60efdf-7bfdc67bd7-j6ctn"
Nov 03 20:31:12.211614 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:12.211149    5570 patch_prober.go:29] interesting pod/e2e-api-b-d1aa66-openshift-default-8457c96657-lt2kw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body=
Nov 03 20:31:12.211614 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:12.211227    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d1aa66-openshift-default-8457c96657-lt2kw" podUID="5a701695-c67d-4cda-a46d-7834b22750ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused"
Nov 03 20:31:12.223346 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:12.223241    5570 patch_prober.go:29] interesting pod/e2e-api-b-7e4389-openshift-default-5b5bd7f55-kpk7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body=
Nov 03 20:31:12.223346 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:12.223299    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-7e4389-openshift-default-5b5bd7f55-kpk7n" podUID="1f5aeb82-01f6-483f-8273-10bd0714e03c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused"
Nov 03 20:31:12.324741 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:12.324707    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-60efdf-7cb6d9fbc8-fwx62"
Nov 03 20:31:12.598039 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:31:12.596193    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeffc03b7_0b9e_4239_baa0_5c6cd120f479.slice/crio-5e3b8247ca9ebf4f87360412db37c15b8ae96e11d5f859986b7cdf1ddc56aa15 WatchSource:0}: Error finding container 5e3b8247ca9ebf4f87360412db37c15b8ae96e11d5f859986b7cdf1ddc56aa15: Status 404 returned error can't find the container with id 5e3b8247ca9ebf4f87360412db37c15b8ae96e11d5f859986b7cdf1ddc56aa15
Nov 03 20:31:12.601588 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:12.600538    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-60efdf-7bfdc67bd7-j6ctn"]
Nov 03 20:31:12.799339 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:12.799153    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-60efdf-7cb6d9fbc8-fwx62"]
Nov 03 20:31:13.067816 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:13.064966    5570 patch_prober.go:29] interesting pod/e2e-mt-eb87f429-openshift-default-64649dfdb5-zdsqp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 20:31:13.067816 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:13.065053    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zdsqp" podUID="f747f534-4e45-446f-948e-c66ce316db5d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 20:31:13.074594 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:13.074561    5570 patch_prober.go:29] interesting pod/e2e-mt-eb87f429-openshift-default-64649dfdb5-gd5bg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 20:31:13.074874 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:13.074843    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-gd5bg" podUID="0037c462-11bc-4f13-9d5a-b4caaec482f5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 20:31:13.210120 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:13.209436    5570 patch_prober.go:29] interesting pod/e2e-api-b-d1aa66-openshift-default-8457c96657-lt2kw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body=
Nov 03 20:31:13.210120 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:13.209489    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d1aa66-openshift-default-8457c96657-lt2kw" podUID="5a701695-c67d-4cda-a46d-7834b22750ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused"
Nov 03 20:31:13.222966 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:13.222919    5570 patch_prober.go:29] interesting pod/e2e-api-b-7e4389-openshift-default-5b5bd7f55-kpk7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body=
Nov 03 20:31:13.223397 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:13.223007    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-7e4389-openshift-default-5b5bd7f55-kpk7n" podUID="1f5aeb82-01f6-483f-8273-10bd0714e03c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused"
Nov 03 20:31:13.277324 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:13.277280    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-60efdf-7cb6d9fbc8-fwx62" event={"ID":"c03d73cd-af2e-4af4-88cb-bca5f26dda01","Type":"ContainerStarted","Data":"61c5e6d2f06cee85a4001247ff9b4756cdb25498d04e3d9a402ac4eaf6e6b593"}
Nov 03 20:31:13.277630 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:13.277609    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-60efdf-7cb6d9fbc8-fwx62" event={"ID":"c03d73cd-af2e-4af4-88cb-bca5f26dda01","Type":"ContainerStarted","Data":"4e385de3d756ba5db012c4f0bf9e339ecba196cb3c6a71088aeb32c2cc89c67b"}
Nov 03 20:31:13.279091 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:13.279062    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-60efdf-7cb6d9fbc8-fwx62"
Nov 03 20:31:13.290568 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:13.290529    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-60efdf-7bfdc67bd7-j6ctn" event={"ID":"effc03b7-0b9e-4239-baa0-5c6cd120f479","Type":"ContainerStarted","Data":"2ec3f67934e86a94a7ef166e62c2b6e9cc6fcf63f2ac547d75064af34674793a"}
Nov 03 20:31:13.290944 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:13.290920    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-60efdf-7bfdc67bd7-j6ctn"
Nov 03 20:31:13.291100 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:13.291084    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-60efdf-7bfdc67bd7-j6ctn" event={"ID":"effc03b7-0b9e-4239-baa0-5c6cd120f479","Type":"ContainerStarted","Data":"5e3b8247ca9ebf4f87360412db37c15b8ae96e11d5f859986b7cdf1ddc56aa15"}
Nov 03 20:31:13.324543 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:13.324424    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-60efdf-7cb6d9fbc8-fwx62" podStartSLOduration=2.324406724 podStartE2EDuration="2.324406724s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:31:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:31:13.323105958 +0000 UTC m=+1607.475767871" watchObservedRunningTime="2026-11-03 20:31:13.324406724 +0000 UTC m=+1607.477068659"
Nov 03 20:31:13.394239 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:13.394172    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-60efdf-7bfdc67bd7-j6ctn" podStartSLOduration=2.394152278 podStartE2EDuration="2.394152278s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:31:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:31:13.391614886 +0000 UTC m=+1607.544276828" watchObservedRunningTime="2026-11-03 20:31:13.394152278 +0000 UTC m=+1607.546814166"
Nov 03 20:31:14.225456 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:14.223741    5570 patch_prober.go:29] interesting pod/e2e-api-b-d1aa66-openshift-default-8457c96657-lt2kw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body=
Nov 03 20:31:14.225456 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:14.223824    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d1aa66-openshift-default-8457c96657-lt2kw" podUID="5a701695-c67d-4cda-a46d-7834b22750ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused"
Nov 03 20:31:14.225456 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:14.224246    5570 patch_prober.go:29] interesting pod/e2e-api-b-7e4389-openshift-default-5b5bd7f55-kpk7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body=
Nov 03 20:31:14.225456 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:14.224283    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-7e4389-openshift-default-5b5bd7f55-kpk7n" podUID="1f5aeb82-01f6-483f-8273-10bd0714e03c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused"
Nov 03 20:31:14.397987 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:14.397950    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d05604-5c69f8bc7-8cb6g"]
Nov 03 20:31:14.403116 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:14.403081    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d05604-5c69f8bc7-8cb6g"
Nov 03 20:31:14.425303 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:14.425183    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-d05604-serving-cert\""
Nov 03 20:31:14.507343 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:14.505966    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d05604-5c69f8bc7-8cb6g"]
Nov 03 20:31:14.528560 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:14.528523    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55419b15-d4a1-4344-a5d2-e995ccbd2487-maas-api-tls\") pod \"maas-api-e2e-api-b-d05604-5c69f8bc7-8cb6g\" (UID: \"55419b15-d4a1-4344-a5d2-e995ccbd2487\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d05604-5c69f8bc7-8cb6g"
Nov 03 20:31:14.528748 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:14.528688    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55419b15-d4a1-4344-a5d2-e995ccbd2487-openshift-service-ca\") pod \"maas-api-e2e-api-b-d05604-5c69f8bc7-8cb6g\" (UID: \"55419b15-d4a1-4344-a5d2-e995ccbd2487\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d05604-5c69f8bc7-8cb6g"
Nov 03 20:31:14.528861 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:14.528847    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55419b15-d4a1-4344-a5d2-e995ccbd2487-metrics-certs\") pod \"maas-api-e2e-api-b-d05604-5c69f8bc7-8cb6g\" (UID: \"55419b15-d4a1-4344-a5d2-e995ccbd2487\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d05604-5c69f8bc7-8cb6g"
Nov 03 20:31:14.528899 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:14.528884    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxnbc\" (UniqueName: \"kubernetes.io/projected/55419b15-d4a1-4344-a5d2-e995ccbd2487-kube-api-access-nxnbc\") pod \"maas-api-e2e-api-b-d05604-5c69f8bc7-8cb6g\" (UID: \"55419b15-d4a1-4344-a5d2-e995ccbd2487\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d05604-5c69f8bc7-8cb6g"
Nov 03 20:31:14.638807 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:14.638534    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55419b15-d4a1-4344-a5d2-e995ccbd2487-maas-api-tls\") pod \"maas-api-e2e-api-b-d05604-5c69f8bc7-8cb6g\" (UID: \"55419b15-d4a1-4344-a5d2-e995ccbd2487\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d05604-5c69f8bc7-8cb6g"
Nov 03 20:31:14.638807 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:14.638706    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55419b15-d4a1-4344-a5d2-e995ccbd2487-openshift-service-ca\") pod \"maas-api-e2e-api-b-d05604-5c69f8bc7-8cb6g\" (UID: \"55419b15-d4a1-4344-a5d2-e995ccbd2487\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d05604-5c69f8bc7-8cb6g"
Nov 03 20:31:14.639083 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:14.638898    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55419b15-d4a1-4344-a5d2-e995ccbd2487-metrics-certs\") pod \"maas-api-e2e-api-b-d05604-5c69f8bc7-8cb6g\" (UID: \"55419b15-d4a1-4344-a5d2-e995ccbd2487\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d05604-5c69f8bc7-8cb6g"
Nov 03 20:31:14.639083 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:14.638946    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxnbc\" (UniqueName: \"kubernetes.io/projected/55419b15-d4a1-4344-a5d2-e995ccbd2487-kube-api-access-nxnbc\") pod \"maas-api-e2e-api-b-d05604-5c69f8bc7-8cb6g\" (UID: \"55419b15-d4a1-4344-a5d2-e995ccbd2487\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d05604-5c69f8bc7-8cb6g"
Nov 03 20:31:14.654859 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:14.643596    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55419b15-d4a1-4344-a5d2-e995ccbd2487-openshift-service-ca\") pod \"maas-api-e2e-api-b-d05604-5c69f8bc7-8cb6g\" (UID: \"55419b15-d4a1-4344-a5d2-e995ccbd2487\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d05604-5c69f8bc7-8cb6g"
Nov 03 20:31:14.674618 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:14.674578    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxnbc\" (UniqueName: \"kubernetes.io/projected/55419b15-d4a1-4344-a5d2-e995ccbd2487-kube-api-access-nxnbc\") pod \"maas-api-e2e-api-b-d05604-5c69f8bc7-8cb6g\" (UID: \"55419b15-d4a1-4344-a5d2-e995ccbd2487\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d05604-5c69f8bc7-8cb6g"
Nov 03 20:31:14.685201 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:14.685133    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55419b15-d4a1-4344-a5d2-e995ccbd2487-maas-api-tls\") pod \"maas-api-e2e-api-b-d05604-5c69f8bc7-8cb6g\" (UID: \"55419b15-d4a1-4344-a5d2-e995ccbd2487\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d05604-5c69f8bc7-8cb6g"
Nov 03 20:31:14.685948 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:14.685915    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55419b15-d4a1-4344-a5d2-e995ccbd2487-metrics-certs\") pod \"maas-api-e2e-api-b-d05604-5c69f8bc7-8cb6g\" (UID: \"55419b15-d4a1-4344-a5d2-e995ccbd2487\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d05604-5c69f8bc7-8cb6g"
Nov 03 20:31:14.726519 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:14.726201    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d05604-5c69f8bc7-8cb6g"
Nov 03 20:31:15.209449 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:15.209388    5570 patch_prober.go:29] interesting pod/e2e-api-b-d1aa66-openshift-default-8457c96657-lt2kw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body=
Nov 03 20:31:15.209653 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:15.209478    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d1aa66-openshift-default-8457c96657-lt2kw" podUID="5a701695-c67d-4cda-a46d-7834b22750ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused"
Nov 03 20:31:15.223927 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:15.223885    5570 patch_prober.go:29] interesting pod/e2e-api-b-7e4389-openshift-default-5b5bd7f55-kpk7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body=
Nov 03 20:31:15.224224 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:15.224193    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-7e4389-openshift-default-5b5bd7f55-kpk7n" podUID="1f5aeb82-01f6-483f-8273-10bd0714e03c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused"
Nov 03 20:31:15.235478 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:15.235447    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d05604-5c69f8bc7-8cb6g"]
Nov 03 20:31:15.235998 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:31:15.235965    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55419b15_d4a1_4344_a5d2_e995ccbd2487.slice/crio-9886a427233b83be189be394640239456b03a30538a88a984b3d5f5aeb25413d WatchSource:0}: Error finding container 9886a427233b83be189be394640239456b03a30538a88a984b3d5f5aeb25413d: Status 404 returned error can't find the container with id 9886a427233b83be189be394640239456b03a30538a88a984b3d5f5aeb25413d
Nov 03 20:31:15.269328 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:15.260715    5570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 20:31:15.321179 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:15.321137    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d05604-5c69f8bc7-8cb6g" event={"ID":"55419b15-d4a1-4344-a5d2-e995ccbd2487","Type":"ContainerStarted","Data":"9886a427233b83be189be394640239456b03a30538a88a984b3d5f5aeb25413d"}
Nov 03 20:31:16.209717 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:16.209676    5570 patch_prober.go:29] interesting pod/e2e-api-b-d1aa66-openshift-default-8457c96657-lt2kw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body=
Nov 03 20:31:16.209970 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:16.209748    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d1aa66-openshift-default-8457c96657-lt2kw" podUID="5a701695-c67d-4cda-a46d-7834b22750ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused"
Nov 03 20:31:16.223425 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:16.223380    5570 patch_prober.go:29] interesting pod/e2e-api-b-7e4389-openshift-default-5b5bd7f55-kpk7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body=
Nov 03 20:31:16.223621 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:16.223454    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-7e4389-openshift-default-5b5bd7f55-kpk7n" podUID="1f5aeb82-01f6-483f-8273-10bd0714e03c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused"
Nov 03 20:31:16.334922 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:16.334826    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d05604-5c69f8bc7-8cb6g" event={"ID":"55419b15-d4a1-4344-a5d2-e995ccbd2487","Type":"ContainerStarted","Data":"a1f50f02c65c3ea829bf7d26c4dc911c5d7b10280db8901df377c0c630197b22"}
Nov 03 20:31:16.336037 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:16.336012    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d05604-5c69f8bc7-8cb6g"
Nov 03 20:31:16.394407 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:16.394348    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d05604-5c69f8bc7-8cb6g" podStartSLOduration=1.944254358 podStartE2EDuration="2.394332123s" totalImagesPullingTime="450.077765ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:31:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:31:16.389018094 +0000 UTC m=+1610.541679981" watchObservedRunningTime="2026-11-03 20:31:16.394332123 +0000 UTC m=+1610.546994054"
Nov 03 20:31:16.731849 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:16.731812    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-gd5bg"
Nov 03 20:31:16.856185 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:16.826335    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0037c462-11bc-4f13-9d5a-b4caaec482f5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0037c462-11bc-4f13-9d5a-b4caaec482f5-istiod-ca-cert\") pod \"0037c462-11bc-4f13-9d5a-b4caaec482f5\" (UID: \"0037c462-11bc-4f13-9d5a-b4caaec482f5\") "
Nov 03 20:31:16.856185 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:16.826516    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0037c462-11bc-4f13-9d5a-b4caaec482f5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0037c462-11bc-4f13-9d5a-b4caaec482f5-istio-data\") pod \"0037c462-11bc-4f13-9d5a-b4caaec482f5\" (UID: \"0037c462-11bc-4f13-9d5a-b4caaec482f5\") "
Nov 03 20:31:16.856185 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:16.826570    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0037c462-11bc-4f13-9d5a-b4caaec482f5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0037c462-11bc-4f13-9d5a-b4caaec482f5-workload-socket\") pod \"0037c462-11bc-4f13-9d5a-b4caaec482f5\" (UID: \"0037c462-11bc-4f13-9d5a-b4caaec482f5\") "
Nov 03 20:31:16.856185 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:16.826616    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0037c462-11bc-4f13-9d5a-b4caaec482f5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0037c462-11bc-4f13-9d5a-b4caaec482f5-istio-envoy\") pod \"0037c462-11bc-4f13-9d5a-b4caaec482f5\" (UID: \"0037c462-11bc-4f13-9d5a-b4caaec482f5\") "
Nov 03 20:31:16.856185 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:16.826688    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0037c462-11bc-4f13-9d5a-b4caaec482f5-kube-api-access-q5kwp\" (UniqueName: \"kubernetes.io/projected/0037c462-11bc-4f13-9d5a-b4caaec482f5-kube-api-access-q5kwp\") pod \"0037c462-11bc-4f13-9d5a-b4caaec482f5\" (UID: \"0037c462-11bc-4f13-9d5a-b4caaec482f5\") "
Nov 03 20:31:16.856185 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:16.826827    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0037c462-11bc-4f13-9d5a-b4caaec482f5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0037c462-11bc-4f13-9d5a-b4caaec482f5-istio-podinfo\") pod \"0037c462-11bc-4f13-9d5a-b4caaec482f5\" (UID: \"0037c462-11bc-4f13-9d5a-b4caaec482f5\") "
Nov 03 20:31:16.856185 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:16.826859    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0037c462-11bc-4f13-9d5a-b4caaec482f5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0037c462-11bc-4f13-9d5a-b4caaec482f5-workload-certs\") pod \"0037c462-11bc-4f13-9d5a-b4caaec482f5\" (UID: \"0037c462-11bc-4f13-9d5a-b4caaec482f5\") "
Nov 03 20:31:16.856185 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:16.826938    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0037c462-11bc-4f13-9d5a-b4caaec482f5-istio-token\" (UniqueName: \"kubernetes.io/projected/0037c462-11bc-4f13-9d5a-b4caaec482f5-istio-token\") pod \"0037c462-11bc-4f13-9d5a-b4caaec482f5\" (UID: \"0037c462-11bc-4f13-9d5a-b4caaec482f5\") "
Nov 03 20:31:16.856185 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:16.827018    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0037c462-11bc-4f13-9d5a-b4caaec482f5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0037c462-11bc-4f13-9d5a-b4caaec482f5-credential-socket\") pod \"0037c462-11bc-4f13-9d5a-b4caaec482f5\" (UID: \"0037c462-11bc-4f13-9d5a-b4caaec482f5\") "
Nov 03 20:31:16.856185 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:16.828604    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0037c462-11bc-4f13-9d5a-b4caaec482f5-istiod-ca-cert" pod "0037c462-11bc-4f13-9d5a-b4caaec482f5" (UID: "0037c462-11bc-4f13-9d5a-b4caaec482f5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:31:16.856185 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:16.828954    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0037c462-11bc-4f13-9d5a-b4caaec482f5-istio-data" pod "0037c462-11bc-4f13-9d5a-b4caaec482f5" (UID: "0037c462-11bc-4f13-9d5a-b4caaec482f5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:31:16.856185 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:16.829300    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0037c462-11bc-4f13-9d5a-b4caaec482f5-workload-socket" pod "0037c462-11bc-4f13-9d5a-b4caaec482f5" (UID: "0037c462-11bc-4f13-9d5a-b4caaec482f5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:31:16.856185 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:16.851300    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0037c462-11bc-4f13-9d5a-b4caaec482f5-workload-certs" pod "0037c462-11bc-4f13-9d5a-b4caaec482f5" (UID: "0037c462-11bc-4f13-9d5a-b4caaec482f5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:31:16.881031 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:16.880707    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0037c462-11bc-4f13-9d5a-b4caaec482f5-credential-socket" pod "0037c462-11bc-4f13-9d5a-b4caaec482f5" (UID: "0037c462-11bc-4f13-9d5a-b4caaec482f5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:31:16.887027 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:16.886989    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0037c462-11bc-4f13-9d5a-b4caaec482f5-istio-token" pod "0037c462-11bc-4f13-9d5a-b4caaec482f5" (UID: "0037c462-11bc-4f13-9d5a-b4caaec482f5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:31:16.896752 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:16.895929    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0037c462-11bc-4f13-9d5a-b4caaec482f5-istio-envoy" pod "0037c462-11bc-4f13-9d5a-b4caaec482f5" (UID: "0037c462-11bc-4f13-9d5a-b4caaec482f5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:31:16.896752 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:16.896013    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0037c462-11bc-4f13-9d5a-b4caaec482f5-istio-podinfo" pod "0037c462-11bc-4f13-9d5a-b4caaec482f5" (UID: "0037c462-11bc-4f13-9d5a-b4caaec482f5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:31:16.896752 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:16.896120    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0037c462-11bc-4f13-9d5a-b4caaec482f5-kube-api-access-q5kwp" pod "0037c462-11bc-4f13-9d5a-b4caaec482f5" (UID: "0037c462-11bc-4f13-9d5a-b4caaec482f5"). InnerVolumeSpecName "kube-api-access-q5kwp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:31:16.918729 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:16.918702    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zdsqp"
Nov 03 20:31:16.930477 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:16.930429    5570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0037c462-11bc-4f13-9d5a-b4caaec482f5-istiod-ca-cert\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:31:16.930477 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:16.930466    5570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0037c462-11bc-4f13-9d5a-b4caaec482f5-istio-data\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:31:16.931121 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:16.930499    5570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0037c462-11bc-4f13-9d5a-b4caaec482f5-workload-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:31:16.931121 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:16.930512    5570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0037c462-11bc-4f13-9d5a-b4caaec482f5-istio-envoy\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:31:16.931121 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:16.930527    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5kwp\" (UniqueName: \"kubernetes.io/projected/0037c462-11bc-4f13-9d5a-b4caaec482f5-kube-api-access-q5kwp\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:31:16.931121 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:16.930542    5570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0037c462-11bc-4f13-9d5a-b4caaec482f5-istio-podinfo\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:31:16.931121 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:16.930557    5570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0037c462-11bc-4f13-9d5a-b4caaec482f5-workload-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:31:16.931121 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:16.930570    5570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0037c462-11bc-4f13-9d5a-b4caaec482f5-istio-token\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:31:16.931121 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:16.930584    5570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0037c462-11bc-4f13-9d5a-b4caaec482f5-credential-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:31:17.047799 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:17.047427    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f747f534-4e45-446f-948e-c66ce316db5d-istio-token\" (UniqueName: \"kubernetes.io/projected/f747f534-4e45-446f-948e-c66ce316db5d-istio-token\") pod \"f747f534-4e45-446f-948e-c66ce316db5d\" (UID: \"f747f534-4e45-446f-948e-c66ce316db5d\") "
Nov 03 20:31:17.047799 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:17.047552    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f747f534-4e45-446f-948e-c66ce316db5d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f747f534-4e45-446f-948e-c66ce316db5d-workload-socket\") pod \"f747f534-4e45-446f-948e-c66ce316db5d\" (UID: \"f747f534-4e45-446f-948e-c66ce316db5d\") "
Nov 03 20:31:17.047799 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:17.047593    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f747f534-4e45-446f-948e-c66ce316db5d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f747f534-4e45-446f-948e-c66ce316db5d-credential-socket\") pod \"f747f534-4e45-446f-948e-c66ce316db5d\" (UID: \"f747f534-4e45-446f-948e-c66ce316db5d\") "
Nov 03 20:31:17.047799 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:17.047652    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f747f534-4e45-446f-948e-c66ce316db5d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f747f534-4e45-446f-948e-c66ce316db5d-istiod-ca-cert\") pod \"f747f534-4e45-446f-948e-c66ce316db5d\" (UID: \"f747f534-4e45-446f-948e-c66ce316db5d\") "
Nov 03 20:31:17.047799 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:17.047703    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f747f534-4e45-446f-948e-c66ce316db5d-kube-api-access-w6768\" (UniqueName: \"kubernetes.io/projected/f747f534-4e45-446f-948e-c66ce316db5d-kube-api-access-w6768\") pod \"f747f534-4e45-446f-948e-c66ce316db5d\" (UID: \"f747f534-4e45-446f-948e-c66ce316db5d\") "
Nov 03 20:31:17.047799 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:17.047793    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f747f534-4e45-446f-948e-c66ce316db5d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f747f534-4e45-446f-948e-c66ce316db5d-istio-podinfo\") pod \"f747f534-4e45-446f-948e-c66ce316db5d\" (UID: \"f747f534-4e45-446f-948e-c66ce316db5d\") "
Nov 03 20:31:17.048300 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:17.047827    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f747f534-4e45-446f-948e-c66ce316db5d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f747f534-4e45-446f-948e-c66ce316db5d-workload-certs\") pod \"f747f534-4e45-446f-948e-c66ce316db5d\" (UID: \"f747f534-4e45-446f-948e-c66ce316db5d\") "
Nov 03 20:31:17.048300 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:17.047882    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f747f534-4e45-446f-948e-c66ce316db5d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f747f534-4e45-446f-948e-c66ce316db5d-istio-data\") pod \"f747f534-4e45-446f-948e-c66ce316db5d\" (UID: \"f747f534-4e45-446f-948e-c66ce316db5d\") "
Nov 03 20:31:17.048300 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:17.048009    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f747f534-4e45-446f-948e-c66ce316db5d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f747f534-4e45-446f-948e-c66ce316db5d-istio-envoy\") pod \"f747f534-4e45-446f-948e-c66ce316db5d\" (UID: \"f747f534-4e45-446f-948e-c66ce316db5d\") "
Nov 03 20:31:17.065102 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:17.055305    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f747f534-4e45-446f-948e-c66ce316db5d-istiod-ca-cert" pod "f747f534-4e45-446f-948e-c66ce316db5d" (UID: "f747f534-4e45-446f-948e-c66ce316db5d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:31:17.065102 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:17.055664    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f747f534-4e45-446f-948e-c66ce316db5d-credential-socket" pod "f747f534-4e45-446f-948e-c66ce316db5d" (UID: "f747f534-4e45-446f-948e-c66ce316db5d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:31:17.065102 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:17.063576    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f747f534-4e45-446f-948e-c66ce316db5d-istio-envoy" pod "f747f534-4e45-446f-948e-c66ce316db5d" (UID: "f747f534-4e45-446f-948e-c66ce316db5d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:31:17.103754 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:17.083181    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f747f534-4e45-446f-948e-c66ce316db5d-istio-token" pod "f747f534-4e45-446f-948e-c66ce316db5d" (UID: "f747f534-4e45-446f-948e-c66ce316db5d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:31:17.106982 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:17.106930    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f747f534-4e45-446f-948e-c66ce316db5d-kube-api-access-w6768" pod "f747f534-4e45-446f-948e-c66ce316db5d" (UID: "f747f534-4e45-446f-948e-c66ce316db5d"). InnerVolumeSpecName "kube-api-access-w6768". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:31:17.182613 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:17.179368    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f747f534-4e45-446f-948e-c66ce316db5d-istio-data" pod "f747f534-4e45-446f-948e-c66ce316db5d" (UID: "f747f534-4e45-446f-948e-c66ce316db5d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:31:17.238913 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:17.182917    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f747f534-4e45-446f-948e-c66ce316db5d-workload-socket" pod "f747f534-4e45-446f-948e-c66ce316db5d" (UID: "f747f534-4e45-446f-948e-c66ce316db5d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:31:17.277797 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:17.266811    5570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f747f534-4e45-446f-948e-c66ce316db5d-workload-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:31:17.277797 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:17.266849    5570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f747f534-4e45-446f-948e-c66ce316db5d-credential-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:31:17.277797 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:17.266867    5570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f747f534-4e45-446f-948e-c66ce316db5d-istiod-ca-cert\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:31:17.277797 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:17.266881    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6768\" (UniqueName: \"kubernetes.io/projected/f747f534-4e45-446f-948e-c66ce316db5d-kube-api-access-w6768\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:31:17.277797 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:17.266895    5570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f747f534-4e45-446f-948e-c66ce316db5d-istio-data\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:31:17.277797 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:17.266907    5570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f747f534-4e45-446f-948e-c66ce316db5d-istio-envoy\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:31:17.277797 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:17.266919    5570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f747f534-4e45-446f-948e-c66ce316db5d-istio-token\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:31:17.309838 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:17.278290    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f747f534-4e45-446f-948e-c66ce316db5d-workload-certs" pod "f747f534-4e45-446f-948e-c66ce316db5d" (UID: "f747f534-4e45-446f-948e-c66ce316db5d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:31:17.309838 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:17.292488    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-7e4389-openshift-default-5b5bd7f55-kpk7n"
Nov 03 20:31:17.309838 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:17.299657    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-7e4389-openshift-default-5b5bd7f55-kpk7n"
Nov 03 20:31:17.309838 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:17.306360    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f747f534-4e45-446f-948e-c66ce316db5d-istio-podinfo" pod "f747f534-4e45-446f-948e-c66ce316db5d" (UID: "f747f534-4e45-446f-948e-c66ce316db5d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:31:17.318306 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:17.318277    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-d1aa66-openshift-default-8457c96657-lt2kw"
Nov 03 20:31:17.321087 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:17.321042    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-d1aa66-openshift-default-8457c96657-lt2kw"
Nov 03 20:31:17.347137 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:17.347097    5570 generic.go:356] "Generic (PLEG): container finished" podID="f747f534-4e45-446f-948e-c66ce316db5d" containerID="1b29b47590450ba9dbd2e2bf29919ee55c3af4cce616436f3a421e68c2808b25" exitCode=0
Nov 03 20:31:17.347583 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:17.347150    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zdsqp"
Nov 03 20:31:17.347583 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:17.347348    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zdsqp" event={"ID":"f747f534-4e45-446f-948e-c66ce316db5d","Type":"ContainerDied","Data":"1b29b47590450ba9dbd2e2bf29919ee55c3af4cce616436f3a421e68c2808b25"}
Nov 03 20:31:17.347583 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:17.347398    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zdsqp" event={"ID":"f747f534-4e45-446f-948e-c66ce316db5d","Type":"ContainerDied","Data":"039c1406177aa8b42956f35632c28b10a3b7b6ce94d0b16815450f8a41ea113d"}
Nov 03 20:31:17.347583 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:17.347427    5570 scope.go:122] "RemoveContainer" containerID="1b29b47590450ba9dbd2e2bf29919ee55c3af4cce616436f3a421e68c2808b25"
Nov 03 20:31:17.350687 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:17.350654    5570 generic.go:356] "Generic (PLEG): container finished" podID="0037c462-11bc-4f13-9d5a-b4caaec482f5" containerID="dfbabe837991c35498bcfdd8e6a2b1187ffc2d56f090498d2294563f36c5f235" exitCode=0
Nov 03 20:31:17.351008 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:17.350814    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-gd5bg" event={"ID":"0037c462-11bc-4f13-9d5a-b4caaec482f5","Type":"ContainerDied","Data":"dfbabe837991c35498bcfdd8e6a2b1187ffc2d56f090498d2294563f36c5f235"}
Nov 03 20:31:17.351255 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:17.351221    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-gd5bg" event={"ID":"0037c462-11bc-4f13-9d5a-b4caaec482f5","Type":"ContainerDied","Data":"e0bec6f8ef33f54f4ddf20b0ae68eb130ebf01b75b170c88730ed027f75c1349"}
Nov 03 20:31:17.351391 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:17.350902    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-gd5bg"
Nov 03 20:31:17.362077 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:17.361908    5570 generic.go:356] "Generic (PLEG): container finished" podID="7b57a22a-cf28-4099-8883-c04c61807dfa" containerID="77b59920db5fd3798fd079759ead749d52749bd7e92d88348b87940208da84d0" exitCode=0
Nov 03 20:31:17.364532 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:17.362050    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-nsjcd" event={"ID":"7b57a22a-cf28-4099-8883-c04c61807dfa","Type":"ContainerDied","Data":"77b59920db5fd3798fd079759ead749d52749bd7e92d88348b87940208da84d0"}
Nov 03 20:31:17.369052 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:17.368658    5570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f747f534-4e45-446f-948e-c66ce316db5d-istio-podinfo\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:31:17.369052 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:17.368676    5570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f747f534-4e45-446f-948e-c66ce316db5d-workload-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:31:17.379660 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:17.379633    5570 scope.go:122] "RemoveContainer" containerID="1b29b47590450ba9dbd2e2bf29919ee55c3af4cce616436f3a421e68c2808b25"
Nov 03 20:31:17.380003 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:31:17.379983    5570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b29b47590450ba9dbd2e2bf29919ee55c3af4cce616436f3a421e68c2808b25\": container with ID starting with 1b29b47590450ba9dbd2e2bf29919ee55c3af4cce616436f3a421e68c2808b25 not found: ID does not exist" containerID="1b29b47590450ba9dbd2e2bf29919ee55c3af4cce616436f3a421e68c2808b25"
Nov 03 20:31:17.380079 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:17.380016    5570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b29b47590450ba9dbd2e2bf29919ee55c3af4cce616436f3a421e68c2808b25"} err="failed to get container status \"1b29b47590450ba9dbd2e2bf29919ee55c3af4cce616436f3a421e68c2808b25\": rpc error: code = NotFound desc = could not find container \"1b29b47590450ba9dbd2e2bf29919ee55c3af4cce616436f3a421e68c2808b25\": container with ID starting with 1b29b47590450ba9dbd2e2bf29919ee55c3af4cce616436f3a421e68c2808b25 not found: ID does not exist"
Nov 03 20:31:17.380079 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:17.380035    5570 scope.go:122] "RemoveContainer" containerID="dfbabe837991c35498bcfdd8e6a2b1187ffc2d56f090498d2294563f36c5f235"
Nov 03 20:31:17.399933 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:17.399917    5570 scope.go:122] "RemoveContainer" containerID="dfbabe837991c35498bcfdd8e6a2b1187ffc2d56f090498d2294563f36c5f235"
Nov 03 20:31:17.400271 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:31:17.400247    5570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dfbabe837991c35498bcfdd8e6a2b1187ffc2d56f090498d2294563f36c5f235\": container with ID starting with dfbabe837991c35498bcfdd8e6a2b1187ffc2d56f090498d2294563f36c5f235 not found: ID does not exist" containerID="dfbabe837991c35498bcfdd8e6a2b1187ffc2d56f090498d2294563f36c5f235"
Nov 03 20:31:17.400401 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:17.400279    5570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfbabe837991c35498bcfdd8e6a2b1187ffc2d56f090498d2294563f36c5f235"} err="failed to get container status \"dfbabe837991c35498bcfdd8e6a2b1187ffc2d56f090498d2294563f36c5f235\": rpc error: code = NotFound desc = could not find container \"dfbabe837991c35498bcfdd8e6a2b1187ffc2d56f090498d2294563f36c5f235\": container with ID starting with dfbabe837991c35498bcfdd8e6a2b1187ffc2d56f090498d2294563f36c5f235 not found: ID does not exist"
Nov 03 20:31:17.400401 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:17.400298    5570 scope.go:122] "RemoveContainer" containerID="e0359761c7f58587a7e9e0c308a42f2df94efc0661527121e1458f5db9608599"
Nov 03 20:31:17.403362 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:17.403343    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-nsjcd"
Nov 03 20:31:17.469728 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:17.469696    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b57a22a-cf28-4099-8883-c04c61807dfa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b57a22a-cf28-4099-8883-c04c61807dfa-credential-socket\") pod \"7b57a22a-cf28-4099-8883-c04c61807dfa\" (UID: \"7b57a22a-cf28-4099-8883-c04c61807dfa\") "
Nov 03 20:31:17.469971 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:17.469884    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b57a22a-cf28-4099-8883-c04c61807dfa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b57a22a-cf28-4099-8883-c04c61807dfa-istiod-ca-cert\") pod \"7b57a22a-cf28-4099-8883-c04c61807dfa\" (UID: \"7b57a22a-cf28-4099-8883-c04c61807dfa\") "
Nov 03 20:31:17.470087 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:17.470034    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b57a22a-cf28-4099-8883-c04c61807dfa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b57a22a-cf28-4099-8883-c04c61807dfa-workload-certs\") pod \"7b57a22a-cf28-4099-8883-c04c61807dfa\" (UID: \"7b57a22a-cf28-4099-8883-c04c61807dfa\") "
Nov 03 20:31:17.470087 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:17.470068    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b57a22a-cf28-4099-8883-c04c61807dfa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b57a22a-cf28-4099-8883-c04c61807dfa-istio-data\") pod \"7b57a22a-cf28-4099-8883-c04c61807dfa\" (UID: \"7b57a22a-cf28-4099-8883-c04c61807dfa\") "
Nov 03 20:31:17.470205 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:17.470075    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b57a22a-cf28-4099-8883-c04c61807dfa-credential-socket" pod "7b57a22a-cf28-4099-8883-c04c61807dfa" (UID: "7b57a22a-cf28-4099-8883-c04c61807dfa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:31:17.470205 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:17.470131    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b57a22a-cf28-4099-8883-c04c61807dfa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b57a22a-cf28-4099-8883-c04c61807dfa-istio-envoy\") pod \"7b57a22a-cf28-4099-8883-c04c61807dfa\" (UID: \"7b57a22a-cf28-4099-8883-c04c61807dfa\") "
Nov 03 20:31:17.470360 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:17.470313    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b57a22a-cf28-4099-8883-c04c61807dfa-workload-certs" pod "7b57a22a-cf28-4099-8883-c04c61807dfa" (UID: "7b57a22a-cf28-4099-8883-c04c61807dfa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:31:17.470419 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:17.470380    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b57a22a-cf28-4099-8883-c04c61807dfa-istio-data" pod "7b57a22a-cf28-4099-8883-c04c61807dfa" (UID: "7b57a22a-cf28-4099-8883-c04c61807dfa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:31:17.470419 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:17.470403    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b57a22a-cf28-4099-8883-c04c61807dfa-istiod-ca-cert" pod "7b57a22a-cf28-4099-8883-c04c61807dfa" (UID: "7b57a22a-cf28-4099-8883-c04c61807dfa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:31:17.470526 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:17.470434    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b57a22a-cf28-4099-8883-c04c61807dfa-kube-api-access-f5vqm\" (UniqueName: \"kubernetes.io/projected/7b57a22a-cf28-4099-8883-c04c61807dfa-kube-api-access-f5vqm\") pod \"7b57a22a-cf28-4099-8883-c04c61807dfa\" (UID: \"7b57a22a-cf28-4099-8883-c04c61807dfa\") "
Nov 03 20:31:17.470643 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:17.470626    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b57a22a-cf28-4099-8883-c04c61807dfa-istio-token\" (UniqueName: \"kubernetes.io/projected/7b57a22a-cf28-4099-8883-c04c61807dfa-istio-token\") pod \"7b57a22a-cf28-4099-8883-c04c61807dfa\" (UID: \"7b57a22a-cf28-4099-8883-c04c61807dfa\") "
Nov 03 20:31:17.470760 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:17.470740    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b57a22a-cf28-4099-8883-c04c61807dfa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b57a22a-cf28-4099-8883-c04c61807dfa-workload-socket\") pod \"7b57a22a-cf28-4099-8883-c04c61807dfa\" (UID: \"7b57a22a-cf28-4099-8883-c04c61807dfa\") "
Nov 03 20:31:17.470885 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:17.470812    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7b57a22a-cf28-4099-8883-c04c61807dfa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b57a22a-cf28-4099-8883-c04c61807dfa-istio-podinfo\") pod \"7b57a22a-cf28-4099-8883-c04c61807dfa\" (UID: \"7b57a22a-cf28-4099-8883-c04c61807dfa\") "
Nov 03 20:31:17.471138 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:17.471115    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b57a22a-cf28-4099-8883-c04c61807dfa-workload-socket" pod "7b57a22a-cf28-4099-8883-c04c61807dfa" (UID: "7b57a22a-cf28-4099-8883-c04c61807dfa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:31:17.472220 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:17.471997    5570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b57a22a-cf28-4099-8883-c04c61807dfa-workload-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:31:17.472220 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:17.472023    5570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b57a22a-cf28-4099-8883-c04c61807dfa-credential-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:31:17.472220 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:17.472040    5570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b57a22a-cf28-4099-8883-c04c61807dfa-istiod-ca-cert\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:31:17.472220 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:17.472054    5570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b57a22a-cf28-4099-8883-c04c61807dfa-workload-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:31:17.472220 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:17.472064    5570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b57a22a-cf28-4099-8883-c04c61807dfa-istio-data\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:31:17.473377 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:17.473343    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b57a22a-cf28-4099-8883-c04c61807dfa-istio-token" pod "7b57a22a-cf28-4099-8883-c04c61807dfa" (UID: "7b57a22a-cf28-4099-8883-c04c61807dfa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:31:17.475223 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:17.475196    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b57a22a-cf28-4099-8883-c04c61807dfa-kube-api-access-f5vqm" pod "7b57a22a-cf28-4099-8883-c04c61807dfa" (UID: "7b57a22a-cf28-4099-8883-c04c61807dfa"). InnerVolumeSpecName "kube-api-access-f5vqm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:31:17.476935 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:17.476912    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b57a22a-cf28-4099-8883-c04c61807dfa-istio-envoy" pod "7b57a22a-cf28-4099-8883-c04c61807dfa" (UID: "7b57a22a-cf28-4099-8883-c04c61807dfa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:31:17.477076 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:17.477009    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7b57a22a-cf28-4099-8883-c04c61807dfa-istio-podinfo" pod "7b57a22a-cf28-4099-8883-c04c61807dfa" (UID: "7b57a22a-cf28-4099-8883-c04c61807dfa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:31:17.534489 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:17.534449    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zdsqp"]
Nov 03 20:31:17.541609 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:17.541567    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-zdsqp"]
Nov 03 20:31:17.574030 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:17.573991    5570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b57a22a-cf28-4099-8883-c04c61807dfa-istio-envoy\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:31:17.574030 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:17.574024    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5vqm\" (UniqueName: \"kubernetes.io/projected/7b57a22a-cf28-4099-8883-c04c61807dfa-kube-api-access-f5vqm\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:31:17.574251 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:17.574041    5570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b57a22a-cf28-4099-8883-c04c61807dfa-istio-token\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:31:17.574251 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:17.574051    5570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b57a22a-cf28-4099-8883-c04c61807dfa-istio-podinfo\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:31:17.637038 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:17.637001    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-gd5bg"]
Nov 03 20:31:17.642154 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:17.642126    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-gd5bg"]
Nov 03 20:31:17.978369 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:17.978330    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0037c462-11bc-4f13-9d5a-b4caaec482f5" path="/var/lib/kubelet/pods/0037c462-11bc-4f13-9d5a-b4caaec482f5/volumes"
Nov 03 20:31:17.979107 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:17.979089    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f747f534-4e45-446f-948e-c66ce316db5d" path="/var/lib/kubelet/pods/f747f534-4e45-446f-948e-c66ce316db5d/volumes"
Nov 03 20:31:18.374136 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:18.374038    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-nsjcd"
Nov 03 20:31:18.374136 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:18.374036    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-nsjcd" event={"ID":"7b57a22a-cf28-4099-8883-c04c61807dfa","Type":"ContainerDied","Data":"2295b4345521a6856cdd923c5ce86f588a760b8eea207412781577b5435570a3"}
Nov 03 20:31:18.374670 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:18.374190    5570 scope.go:122] "RemoveContainer" containerID="77b59920db5fd3798fd079759ead749d52749bd7e92d88348b87940208da84d0"
Nov 03 20:31:18.496889 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:18.496851    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-nsjcd"]
Nov 03 20:31:18.502908 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:18.502846    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-eb87f429-openshift-default-64649dfdb5-nsjcd"]
Nov 03 20:31:19.967422 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:19.967390    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b57a22a-cf28-4099-8883-c04c61807dfa" path="/var/lib/kubelet/pods/7b57a22a-cf28-4099-8883-c04c61807dfa/volumes"
Nov 03 20:31:23.383650 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:23.383619    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d05604-5c69f8bc7-8cb6g"
Nov 03 20:31:24.315308 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:24.315251    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-60efdf-7bfdc67bd7-j6ctn"
Nov 03 20:31:25.324371 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:25.324336    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-60efdf-7cb6d9fbc8-fwx62"
Nov 03 20:31:25.461012 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:25.460977    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c4a71d-5c8bb448f4-jsmhw"]
Nov 03 20:31:25.550207 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:25.550152    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c4a71d-665484ddc9-csmw4"]
Nov 03 20:31:27.229972 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:31:27.229937    5570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8bwhz" containerName="extractor"
Nov 03 20:31:30.461592 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:30.461541    5570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-c4a71d-5c8bb448f4-jsmhw" podUID="be6cbff2-19be-4730-b7b4-ab513d8c6859" containerName="payload-processing" containerID="cri-o://b59566cf486a15588b4da19a9b55f09732218dc808043b8b397bb652190b1c29" gracePeriod=25
Nov 03 20:31:30.550552 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:30.550506    5570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c4a71d-665484ddc9-csmw4" podUID="7b9f0474-e24d-4b53-bf9b-d7292207971b" containerName="payload-pre-processing" containerID="cri-o://f9818b0562a1e45412512a0dd2060af8663585c5c08b69a8840938ed5dc0e65d" gracePeriod=25
Nov 03 20:31:30.913961 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:30.913923    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-c4a71d-5c8bb448f4-jsmhw"
Nov 03 20:31:30.916975 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:30.916954    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c4a71d-665484ddc9-csmw4"
Nov 03 20:31:31.048437 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:31.048335    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b9f0474-e24d-4b53-bf9b-d7292207971b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b9f0474-e24d-4b53-bf9b-d7292207971b-plugins-config-volume\") pod \"7b9f0474-e24d-4b53-bf9b-d7292207971b\" (UID: \"7b9f0474-e24d-4b53-bf9b-d7292207971b\") "
Nov 03 20:31:31.048638 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:31.048535    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be6cbff2-19be-4730-b7b4-ab513d8c6859-kube-api-access-vw4mt\" (UniqueName: \"kubernetes.io/projected/be6cbff2-19be-4730-b7b4-ab513d8c6859-kube-api-access-vw4mt\") pod \"be6cbff2-19be-4730-b7b4-ab513d8c6859\" (UID: \"be6cbff2-19be-4730-b7b4-ab513d8c6859\") "
Nov 03 20:31:31.048705 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:31.048659    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b9f0474-e24d-4b53-bf9b-d7292207971b-kube-api-access-6d4sk\" (UniqueName: \"kubernetes.io/projected/7b9f0474-e24d-4b53-bf9b-d7292207971b-kube-api-access-6d4sk\") pod \"7b9f0474-e24d-4b53-bf9b-d7292207971b\" (UID: \"7b9f0474-e24d-4b53-bf9b-d7292207971b\") "
Nov 03 20:31:31.048767 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:31.048723    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be6cbff2-19be-4730-b7b4-ab513d8c6859-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be6cbff2-19be-4730-b7b4-ab513d8c6859-plugins-config-volume\") pod \"be6cbff2-19be-4730-b7b4-ab513d8c6859\" (UID: \"be6cbff2-19be-4730-b7b4-ab513d8c6859\") "
Nov 03 20:31:31.048767 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:31.048742    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b9f0474-e24d-4b53-bf9b-d7292207971b-plugins-config-volume" pod "7b9f0474-e24d-4b53-bf9b-d7292207971b" (UID: "7b9f0474-e24d-4b53-bf9b-d7292207971b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:31:31.049184 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:31.049154    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be6cbff2-19be-4730-b7b4-ab513d8c6859-plugins-config-volume" pod "be6cbff2-19be-4730-b7b4-ab513d8c6859" (UID: "be6cbff2-19be-4730-b7b4-ab513d8c6859"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:31:31.049941 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:31.049800    5570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be6cbff2-19be-4730-b7b4-ab513d8c6859-plugins-config-volume\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:31:31.049941 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:31.049830    5570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b9f0474-e24d-4b53-bf9b-d7292207971b-plugins-config-volume\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:31:31.051037 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:31.051015    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be6cbff2-19be-4730-b7b4-ab513d8c6859-kube-api-access-vw4mt" pod "be6cbff2-19be-4730-b7b4-ab513d8c6859" (UID: "be6cbff2-19be-4730-b7b4-ab513d8c6859"). InnerVolumeSpecName "kube-api-access-vw4mt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:31:31.051367 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:31.051339    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b9f0474-e24d-4b53-bf9b-d7292207971b-kube-api-access-6d4sk" pod "7b9f0474-e24d-4b53-bf9b-d7292207971b" (UID: "7b9f0474-e24d-4b53-bf9b-d7292207971b"). InnerVolumeSpecName "kube-api-access-6d4sk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:31:31.152009 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:31.151973    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6d4sk\" (UniqueName: \"kubernetes.io/projected/7b9f0474-e24d-4b53-bf9b-d7292207971b-kube-api-access-6d4sk\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:31:31.152009 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:31.152006    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vw4mt\" (UniqueName: \"kubernetes.io/projected/be6cbff2-19be-4730-b7b4-ab513d8c6859-kube-api-access-vw4mt\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:31:31.542106 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:31.530316    5570 generic.go:356] "Generic (PLEG): container finished" podID="be6cbff2-19be-4730-b7b4-ab513d8c6859" containerID="b59566cf486a15588b4da19a9b55f09732218dc808043b8b397bb652190b1c29" exitCode=0
Nov 03 20:31:31.542106 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:31.530476    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-c4a71d-5c8bb448f4-jsmhw" event={"ID":"be6cbff2-19be-4730-b7b4-ab513d8c6859","Type":"ContainerDied","Data":"b59566cf486a15588b4da19a9b55f09732218dc808043b8b397bb652190b1c29"}
Nov 03 20:31:31.542106 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:31.530529    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-c4a71d-5c8bb448f4-jsmhw" event={"ID":"be6cbff2-19be-4730-b7b4-ab513d8c6859","Type":"ContainerDied","Data":"5b1bee1ad41a04fef77a6e7bf5bc9941c608ddb0c687d1109ab40297e192fec5"}
Nov 03 20:31:31.542106 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:31.530559    5570 scope.go:122] "RemoveContainer" containerID="b59566cf486a15588b4da19a9b55f09732218dc808043b8b397bb652190b1c29"
Nov 03 20:31:31.542106 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:31.530721    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-c4a71d-5c8bb448f4-jsmhw"
Nov 03 20:31:31.549311 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:31.549278    5570 generic.go:356] "Generic (PLEG): container finished" podID="7b9f0474-e24d-4b53-bf9b-d7292207971b" containerID="f9818b0562a1e45412512a0dd2060af8663585c5c08b69a8840938ed5dc0e65d" exitCode=0
Nov 03 20:31:31.550449 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:31.549564    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c4a71d-665484ddc9-csmw4"
Nov 03 20:31:31.551896 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:31.549587    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c4a71d-665484ddc9-csmw4" event={"ID":"7b9f0474-e24d-4b53-bf9b-d7292207971b","Type":"ContainerDied","Data":"f9818b0562a1e45412512a0dd2060af8663585c5c08b69a8840938ed5dc0e65d"}
Nov 03 20:31:31.551896 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:31.551753    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c4a71d-665484ddc9-csmw4" event={"ID":"7b9f0474-e24d-4b53-bf9b-d7292207971b","Type":"ContainerDied","Data":"1baf64282060eecf2b9993e066ea62ae80d86908f81e455c0b425689b2f0856f"}
Nov 03 20:31:31.574434 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:31.574409    5570 scope.go:122] "RemoveContainer" containerID="b59566cf486a15588b4da19a9b55f09732218dc808043b8b397bb652190b1c29"
Nov 03 20:31:31.574886 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:31:31.574847    5570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b59566cf486a15588b4da19a9b55f09732218dc808043b8b397bb652190b1c29\": container with ID starting with b59566cf486a15588b4da19a9b55f09732218dc808043b8b397bb652190b1c29 not found: ID does not exist" containerID="b59566cf486a15588b4da19a9b55f09732218dc808043b8b397bb652190b1c29"
Nov 03 20:31:31.575017 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:31.574896    5570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b59566cf486a15588b4da19a9b55f09732218dc808043b8b397bb652190b1c29"} err="failed to get container status \"b59566cf486a15588b4da19a9b55f09732218dc808043b8b397bb652190b1c29\": rpc error: code = NotFound desc = could not find container \"b59566cf486a15588b4da19a9b55f09732218dc808043b8b397bb652190b1c29\": container with ID starting with b59566cf486a15588b4da19a9b55f09732218dc808043b8b397bb652190b1c29 not found: ID does not exist"
Nov 03 20:31:31.575017 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:31.574924    5570 scope.go:122] "RemoveContainer" containerID="f9818b0562a1e45412512a0dd2060af8663585c5c08b69a8840938ed5dc0e65d"
Nov 03 20:31:31.586349 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:31.586323    5570 scope.go:122] "RemoveContainer" containerID="f9818b0562a1e45412512a0dd2060af8663585c5c08b69a8840938ed5dc0e65d"
Nov 03 20:31:31.586659 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:31:31.586637    5570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9818b0562a1e45412512a0dd2060af8663585c5c08b69a8840938ed5dc0e65d\": container with ID starting with f9818b0562a1e45412512a0dd2060af8663585c5c08b69a8840938ed5dc0e65d not found: ID does not exist" containerID="f9818b0562a1e45412512a0dd2060af8663585c5c08b69a8840938ed5dc0e65d"
Nov 03 20:31:31.586744 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:31.586671    5570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9818b0562a1e45412512a0dd2060af8663585c5c08b69a8840938ed5dc0e65d"} err="failed to get container status \"f9818b0562a1e45412512a0dd2060af8663585c5c08b69a8840938ed5dc0e65d\": rpc error: code = NotFound desc = could not find container \"f9818b0562a1e45412512a0dd2060af8663585c5c08b69a8840938ed5dc0e65d\": container with ID starting with f9818b0562a1e45412512a0dd2060af8663585c5c08b69a8840938ed5dc0e65d not found: ID does not exist"
Nov 03 20:31:31.618544 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:31.618511    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c4a71d-5c8bb448f4-jsmhw"]
Nov 03 20:31:31.631575 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:31.631543    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c4a71d-5c8bb448f4-jsmhw"]
Nov 03 20:31:31.655440 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:31.655391    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c4a71d-665484ddc9-csmw4"]
Nov 03 20:31:31.659881 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:31.659846    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c4a71d-665484ddc9-csmw4"]
Nov 03 20:31:31.975505 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:31.975426    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b9f0474-e24d-4b53-bf9b-d7292207971b" path="/var/lib/kubelet/pods/7b9f0474-e24d-4b53-bf9b-d7292207971b/volumes"
Nov 03 20:31:31.976322 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:31.976292    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be6cbff2-19be-4730-b7b4-ab513d8c6859" path="/var/lib/kubelet/pods/be6cbff2-19be-4730-b7b4-ab513d8c6859/volumes"
Nov 03 20:31:32.380118 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:32.380024    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g69f8"]
Nov 03 20:31:36.552799 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:36.550280    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/distinct-2-w0-e40816-kserve-79fb46654d-s242x" podUID="4ad9b048-925c-4d01-a839-a5f25d4dd02d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:31:36.596812 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:36.594914    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/premium-w4-e80203-kserve-5f9f445456-bms9d" podUID="6ad76efc-20c2-4951-b22a-c5a39cd39ed4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:31:36.690936 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:36.690877    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-482722/distinct-2-w3-482722-kserve-74bf58cf7d-v9cjw" podUID="9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:31:36.898132 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:36.897954    5570 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-2x92x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:31:36.899989 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:36.899939    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-2x92x" podUID="2ac2fc9c-0a3e-4326-a4f8-8856b1a30e36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:31:39.938833 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:39.938789    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-60efdf-57848f5c8c-qjzhb"]
Nov 03 20:31:39.939734 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:39.939709    5570 cpu_manager.go:414] "containerMap: removing container" podUID="7b9f0474-e24d-4b53-bf9b-d7292207971b" containerName="payload-pre-processing"
Nov 03 20:31:39.939734 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:39.939731    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b9f0474-e24d-4b53-bf9b-d7292207971b" containerName="payload-pre-processing"
Nov 03 20:31:39.939925 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:39.939748    5570 cpu_manager.go:414] "containerMap: removing container" podUID="0037c462-11bc-4f13-9d5a-b4caaec482f5" containerName="istio-proxy"
Nov 03 20:31:39.939925 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:39.939753    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0037c462-11bc-4f13-9d5a-b4caaec482f5" containerName="istio-proxy"
Nov 03 20:31:39.939925 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:39.939792    5570 cpu_manager.go:414] "containerMap: removing container" podUID="f747f534-4e45-446f-948e-c66ce316db5d" containerName="istio-proxy"
Nov 03 20:31:39.939925 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:39.939799    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f747f534-4e45-446f-948e-c66ce316db5d" containerName="istio-proxy"
Nov 03 20:31:39.939925 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:39.939810    5570 cpu_manager.go:414] "containerMap: removing container" podUID="7b57a22a-cf28-4099-8883-c04c61807dfa" containerName="istio-proxy"
Nov 03 20:31:39.939925 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:39.939817    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b57a22a-cf28-4099-8883-c04c61807dfa" containerName="istio-proxy"
Nov 03 20:31:39.939925 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:39.939842    5570 cpu_manager.go:414] "containerMap: removing container" podUID="be6cbff2-19be-4730-b7b4-ab513d8c6859" containerName="payload-processing"
Nov 03 20:31:39.939925 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:39.939848    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be6cbff2-19be-4730-b7b4-ab513d8c6859" containerName="payload-processing"
Nov 03 20:31:39.940230 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:39.939988    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b57a22a-cf28-4099-8883-c04c61807dfa" containerName="istio-proxy"
Nov 03 20:31:39.940230 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:39.940001    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b9f0474-e24d-4b53-bf9b-d7292207971b" containerName="payload-pre-processing"
Nov 03 20:31:39.940230 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:39.940008    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="0037c462-11bc-4f13-9d5a-b4caaec482f5" containerName="istio-proxy"
Nov 03 20:31:39.940230 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:39.940018    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="f747f534-4e45-446f-948e-c66ce316db5d" containerName="istio-proxy"
Nov 03 20:31:39.940230 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:39.940024    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b57a22a-cf28-4099-8883-c04c61807dfa" containerName="istio-proxy"
Nov 03 20:31:39.940230 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:39.940033    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="be6cbff2-19be-4730-b7b4-ab513d8c6859" containerName="payload-processing"
Nov 03 20:31:39.943821 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:39.943764    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-60efdf-57848f5c8c-qjzhb"
Nov 03 20:31:39.971694 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:39.971513    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-60efdf-57848f5c8c-qjzhb"]
Nov 03 20:31:39.972319 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:39.972281    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-60efdf-serving-cert\""
Nov 03 20:31:40.057314 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:40.057247    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca0be3e9-2414-4d9c-ae41-fcbe4dddb52a-metrics-certs\") pod \"maas-api-e2e-inf-a-60efdf-57848f5c8c-qjzhb\" (UID: \"ca0be3e9-2414-4d9c-ae41-fcbe4dddb52a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-60efdf-57848f5c8c-qjzhb"
Nov 03 20:31:40.057756 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:40.057729    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca0be3e9-2414-4d9c-ae41-fcbe4dddb52a-openshift-service-ca\") pod \"maas-api-e2e-inf-a-60efdf-57848f5c8c-qjzhb\" (UID: \"ca0be3e9-2414-4d9c-ae41-fcbe4dddb52a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-60efdf-57848f5c8c-qjzhb"
Nov 03 20:31:40.058311 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:40.058286    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6v6n6\" (UniqueName: \"kubernetes.io/projected/ca0be3e9-2414-4d9c-ae41-fcbe4dddb52a-kube-api-access-6v6n6\") pod \"maas-api-e2e-inf-a-60efdf-57848f5c8c-qjzhb\" (UID: \"ca0be3e9-2414-4d9c-ae41-fcbe4dddb52a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-60efdf-57848f5c8c-qjzhb"
Nov 03 20:31:40.058488 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:40.058473    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca0be3e9-2414-4d9c-ae41-fcbe4dddb52a-maas-api-tls\") pod \"maas-api-e2e-inf-a-60efdf-57848f5c8c-qjzhb\" (UID: \"ca0be3e9-2414-4d9c-ae41-fcbe4dddb52a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-60efdf-57848f5c8c-qjzhb"
Nov 03 20:31:40.160149 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:40.160107    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6v6n6\" (UniqueName: \"kubernetes.io/projected/ca0be3e9-2414-4d9c-ae41-fcbe4dddb52a-kube-api-access-6v6n6\") pod \"maas-api-e2e-inf-a-60efdf-57848f5c8c-qjzhb\" (UID: \"ca0be3e9-2414-4d9c-ae41-fcbe4dddb52a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-60efdf-57848f5c8c-qjzhb"
Nov 03 20:31:40.160947 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:40.160916    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca0be3e9-2414-4d9c-ae41-fcbe4dddb52a-maas-api-tls\") pod \"maas-api-e2e-inf-a-60efdf-57848f5c8c-qjzhb\" (UID: \"ca0be3e9-2414-4d9c-ae41-fcbe4dddb52a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-60efdf-57848f5c8c-qjzhb"
Nov 03 20:31:40.161789 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:40.161743    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca0be3e9-2414-4d9c-ae41-fcbe4dddb52a-metrics-certs\") pod \"maas-api-e2e-inf-a-60efdf-57848f5c8c-qjzhb\" (UID: \"ca0be3e9-2414-4d9c-ae41-fcbe4dddb52a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-60efdf-57848f5c8c-qjzhb"
Nov 03 20:31:40.162588 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:40.162555    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca0be3e9-2414-4d9c-ae41-fcbe4dddb52a-openshift-service-ca\") pod \"maas-api-e2e-inf-a-60efdf-57848f5c8c-qjzhb\" (UID: \"ca0be3e9-2414-4d9c-ae41-fcbe4dddb52a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-60efdf-57848f5c8c-qjzhb"
Nov 03 20:31:40.164005 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:40.163416    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca0be3e9-2414-4d9c-ae41-fcbe4dddb52a-openshift-service-ca\") pod \"maas-api-e2e-inf-a-60efdf-57848f5c8c-qjzhb\" (UID: \"ca0be3e9-2414-4d9c-ae41-fcbe4dddb52a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-60efdf-57848f5c8c-qjzhb"
Nov 03 20:31:40.177286 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:40.177247    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca0be3e9-2414-4d9c-ae41-fcbe4dddb52a-metrics-certs\") pod \"maas-api-e2e-inf-a-60efdf-57848f5c8c-qjzhb\" (UID: \"ca0be3e9-2414-4d9c-ae41-fcbe4dddb52a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-60efdf-57848f5c8c-qjzhb"
Nov 03 20:31:40.198767 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:40.198647    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6v6n6\" (UniqueName: \"kubernetes.io/projected/ca0be3e9-2414-4d9c-ae41-fcbe4dddb52a-kube-api-access-6v6n6\") pod \"maas-api-e2e-inf-a-60efdf-57848f5c8c-qjzhb\" (UID: \"ca0be3e9-2414-4d9c-ae41-fcbe4dddb52a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-60efdf-57848f5c8c-qjzhb"
Nov 03 20:31:40.204893 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:40.204850    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca0be3e9-2414-4d9c-ae41-fcbe4dddb52a-maas-api-tls\") pod \"maas-api-e2e-inf-a-60efdf-57848f5c8c-qjzhb\" (UID: \"ca0be3e9-2414-4d9c-ae41-fcbe4dddb52a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-60efdf-57848f5c8c-qjzhb"
Nov 03 20:31:40.267798 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:40.261998    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-60efdf-57848f5c8c-qjzhb"
Nov 03 20:31:40.598438 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:40.598410    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-60efdf-57848f5c8c-qjzhb"]
Nov 03 20:31:40.599718 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:31:40.599687    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca0be3e9_2414_4d9c_ae41_fcbe4dddb52a.slice/crio-fcd179dae740b6a6ec64118dedae08cd35fef554e78dc03b798b1aa37f8baedd WatchSource:0}: Error finding container fcd179dae740b6a6ec64118dedae08cd35fef554e78dc03b798b1aa37f8baedd: Status 404 returned error can't find the container with id fcd179dae740b6a6ec64118dedae08cd35fef554e78dc03b798b1aa37f8baedd
Nov 03 20:31:40.759737 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:40.759696    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-60efdf-57848f5c8c-qjzhb" event={"ID":"ca0be3e9-2414-4d9c-ae41-fcbe4dddb52a","Type":"ContainerStarted","Data":"fcd179dae740b6a6ec64118dedae08cd35fef554e78dc03b798b1aa37f8baedd"}
Nov 03 20:31:41.818470 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:41.817576    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-60efdf-57848f5c8c-qjzhb" event={"ID":"ca0be3e9-2414-4d9c-ae41-fcbe4dddb52a","Type":"ContainerStarted","Data":"1e461e7c4090b27d313b295321bafd3383e35d71c468c4df753a994ba95cdf41"}
Nov 03 20:31:41.819534 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:41.819007    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-60efdf-57848f5c8c-qjzhb"
Nov 03 20:31:41.863117 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:41.863058    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-60efdf-57848f5c8c-qjzhb" podStartSLOduration=2.309229535 podStartE2EDuration="2.8630408s" totalImagesPullingTime="553.811265ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:31:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:31:41.860121329 +0000 UTC m=+1636.012783215" watchObservedRunningTime="2026-11-03 20:31:41.8630408 +0000 UTC m=+1636.015702686"
Nov 03 20:31:46.241740 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:46.241700    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-addf1a-56979459f6-bt9cd"]
Nov 03 20:31:46.242275 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:46.241978    5570 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-addf1a-56979459f6-bt9cd" podUID="5d450803-ade0-428d-9029-d71459b2fc22" containerName="maas-api" containerID="cri-o://e99ba3bf4478c424bb398929088eeb78d64e97916e71dc67e18a610c4719b75c" gracePeriod=30
Nov 03 20:31:46.942896 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:46.937117    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-addf1a-56979459f6-bt9cd"
Nov 03 20:31:46.966361 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:46.966305    5570 generic.go:356] "Generic (PLEG): container finished" podID="5d450803-ade0-428d-9029-d71459b2fc22" containerID="e99ba3bf4478c424bb398929088eeb78d64e97916e71dc67e18a610c4719b75c" exitCode=0
Nov 03 20:31:46.966655 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:46.966632    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-addf1a-56979459f6-bt9cd" event={"ID":"5d450803-ade0-428d-9029-d71459b2fc22","Type":"ContainerDied","Data":"e99ba3bf4478c424bb398929088eeb78d64e97916e71dc67e18a610c4719b75c"}
Nov 03 20:31:46.966821 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:46.966800    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-addf1a-56979459f6-bt9cd" event={"ID":"5d450803-ade0-428d-9029-d71459b2fc22","Type":"ContainerDied","Data":"81f9208c3c9ccc487af5a750532b9a3dac39d3ae15210fcaab1225e4d5c85492"}
Nov 03 20:31:46.966981 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:46.966967    5570 scope.go:122] "RemoveContainer" containerID="e99ba3bf4478c424bb398929088eeb78d64e97916e71dc67e18a610c4719b75c"
Nov 03 20:31:47.031935 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:47.031585    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5d450803-ade0-428d-9029-d71459b2fc22-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d450803-ade0-428d-9029-d71459b2fc22-maas-api-tls\") pod \"5d450803-ade0-428d-9029-d71459b2fc22\" (UID: \"5d450803-ade0-428d-9029-d71459b2fc22\") "
Nov 03 20:31:47.031935 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:47.031653    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d450803-ade0-428d-9029-d71459b2fc22-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d450803-ade0-428d-9029-d71459b2fc22-openshift-service-ca\") pod \"5d450803-ade0-428d-9029-d71459b2fc22\" (UID: \"5d450803-ade0-428d-9029-d71459b2fc22\") "
Nov 03 20:31:47.032871 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:47.032504    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5d450803-ade0-428d-9029-d71459b2fc22-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d450803-ade0-428d-9029-d71459b2fc22-metrics-certs\") pod \"5d450803-ade0-428d-9029-d71459b2fc22\" (UID: \"5d450803-ade0-428d-9029-d71459b2fc22\") "
Nov 03 20:31:47.032871 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:47.032557    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d450803-ade0-428d-9029-d71459b2fc22-kube-api-access-qkrq4\" (UniqueName: \"kubernetes.io/projected/5d450803-ade0-428d-9029-d71459b2fc22-kube-api-access-qkrq4\") pod \"5d450803-ade0-428d-9029-d71459b2fc22\" (UID: \"5d450803-ade0-428d-9029-d71459b2fc22\") "
Nov 03 20:31:47.033904 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:47.033853    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d450803-ade0-428d-9029-d71459b2fc22-openshift-service-ca" pod "5d450803-ade0-428d-9029-d71459b2fc22" (UID: "5d450803-ade0-428d-9029-d71459b2fc22"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:31:47.043300 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:47.043226    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d450803-ade0-428d-9029-d71459b2fc22-maas-api-tls" pod "5d450803-ade0-428d-9029-d71459b2fc22" (UID: "5d450803-ade0-428d-9029-d71459b2fc22"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:31:47.045127 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:47.045099    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d450803-ade0-428d-9029-d71459b2fc22-kube-api-access-qkrq4" pod "5d450803-ade0-428d-9029-d71459b2fc22" (UID: "5d450803-ade0-428d-9029-d71459b2fc22"). InnerVolumeSpecName "kube-api-access-qkrq4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:31:47.045272 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:47.045238    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d450803-ade0-428d-9029-d71459b2fc22-metrics-certs" pod "5d450803-ade0-428d-9029-d71459b2fc22" (UID: "5d450803-ade0-428d-9029-d71459b2fc22"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:31:47.052004 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:47.049584    5570 scope.go:122] "RemoveContainer" containerID="e99ba3bf4478c424bb398929088eeb78d64e97916e71dc67e18a610c4719b75c"
Nov 03 20:31:47.052004 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:31:47.050203    5570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e99ba3bf4478c424bb398929088eeb78d64e97916e71dc67e18a610c4719b75c\": container with ID starting with e99ba3bf4478c424bb398929088eeb78d64e97916e71dc67e18a610c4719b75c not found: ID does not exist" containerID="e99ba3bf4478c424bb398929088eeb78d64e97916e71dc67e18a610c4719b75c"
Nov 03 20:31:47.052004 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:47.050239    5570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e99ba3bf4478c424bb398929088eeb78d64e97916e71dc67e18a610c4719b75c"} err="failed to get container status \"e99ba3bf4478c424bb398929088eeb78d64e97916e71dc67e18a610c4719b75c\": rpc error: code = NotFound desc = could not find container \"e99ba3bf4478c424bb398929088eeb78d64e97916e71dc67e18a610c4719b75c\": container with ID starting with e99ba3bf4478c424bb398929088eeb78d64e97916e71dc67e18a610c4719b75c not found: ID does not exist"
Nov 03 20:31:47.180827 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:47.169256    5570 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d450803-ade0-428d-9029-d71459b2fc22-maas-api-tls\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:31:47.180827 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:47.169296    5570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d450803-ade0-428d-9029-d71459b2fc22-openshift-service-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:31:47.180827 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:47.169309    5570 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d450803-ade0-428d-9029-d71459b2fc22-metrics-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:31:47.180827 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:47.169322    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qkrq4\" (UniqueName: \"kubernetes.io/projected/5d450803-ade0-428d-9029-d71459b2fc22-kube-api-access-qkrq4\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:31:47.982990 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:47.982961    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-addf1a-56979459f6-bt9cd"
Nov 03 20:31:48.038692 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:48.038652    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-addf1a-56979459f6-bt9cd"]
Nov 03 20:31:48.052699 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:48.052666    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-addf1a-56979459f6-bt9cd"]
Nov 03 20:31:48.841284 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:48.841250    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-60efdf-57848f5c8c-qjzhb"
Nov 03 20:31:49.967559 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:49.967510    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d450803-ade0-428d-9029-d71459b2fc22" path="/var/lib/kubelet/pods/5d450803-ade0-428d-9029-d71459b2fc22/volumes"
Nov 03 20:31:54.580893 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:54.580855    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-addf1a-openshift-default-69ffb7968c-pgb22"]
Nov 03 20:31:54.581692 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:54.581661    5570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-addf1a-openshift-default-69ffb7968c-pgb22" podUID="e5d67e4b-c834-426f-8df5-8d6dc4ae77ab" containerName="istio-proxy" containerID="cri-o://0288429d09834921619e75ac8123d49724e489c65d28a7167ef2b79afb94848f" gracePeriod=30
Nov 03 20:31:57.230817 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:31:57.230756    5570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8bwhz" containerName="extractor"
Nov 03 20:31:59.419184 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:59.419142    5570 patch_prober.go:29] interesting pod/e2e-api-a-addf1a-openshift-default-69ffb7968c-pgb22 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 20:31:59.419721 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:31:59.419403    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-addf1a-openshift-default-69ffb7968c-pgb22" podUID="e5d67e4b-c834-426f-8df5-8d6dc4ae77ab" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 20:32:00.014525 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:00.014481    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-addf1a-openshift-default-69ffb7968c-pgb22"
Nov 03 20:32:00.103746 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:00.103715    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e5d67e4b-c834-426f-8df5-8d6dc4ae77ab-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5d67e4b-c834-426f-8df5-8d6dc4ae77ab-istio-podinfo\") pod \"e5d67e4b-c834-426f-8df5-8d6dc4ae77ab\" (UID: \"e5d67e4b-c834-426f-8df5-8d6dc4ae77ab\") "
Nov 03 20:32:00.147931 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:00.103963    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5d67e4b-c834-426f-8df5-8d6dc4ae77ab-kube-api-access-48wkg\" (UniqueName: \"kubernetes.io/projected/e5d67e4b-c834-426f-8df5-8d6dc4ae77ab-kube-api-access-48wkg\") pod \"e5d67e4b-c834-426f-8df5-8d6dc4ae77ab\" (UID: \"e5d67e4b-c834-426f-8df5-8d6dc4ae77ab\") "
Nov 03 20:32:00.147931 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:00.104082    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5d67e4b-c834-426f-8df5-8d6dc4ae77ab-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5d67e4b-c834-426f-8df5-8d6dc4ae77ab-istiod-ca-cert\") pod \"e5d67e4b-c834-426f-8df5-8d6dc4ae77ab\" (UID: \"e5d67e4b-c834-426f-8df5-8d6dc4ae77ab\") "
Nov 03 20:32:00.147931 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:00.104147    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5d67e4b-c834-426f-8df5-8d6dc4ae77ab-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5d67e4b-c834-426f-8df5-8d6dc4ae77ab-credential-socket\") pod \"e5d67e4b-c834-426f-8df5-8d6dc4ae77ab\" (UID: \"e5d67e4b-c834-426f-8df5-8d6dc4ae77ab\") "
Nov 03 20:32:00.147931 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:00.104190    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5d67e4b-c834-426f-8df5-8d6dc4ae77ab-istio-token\" (UniqueName: \"kubernetes.io/projected/e5d67e4b-c834-426f-8df5-8d6dc4ae77ab-istio-token\") pod \"e5d67e4b-c834-426f-8df5-8d6dc4ae77ab\" (UID: \"e5d67e4b-c834-426f-8df5-8d6dc4ae77ab\") "
Nov 03 20:32:00.147931 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:00.104231    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5d67e4b-c834-426f-8df5-8d6dc4ae77ab-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5d67e4b-c834-426f-8df5-8d6dc4ae77ab-istio-data\") pod \"e5d67e4b-c834-426f-8df5-8d6dc4ae77ab\" (UID: \"e5d67e4b-c834-426f-8df5-8d6dc4ae77ab\") "
Nov 03 20:32:00.147931 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:00.104331    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5d67e4b-c834-426f-8df5-8d6dc4ae77ab-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5d67e4b-c834-426f-8df5-8d6dc4ae77ab-istio-envoy\") pod \"e5d67e4b-c834-426f-8df5-8d6dc4ae77ab\" (UID: \"e5d67e4b-c834-426f-8df5-8d6dc4ae77ab\") "
Nov 03 20:32:00.147931 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:00.104397    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5d67e4b-c834-426f-8df5-8d6dc4ae77ab-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5d67e4b-c834-426f-8df5-8d6dc4ae77ab-workload-certs\") pod \"e5d67e4b-c834-426f-8df5-8d6dc4ae77ab\" (UID: \"e5d67e4b-c834-426f-8df5-8d6dc4ae77ab\") "
Nov 03 20:32:00.147931 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:00.104393    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5d67e4b-c834-426f-8df5-8d6dc4ae77ab-credential-socket" pod "e5d67e4b-c834-426f-8df5-8d6dc4ae77ab" (UID: "e5d67e4b-c834-426f-8df5-8d6dc4ae77ab"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:32:00.147931 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:00.104428    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5d67e4b-c834-426f-8df5-8d6dc4ae77ab-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5d67e4b-c834-426f-8df5-8d6dc4ae77ab-workload-socket\") pod \"e5d67e4b-c834-426f-8df5-8d6dc4ae77ab\" (UID: \"e5d67e4b-c834-426f-8df5-8d6dc4ae77ab\") "
Nov 03 20:32:00.147931 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:00.104971    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5d67e4b-c834-426f-8df5-8d6dc4ae77ab-workload-certs" pod "e5d67e4b-c834-426f-8df5-8d6dc4ae77ab" (UID: "e5d67e4b-c834-426f-8df5-8d6dc4ae77ab"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:32:00.147931 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:00.105132    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5d67e4b-c834-426f-8df5-8d6dc4ae77ab-istio-data" pod "e5d67e4b-c834-426f-8df5-8d6dc4ae77ab" (UID: "e5d67e4b-c834-426f-8df5-8d6dc4ae77ab"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:32:00.147931 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:00.105355    5570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5d67e4b-c834-426f-8df5-8d6dc4ae77ab-workload-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:32:00.147931 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:00.105521    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5d67e4b-c834-426f-8df5-8d6dc4ae77ab-istiod-ca-cert" pod "e5d67e4b-c834-426f-8df5-8d6dc4ae77ab" (UID: "e5d67e4b-c834-426f-8df5-8d6dc4ae77ab"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:32:00.147931 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:00.105533    5570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5d67e4b-c834-426f-8df5-8d6dc4ae77ab-credential-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:32:00.147931 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:00.105552    5570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5d67e4b-c834-426f-8df5-8d6dc4ae77ab-istio-data\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:32:00.147931 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:00.105575    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5d67e4b-c834-426f-8df5-8d6dc4ae77ab-workload-socket" pod "e5d67e4b-c834-426f-8df5-8d6dc4ae77ab" (UID: "e5d67e4b-c834-426f-8df5-8d6dc4ae77ab"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:32:00.147931 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:00.106552    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e5d67e4b-c834-426f-8df5-8d6dc4ae77ab-istio-podinfo" pod "e5d67e4b-c834-426f-8df5-8d6dc4ae77ab" (UID: "e5d67e4b-c834-426f-8df5-8d6dc4ae77ab"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:32:00.147931 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:00.107169    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5d67e4b-c834-426f-8df5-8d6dc4ae77ab-istio-envoy" pod "e5d67e4b-c834-426f-8df5-8d6dc4ae77ab" (UID: "e5d67e4b-c834-426f-8df5-8d6dc4ae77ab"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:32:00.147931 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:00.107486    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5d67e4b-c834-426f-8df5-8d6dc4ae77ab-istio-token" pod "e5d67e4b-c834-426f-8df5-8d6dc4ae77ab" (UID: "e5d67e4b-c834-426f-8df5-8d6dc4ae77ab"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:32:00.147931 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:00.107583    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5d67e4b-c834-426f-8df5-8d6dc4ae77ab-kube-api-access-48wkg" pod "e5d67e4b-c834-426f-8df5-8d6dc4ae77ab" (UID: "e5d67e4b-c834-426f-8df5-8d6dc4ae77ab"). InnerVolumeSpecName "kube-api-access-48wkg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:32:00.147931 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:00.144708    5570 generic.go:356] "Generic (PLEG): container finished" podID="e5d67e4b-c834-426f-8df5-8d6dc4ae77ab" containerID="0288429d09834921619e75ac8123d49724e489c65d28a7167ef2b79afb94848f" exitCode=0
Nov 03 20:32:00.147931 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:00.144763    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-addf1a-openshift-default-69ffb7968c-pgb22"
Nov 03 20:32:00.147931 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:00.144819    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-addf1a-openshift-default-69ffb7968c-pgb22" event={"ID":"e5d67e4b-c834-426f-8df5-8d6dc4ae77ab","Type":"ContainerDied","Data":"0288429d09834921619e75ac8123d49724e489c65d28a7167ef2b79afb94848f"}
Nov 03 20:32:00.147931 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:00.144881    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-addf1a-openshift-default-69ffb7968c-pgb22" event={"ID":"e5d67e4b-c834-426f-8df5-8d6dc4ae77ab","Type":"ContainerDied","Data":"08d353ae76158d0f9cd74eab287bde363c8bddab6fde6e2150fb1df959ccdd30"}
Nov 03 20:32:00.147931 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:00.144910    5570 scope.go:122] "RemoveContainer" containerID="0288429d09834921619e75ac8123d49724e489c65d28a7167ef2b79afb94848f"
Nov 03 20:32:00.156657 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:00.156636    5570 scope.go:122] "RemoveContainer" containerID="5ede59604e62a22821fb828b83994fcbd9b4102722a967137e0cf46fa579cc50"
Nov 03 20:32:00.167174 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:00.167152    5570 scope.go:122] "RemoveContainer" containerID="0288429d09834921619e75ac8123d49724e489c65d28a7167ef2b79afb94848f"
Nov 03 20:32:00.167544 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:32:00.167522    5570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0288429d09834921619e75ac8123d49724e489c65d28a7167ef2b79afb94848f\": container with ID starting with 0288429d09834921619e75ac8123d49724e489c65d28a7167ef2b79afb94848f not found: ID does not exist" containerID="0288429d09834921619e75ac8123d49724e489c65d28a7167ef2b79afb94848f"
Nov 03 20:32:00.167600 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:00.167554    5570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0288429d09834921619e75ac8123d49724e489c65d28a7167ef2b79afb94848f"} err="failed to get container status \"0288429d09834921619e75ac8123d49724e489c65d28a7167ef2b79afb94848f\": rpc error: code = NotFound desc = could not find container \"0288429d09834921619e75ac8123d49724e489c65d28a7167ef2b79afb94848f\": container with ID starting with 0288429d09834921619e75ac8123d49724e489c65d28a7167ef2b79afb94848f not found: ID does not exist"
Nov 03 20:32:00.167600 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:00.167572    5570 scope.go:122] "RemoveContainer" containerID="5ede59604e62a22821fb828b83994fcbd9b4102722a967137e0cf46fa579cc50"
Nov 03 20:32:00.167895 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:32:00.167873    5570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ede59604e62a22821fb828b83994fcbd9b4102722a967137e0cf46fa579cc50\": container with ID starting with 5ede59604e62a22821fb828b83994fcbd9b4102722a967137e0cf46fa579cc50 not found: ID does not exist" containerID="5ede59604e62a22821fb828b83994fcbd9b4102722a967137e0cf46fa579cc50"
Nov 03 20:32:00.167978 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:00.167905    5570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ede59604e62a22821fb828b83994fcbd9b4102722a967137e0cf46fa579cc50"} err="failed to get container status \"5ede59604e62a22821fb828b83994fcbd9b4102722a967137e0cf46fa579cc50\": rpc error: code = NotFound desc = could not find container \"5ede59604e62a22821fb828b83994fcbd9b4102722a967137e0cf46fa579cc50\": container with ID starting with 5ede59604e62a22821fb828b83994fcbd9b4102722a967137e0cf46fa579cc50 not found: ID does not exist"
Nov 03 20:32:00.200940 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:00.200906    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-addf1a-openshift-default-69ffb7968c-pgb22"]
Nov 03 20:32:00.206941 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:00.206908    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-addf1a-openshift-default-69ffb7968c-pgb22"]
Nov 03 20:32:00.207213 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:00.207195    5570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5d67e4b-c834-426f-8df5-8d6dc4ae77ab-workload-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:32:00.207266 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:00.207216    5570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5d67e4b-c834-426f-8df5-8d6dc4ae77ab-istio-podinfo\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:32:00.207266 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:00.207225    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48wkg\" (UniqueName: \"kubernetes.io/projected/e5d67e4b-c834-426f-8df5-8d6dc4ae77ab-kube-api-access-48wkg\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:32:00.207266 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:00.207234    5570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5d67e4b-c834-426f-8df5-8d6dc4ae77ab-istiod-ca-cert\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:32:00.207266 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:00.207243    5570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5d67e4b-c834-426f-8df5-8d6dc4ae77ab-istio-token\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:32:00.207266 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:00.207251    5570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5d67e4b-c834-426f-8df5-8d6dc4ae77ab-istio-envoy\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:32:01.154594 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:01.154560    5570 generic.go:356] "Generic (PLEG): container finished" podID="b9ec153a-ac81-4e70-8524-c3121c31c41f" containerID="ee06ea841d618a90e18d8c5fa9a485cb9edcb998c251762a4960ceceeef12d2b" exitCode=0
Nov 03 20:32:01.155048 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:01.154636    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d05604-openshift-default-6bdffb549c-w62g5" event={"ID":"b9ec153a-ac81-4e70-8524-c3121c31c41f","Type":"ContainerDied","Data":"ee06ea841d618a90e18d8c5fa9a485cb9edcb998c251762a4960ceceeef12d2b"}
Nov 03 20:32:01.155221 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:01.155205    5570 scope.go:122] "RemoveContainer" containerID="ee06ea841d618a90e18d8c5fa9a485cb9edcb998c251762a4960ceceeef12d2b"
Nov 03 20:32:01.157112 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:01.157081    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:32:01.157214 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:01.157153    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:32:01.157214 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:01.157183    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:32:01.975798 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:01.974894    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5d67e4b-c834-426f-8df5-8d6dc4ae77ab" path="/var/lib/kubelet/pods/e5d67e4b-c834-426f-8df5-8d6dc4ae77ab/volumes"
Nov 03 20:32:02.169129 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:02.169094    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d05604-openshift-default-6bdffb549c-w62g5" event={"ID":"b9ec153a-ac81-4e70-8524-c3121c31c41f","Type":"ContainerStarted","Data":"e9716d133038c46e93015bdc1dbb0c3fd5a6381f966a6284449b58660e1465e2"}
Nov 03 20:32:03.007025 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:03.006987    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-fab871-5c95886dbd-x49r6"]
Nov 03 20:32:03.064949 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:03.064406    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-fab871-5c8bd9f8dc-qv2jl"]
Nov 03 20:32:03.133226 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:03.133193    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-d05604-openshift-default-6bdffb549c-w62g5"
Nov 03 20:32:03.133502 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:03.133405    5570 patch_prober.go:29] interesting pod/e2e-api-b-d05604-openshift-default-6bdffb549c-w62g5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body=
Nov 03 20:32:03.133502 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:03.133462    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d05604-openshift-default-6bdffb549c-w62g5" podUID="b9ec153a-ac81-4e70-8524-c3121c31c41f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused"
Nov 03 20:32:04.133933 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:04.133890    5570 patch_prober.go:29] interesting pod/e2e-api-b-d05604-openshift-default-6bdffb549c-w62g5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body=
Nov 03 20:32:04.134443 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:04.133961    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d05604-openshift-default-6bdffb549c-w62g5" podUID="b9ec153a-ac81-4e70-8524-c3121c31c41f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused"
Nov 03 20:32:04.604023 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:04.603988    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-de82c8-78dc548fc4-24g87"]
Nov 03 20:32:05.133728 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:05.133696    5570 patch_prober.go:29] interesting pod/e2e-api-b-d05604-openshift-default-6bdffb549c-w62g5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body=
Nov 03 20:32:05.133938 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:05.133749    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d05604-openshift-default-6bdffb549c-w62g5" podUID="b9ec153a-ac81-4e70-8524-c3121c31c41f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused"
Nov 03 20:32:05.459422 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:05.459374    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d1aa66-bcd7bb5b6-d9dcs"]
Nov 03 20:32:05.505632 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:05.505563    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d1aa66-7b7c6c6455-n8n8v"]
Nov 03 20:32:06.133441 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:06.133385    5570 patch_prober.go:29] interesting pod/e2e-api-b-d05604-openshift-default-6bdffb549c-w62g5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body=
Nov 03 20:32:06.133661 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:06.133461    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d05604-openshift-default-6bdffb549c-w62g5" podUID="b9ec153a-ac81-4e70-8524-c3121c31c41f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused"
Nov 03 20:32:06.916818 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:06.916760    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g69f8"]
Nov 03 20:32:07.222874 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:07.160871    5570 patch_prober.go:29] interesting pod/e2e-api-b-d05604-openshift-default-6bdffb549c-w62g5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body=
Nov 03 20:32:07.222874 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:07.160929    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d05604-openshift-default-6bdffb549c-w62g5" podUID="b9ec153a-ac81-4e70-8524-c3121c31c41f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused"
Nov 03 20:32:08.009493 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:08.008868    5570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-fab871-5c95886dbd-x49r6" podUID="1b83e8f6-3b31-427c-8c6c-57e57d1bc4d4" containerName="payload-processing" containerID="cri-o://f8a37f0f08f601fa35ac6be2ab6ab87b4c71ce32b11bb77a0c2d8381bfb28596" gracePeriod=25
Nov 03 20:32:08.079467 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:08.073460    5570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fab871-5c8bd9f8dc-qv2jl" podUID="ef03b5c7-3817-4d77-b2c8-e450267276cc" containerName="payload-pre-processing" containerID="cri-o://a34dcc00f4c853fe62f1d9a0716d2150d699d4107fbb70da668bbb6965c676d9" gracePeriod=25
Nov 03 20:32:08.134272 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:08.134012    5570 patch_prober.go:29] interesting pod/e2e-api-b-d05604-openshift-default-6bdffb549c-w62g5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body=
Nov 03 20:32:08.134492 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:08.134467    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d05604-openshift-default-6bdffb549c-w62g5" podUID="b9ec153a-ac81-4e70-8524-c3121c31c41f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused"
Nov 03 20:32:08.388243 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:08.388170    5570 generic.go:356] "Generic (PLEG): container finished" podID="1b83e8f6-3b31-427c-8c6c-57e57d1bc4d4" containerID="f8a37f0f08f601fa35ac6be2ab6ab87b4c71ce32b11bb77a0c2d8381bfb28596" exitCode=0
Nov 03 20:32:08.388590 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:08.388506    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-fab871-5c95886dbd-x49r6" event={"ID":"1b83e8f6-3b31-427c-8c6c-57e57d1bc4d4","Type":"ContainerDied","Data":"f8a37f0f08f601fa35ac6be2ab6ab87b4c71ce32b11bb77a0c2d8381bfb28596"}
Nov 03 20:32:08.391807 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:08.391719    5570 generic.go:356] "Generic (PLEG): container finished" podID="ef03b5c7-3817-4d77-b2c8-e450267276cc" containerID="a34dcc00f4c853fe62f1d9a0716d2150d699d4107fbb70da668bbb6965c676d9" exitCode=0
Nov 03 20:32:08.392118 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:08.392040    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fab871-5c8bd9f8dc-qv2jl" event={"ID":"ef03b5c7-3817-4d77-b2c8-e450267276cc","Type":"ContainerDied","Data":"a34dcc00f4c853fe62f1d9a0716d2150d699d4107fbb70da668bbb6965c676d9"}
Nov 03 20:32:08.409637 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:08.409593    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-d05604-openshift-default-6bdffb549c-w62g5"
Nov 03 20:32:08.492578 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:08.492387    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-fab871-5c95886dbd-x49r6"
Nov 03 20:32:08.767251 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:08.758203    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b83e8f6-3b31-427c-8c6c-57e57d1bc4d4-kube-api-access-fcnvf\" (UniqueName: \"kubernetes.io/projected/1b83e8f6-3b31-427c-8c6c-57e57d1bc4d4-kube-api-access-fcnvf\") pod \"1b83e8f6-3b31-427c-8c6c-57e57d1bc4d4\" (UID: \"1b83e8f6-3b31-427c-8c6c-57e57d1bc4d4\") "
Nov 03 20:32:08.767251 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:08.758304    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b83e8f6-3b31-427c-8c6c-57e57d1bc4d4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b83e8f6-3b31-427c-8c6c-57e57d1bc4d4-plugins-config-volume\") pod \"1b83e8f6-3b31-427c-8c6c-57e57d1bc4d4\" (UID: \"1b83e8f6-3b31-427c-8c6c-57e57d1bc4d4\") "
Nov 03 20:32:08.767251 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:08.759533    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b83e8f6-3b31-427c-8c6c-57e57d1bc4d4-plugins-config-volume" pod "1b83e8f6-3b31-427c-8c6c-57e57d1bc4d4" (UID: "1b83e8f6-3b31-427c-8c6c-57e57d1bc4d4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:32:08.864219 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:08.864168    5570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b83e8f6-3b31-427c-8c6c-57e57d1bc4d4-plugins-config-volume\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:32:08.865249 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:08.865213    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b83e8f6-3b31-427c-8c6c-57e57d1bc4d4-kube-api-access-fcnvf" pod "1b83e8f6-3b31-427c-8c6c-57e57d1bc4d4" (UID: "1b83e8f6-3b31-427c-8c6c-57e57d1bc4d4"). InnerVolumeSpecName "kube-api-access-fcnvf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:32:08.966526 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:08.966492    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fcnvf\" (UniqueName: \"kubernetes.io/projected/1b83e8f6-3b31-427c-8c6c-57e57d1bc4d4-kube-api-access-fcnvf\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:32:08.970253 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:08.970164    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fab871-5c8bd9f8dc-qv2jl"
Nov 03 20:32:09.071737 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:09.071627    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef03b5c7-3817-4d77-b2c8-e450267276cc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef03b5c7-3817-4d77-b2c8-e450267276cc-plugins-config-volume\") pod \"ef03b5c7-3817-4d77-b2c8-e450267276cc\" (UID: \"ef03b5c7-3817-4d77-b2c8-e450267276cc\") "
Nov 03 20:32:09.071737 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:09.071721    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef03b5c7-3817-4d77-b2c8-e450267276cc-kube-api-access-9mlbr\" (UniqueName: \"kubernetes.io/projected/ef03b5c7-3817-4d77-b2c8-e450267276cc-kube-api-access-9mlbr\") pod \"ef03b5c7-3817-4d77-b2c8-e450267276cc\" (UID: \"ef03b5c7-3817-4d77-b2c8-e450267276cc\") "
Nov 03 20:32:09.072340 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:09.072245    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef03b5c7-3817-4d77-b2c8-e450267276cc-plugins-config-volume" pod "ef03b5c7-3817-4d77-b2c8-e450267276cc" (UID: "ef03b5c7-3817-4d77-b2c8-e450267276cc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:32:09.072728 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:09.072513    5570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef03b5c7-3817-4d77-b2c8-e450267276cc-plugins-config-volume\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:32:09.075484 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:09.075451    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef03b5c7-3817-4d77-b2c8-e450267276cc-kube-api-access-9mlbr" pod "ef03b5c7-3817-4d77-b2c8-e450267276cc" (UID: "ef03b5c7-3817-4d77-b2c8-e450267276cc"). InnerVolumeSpecName "kube-api-access-9mlbr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:32:09.162861 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:09.146144    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-d05604-openshift-default-6bdffb549c-w62g5"
Nov 03 20:32:09.174568 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:09.174533    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mlbr\" (UniqueName: \"kubernetes.io/projected/ef03b5c7-3817-4d77-b2c8-e450267276cc-kube-api-access-9mlbr\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:32:09.401999 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:09.401910    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-fab871-5c95886dbd-x49r6"
Nov 03 20:32:09.401999 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:09.401934    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-fab871-5c95886dbd-x49r6" event={"ID":"1b83e8f6-3b31-427c-8c6c-57e57d1bc4d4","Type":"ContainerDied","Data":"f524d4b41605a282ffa347ade405170c16f829ef0f24f5060702df5626de69ed"}
Nov 03 20:32:09.401999 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:09.401996    5570 scope.go:122] "RemoveContainer" containerID="f8a37f0f08f601fa35ac6be2ab6ab87b4c71ce32b11bb77a0c2d8381bfb28596"
Nov 03 20:32:09.403974 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:09.403950    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fab871-5c8bd9f8dc-qv2jl"
Nov 03 20:32:09.404134 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:09.403981    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fab871-5c8bd9f8dc-qv2jl" event={"ID":"ef03b5c7-3817-4d77-b2c8-e450267276cc","Type":"ContainerDied","Data":"c63c5161e9acba9128d56132603bba955ef50833ae0165eec30db2bce3d9a56d"}
Nov 03 20:32:09.405272 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:09.405249    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-d05604-openshift-default-6bdffb549c-w62g5"
Nov 03 20:32:09.414829 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:09.414806    5570 scope.go:122] "RemoveContainer" containerID="a34dcc00f4c853fe62f1d9a0716d2150d699d4107fbb70da668bbb6965c676d9"
Nov 03 20:32:09.508627 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:09.496996    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-fab871-5c8bd9f8dc-qv2jl"]
Nov 03 20:32:09.508627 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:09.507237    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-fab871-5c8bd9f8dc-qv2jl"]
Nov 03 20:32:09.618800 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:09.610989    5570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-de82c8-78dc548fc4-24g87" podUID="ec1dc0b9-d778-48b7-986a-0d460bb6a52f" containerName="payload-processing" containerID="cri-o://993c17641348655709085554057bbcd967eca31e3684298b29a1ef1929137826" gracePeriod=25
Nov 03 20:32:09.656864 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:09.636542    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-fab871-5c95886dbd-x49r6"]
Nov 03 20:32:09.656864 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:09.647537    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-fab871-5c95886dbd-x49r6"]
Nov 03 20:32:09.962128 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:09.962018    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-de82c8-78dc548fc4-24g87"
Nov 03 20:32:09.972043 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:09.972004    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b83e8f6-3b31-427c-8c6c-57e57d1bc4d4" path="/var/lib/kubelet/pods/1b83e8f6-3b31-427c-8c6c-57e57d1bc4d4/volumes"
Nov 03 20:32:09.972597 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:09.972578    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef03b5c7-3817-4d77-b2c8-e450267276cc" path="/var/lib/kubelet/pods/ef03b5c7-3817-4d77-b2c8-e450267276cc/volumes"
Nov 03 20:32:10.016578 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:10.016540    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec1dc0b9-d778-48b7-986a-0d460bb6a52f-kube-api-access-hhp6s\" (UniqueName: \"kubernetes.io/projected/ec1dc0b9-d778-48b7-986a-0d460bb6a52f-kube-api-access-hhp6s\") pod \"ec1dc0b9-d778-48b7-986a-0d460bb6a52f\" (UID: \"ec1dc0b9-d778-48b7-986a-0d460bb6a52f\") "
Nov 03 20:32:10.016823 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:10.016799    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec1dc0b9-d778-48b7-986a-0d460bb6a52f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec1dc0b9-d778-48b7-986a-0d460bb6a52f-plugins-config-volume\") pod \"ec1dc0b9-d778-48b7-986a-0d460bb6a52f\" (UID: \"ec1dc0b9-d778-48b7-986a-0d460bb6a52f\") "
Nov 03 20:32:10.017181 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:10.017155    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec1dc0b9-d778-48b7-986a-0d460bb6a52f-plugins-config-volume" pod "ec1dc0b9-d778-48b7-986a-0d460bb6a52f" (UID: "ec1dc0b9-d778-48b7-986a-0d460bb6a52f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:32:10.017829 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:10.017766    5570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec1dc0b9-d778-48b7-986a-0d460bb6a52f-plugins-config-volume\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:32:10.019022 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:10.018994    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec1dc0b9-d778-48b7-986a-0d460bb6a52f-kube-api-access-hhp6s" pod "ec1dc0b9-d778-48b7-986a-0d460bb6a52f" (UID: "ec1dc0b9-d778-48b7-986a-0d460bb6a52f"). InnerVolumeSpecName "kube-api-access-hhp6s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:32:10.120190 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:10.120152    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhp6s\" (UniqueName: \"kubernetes.io/projected/ec1dc0b9-d778-48b7-986a-0d460bb6a52f-kube-api-access-hhp6s\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:32:10.414971 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:10.414932    5570 generic.go:356] "Generic (PLEG): container finished" podID="ec1dc0b9-d778-48b7-986a-0d460bb6a52f" containerID="993c17641348655709085554057bbcd967eca31e3684298b29a1ef1929137826" exitCode=0
Nov 03 20:32:10.415171 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:10.414980    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-de82c8-78dc548fc4-24g87"
Nov 03 20:32:10.415171 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:10.414991    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-de82c8-78dc548fc4-24g87" event={"ID":"ec1dc0b9-d778-48b7-986a-0d460bb6a52f","Type":"ContainerDied","Data":"993c17641348655709085554057bbcd967eca31e3684298b29a1ef1929137826"}
Nov 03 20:32:10.415171 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:10.415051    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-de82c8-78dc548fc4-24g87" event={"ID":"ec1dc0b9-d778-48b7-986a-0d460bb6a52f","Type":"ContainerDied","Data":"9f9f5f310addd3b32215994257334bf8ab8e4a389edd3bb6eac7c9cbddde890f"}
Nov 03 20:32:10.415171 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:10.415076    5570 scope.go:122] "RemoveContainer" containerID="993c17641348655709085554057bbcd967eca31e3684298b29a1ef1929137826"
Nov 03 20:32:10.428665 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:10.428638    5570 scope.go:122] "RemoveContainer" containerID="993c17641348655709085554057bbcd967eca31e3684298b29a1ef1929137826"
Nov 03 20:32:10.429911 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:32:10.429876    5570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"993c17641348655709085554057bbcd967eca31e3684298b29a1ef1929137826\": container with ID starting with 993c17641348655709085554057bbcd967eca31e3684298b29a1ef1929137826 not found: ID does not exist" containerID="993c17641348655709085554057bbcd967eca31e3684298b29a1ef1929137826"
Nov 03 20:32:10.430070 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:10.429919    5570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"993c17641348655709085554057bbcd967eca31e3684298b29a1ef1929137826"} err="failed to get container status \"993c17641348655709085554057bbcd967eca31e3684298b29a1ef1929137826\": rpc error: code = NotFound desc = could not find container \"993c17641348655709085554057bbcd967eca31e3684298b29a1ef1929137826\": container with ID starting with 993c17641348655709085554057bbcd967eca31e3684298b29a1ef1929137826 not found: ID does not exist"
Nov 03 20:32:10.461977 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:10.461912    5570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-d1aa66-bcd7bb5b6-d9dcs" podUID="1da1c326-d687-48d8-aa52-42392ef0e1d2" containerName="payload-processing" containerID="cri-o://2980b208da095e8ce4d4bb2471ca321f444bf4bde5f56d9fc62e72ab0035cfa8" gracePeriod=25
Nov 03 20:32:10.491753 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:10.491714    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-de82c8-78dc548fc4-24g87"]
Nov 03 20:32:10.507001 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:10.506952    5570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d1aa66-7b7c6c6455-n8n8v" podUID="51d6f999-63b2-4461-a82f-65ff671546f3" containerName="payload-pre-processing" containerID="cri-o://5602baac61cd7ea0e2f1947dda9357aee0ec96356550d84113e2ab3bd9ceda1d" gracePeriod=25
Nov 03 20:32:10.510092 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:10.510063    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-de82c8-78dc548fc4-24g87"]
Nov 03 20:32:10.778526 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:10.778489    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c4a71d-585549556c-x2sdc"]
Nov 03 20:32:10.778731 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:10.778703    5570 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c4a71d-585549556c-x2sdc" podUID="18e420ed-849e-4e73-b1b0-ef8167ddd383" containerName="maas-api" containerID="cri-o://4662eba89a9576abeccb320306f55399dd68ad648321d3b7748aa9e695abdb92" gracePeriod=30
Nov 03 20:32:10.895329 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:10.895292    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-d1aa66-bcd7bb5b6-d9dcs"
Nov 03 20:32:10.903393 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:10.903368    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d1aa66-7b7c6c6455-n8n8v"
Nov 03 20:32:10.931763 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:10.931725    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1da1c326-d687-48d8-aa52-42392ef0e1d2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1da1c326-d687-48d8-aa52-42392ef0e1d2-plugins-config-volume\") pod \"1da1c326-d687-48d8-aa52-42392ef0e1d2\" (UID: \"1da1c326-d687-48d8-aa52-42392ef0e1d2\") "
Nov 03 20:32:10.931763 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:10.931785    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1da1c326-d687-48d8-aa52-42392ef0e1d2-kube-api-access-v2c2g\" (UniqueName: \"kubernetes.io/projected/1da1c326-d687-48d8-aa52-42392ef0e1d2-kube-api-access-v2c2g\") pod \"1da1c326-d687-48d8-aa52-42392ef0e1d2\" (UID: \"1da1c326-d687-48d8-aa52-42392ef0e1d2\") "
Nov 03 20:32:10.932745 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:10.931956    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51d6f999-63b2-4461-a82f-65ff671546f3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51d6f999-63b2-4461-a82f-65ff671546f3-plugins-config-volume\") pod \"51d6f999-63b2-4461-a82f-65ff671546f3\" (UID: \"51d6f999-63b2-4461-a82f-65ff671546f3\") "
Nov 03 20:32:10.932745 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:10.932014    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51d6f999-63b2-4461-a82f-65ff671546f3-kube-api-access-5spxf\" (UniqueName: \"kubernetes.io/projected/51d6f999-63b2-4461-a82f-65ff671546f3-kube-api-access-5spxf\") pod \"51d6f999-63b2-4461-a82f-65ff671546f3\" (UID: \"51d6f999-63b2-4461-a82f-65ff671546f3\") "
Nov 03 20:32:10.932745 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:10.932503    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1da1c326-d687-48d8-aa52-42392ef0e1d2-plugins-config-volume" pod "1da1c326-d687-48d8-aa52-42392ef0e1d2" (UID: "1da1c326-d687-48d8-aa52-42392ef0e1d2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:32:10.932745 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:10.932582    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51d6f999-63b2-4461-a82f-65ff671546f3-plugins-config-volume" pod "51d6f999-63b2-4461-a82f-65ff671546f3" (UID: "51d6f999-63b2-4461-a82f-65ff671546f3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:32:10.933039 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:10.933019    5570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1da1c326-d687-48d8-aa52-42392ef0e1d2-plugins-config-volume\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:32:10.933104 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:10.933042    5570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51d6f999-63b2-4461-a82f-65ff671546f3-plugins-config-volume\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:32:10.934405 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:10.934372    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1da1c326-d687-48d8-aa52-42392ef0e1d2-kube-api-access-v2c2g" pod "1da1c326-d687-48d8-aa52-42392ef0e1d2" (UID: "1da1c326-d687-48d8-aa52-42392ef0e1d2"). InnerVolumeSpecName "kube-api-access-v2c2g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:32:10.934827 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:10.934748    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51d6f999-63b2-4461-a82f-65ff671546f3-kube-api-access-5spxf" pod "51d6f999-63b2-4461-a82f-65ff671546f3" (UID: "51d6f999-63b2-4461-a82f-65ff671546f3"). InnerVolumeSpecName "kube-api-access-5spxf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:32:11.326796 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:11.323853    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2c2g\" (UniqueName: \"kubernetes.io/projected/1da1c326-d687-48d8-aa52-42392ef0e1d2-kube-api-access-v2c2g\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:32:11.326796 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:11.323897    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5spxf\" (UniqueName: \"kubernetes.io/projected/51d6f999-63b2-4461-a82f-65ff671546f3-kube-api-access-5spxf\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:32:11.818546 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:11.817542    5570 generic.go:356] "Generic (PLEG): container finished" podID="18e420ed-849e-4e73-b1b0-ef8167ddd383" containerID="4662eba89a9576abeccb320306f55399dd68ad648321d3b7748aa9e695abdb92" exitCode=0
Nov 03 20:32:11.818546 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:11.817907    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c4a71d-585549556c-x2sdc" event={"ID":"18e420ed-849e-4e73-b1b0-ef8167ddd383","Type":"ContainerDied","Data":"4662eba89a9576abeccb320306f55399dd68ad648321d3b7748aa9e695abdb92"}
Nov 03 20:32:11.829236 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:11.825552    5570 generic.go:356] "Generic (PLEG): container finished" podID="1da1c326-d687-48d8-aa52-42392ef0e1d2" containerID="2980b208da095e8ce4d4bb2471ca321f444bf4bde5f56d9fc62e72ab0035cfa8" exitCode=0
Nov 03 20:32:11.829236 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:11.825713    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-d1aa66-bcd7bb5b6-d9dcs" event={"ID":"1da1c326-d687-48d8-aa52-42392ef0e1d2","Type":"ContainerDied","Data":"2980b208da095e8ce4d4bb2471ca321f444bf4bde5f56d9fc62e72ab0035cfa8"}
Nov 03 20:32:11.829236 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:11.825764    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-d1aa66-bcd7bb5b6-d9dcs"
Nov 03 20:32:11.829236 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:11.825823    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-d1aa66-bcd7bb5b6-d9dcs" event={"ID":"1da1c326-d687-48d8-aa52-42392ef0e1d2","Type":"ContainerDied","Data":"870aaf0381fa3bfbf2baa50551ae333d317b66fcf81d992b4b871c519bcdcbb4"}
Nov 03 20:32:11.829236 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:11.826660    5570 scope.go:122] "RemoveContainer" containerID="2980b208da095e8ce4d4bb2471ca321f444bf4bde5f56d9fc62e72ab0035cfa8"
Nov 03 20:32:11.829236 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:11.828374    5570 generic.go:356] "Generic (PLEG): container finished" podID="51d6f999-63b2-4461-a82f-65ff671546f3" containerID="5602baac61cd7ea0e2f1947dda9357aee0ec96356550d84113e2ab3bd9ceda1d" exitCode=0
Nov 03 20:32:11.829236 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:11.828463    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d1aa66-7b7c6c6455-n8n8v" event={"ID":"51d6f999-63b2-4461-a82f-65ff671546f3","Type":"ContainerDied","Data":"5602baac61cd7ea0e2f1947dda9357aee0ec96356550d84113e2ab3bd9ceda1d"}
Nov 03 20:32:11.829236 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:11.828520    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d1aa66-7b7c6c6455-n8n8v" event={"ID":"51d6f999-63b2-4461-a82f-65ff671546f3","Type":"ContainerDied","Data":"2ff7dec511122d2725ecce94b248d5501eac506fc207cc0b547ddfb264d69487"}
Nov 03 20:32:11.829236 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:11.828651    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d1aa66-7b7c6c6455-n8n8v"
Nov 03 20:32:11.878441 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:11.878410    5570 scope.go:122] "RemoveContainer" containerID="2980b208da095e8ce4d4bb2471ca321f444bf4bde5f56d9fc62e72ab0035cfa8"
Nov 03 20:32:11.878946 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:32:11.878907    5570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2980b208da095e8ce4d4bb2471ca321f444bf4bde5f56d9fc62e72ab0035cfa8\": container with ID starting with 2980b208da095e8ce4d4bb2471ca321f444bf4bde5f56d9fc62e72ab0035cfa8 not found: ID does not exist" containerID="2980b208da095e8ce4d4bb2471ca321f444bf4bde5f56d9fc62e72ab0035cfa8"
Nov 03 20:32:11.879071 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:11.878962    5570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2980b208da095e8ce4d4bb2471ca321f444bf4bde5f56d9fc62e72ab0035cfa8"} err="failed to get container status \"2980b208da095e8ce4d4bb2471ca321f444bf4bde5f56d9fc62e72ab0035cfa8\": rpc error: code = NotFound desc = could not find container \"2980b208da095e8ce4d4bb2471ca321f444bf4bde5f56d9fc62e72ab0035cfa8\": container with ID starting with 2980b208da095e8ce4d4bb2471ca321f444bf4bde5f56d9fc62e72ab0035cfa8 not found: ID does not exist"
Nov 03 20:32:11.879071 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:11.878984    5570 scope.go:122] "RemoveContainer" containerID="5602baac61cd7ea0e2f1947dda9357aee0ec96356550d84113e2ab3bd9ceda1d"
Nov 03 20:32:11.894722 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:11.894381    5570 scope.go:122] "RemoveContainer" containerID="5602baac61cd7ea0e2f1947dda9357aee0ec96356550d84113e2ab3bd9ceda1d"
Nov 03 20:32:11.894954 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:32:11.894799    5570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5602baac61cd7ea0e2f1947dda9357aee0ec96356550d84113e2ab3bd9ceda1d\": container with ID starting with 5602baac61cd7ea0e2f1947dda9357aee0ec96356550d84113e2ab3bd9ceda1d not found: ID does not exist" containerID="5602baac61cd7ea0e2f1947dda9357aee0ec96356550d84113e2ab3bd9ceda1d"
Nov 03 20:32:11.894954 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:11.894835    5570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5602baac61cd7ea0e2f1947dda9357aee0ec96356550d84113e2ab3bd9ceda1d"} err="failed to get container status \"5602baac61cd7ea0e2f1947dda9357aee0ec96356550d84113e2ab3bd9ceda1d\": rpc error: code = NotFound desc = could not find container \"5602baac61cd7ea0e2f1947dda9357aee0ec96356550d84113e2ab3bd9ceda1d\": container with ID starting with 5602baac61cd7ea0e2f1947dda9357aee0ec96356550d84113e2ab3bd9ceda1d not found: ID does not exist"
Nov 03 20:32:11.995845 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:11.995800    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c4a71d-585549556c-x2sdc"
Nov 03 20:32:12.001437 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:12.001406    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec1dc0b9-d778-48b7-986a-0d460bb6a52f" path="/var/lib/kubelet/pods/ec1dc0b9-d778-48b7-986a-0d460bb6a52f/volumes"
Nov 03 20:32:12.001765 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:12.001750    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d1aa66-bcd7bb5b6-d9dcs"]
Nov 03 20:32:12.001892 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:12.001802    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d1aa66-bcd7bb5b6-d9dcs"]
Nov 03 20:32:12.001892 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:12.001826    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d1aa66-7b7c6c6455-n8n8v"]
Nov 03 20:32:12.004024 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:12.003997    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d1aa66-7b7c6c6455-n8n8v"]
Nov 03 20:32:12.034315 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:12.034281    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18e420ed-849e-4e73-b1b0-ef8167ddd383-metrics-certs\" (UniqueName: \"kubernetes.io/secret/18e420ed-849e-4e73-b1b0-ef8167ddd383-metrics-certs\") pod \"18e420ed-849e-4e73-b1b0-ef8167ddd383\" (UID: \"18e420ed-849e-4e73-b1b0-ef8167ddd383\") "
Nov 03 20:32:12.034576 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:12.034560    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18e420ed-849e-4e73-b1b0-ef8167ddd383-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18e420ed-849e-4e73-b1b0-ef8167ddd383-openshift-service-ca\") pod \"18e420ed-849e-4e73-b1b0-ef8167ddd383\" (UID: \"18e420ed-849e-4e73-b1b0-ef8167ddd383\") "
Nov 03 20:32:12.034851 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:12.034832    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18e420ed-849e-4e73-b1b0-ef8167ddd383-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18e420ed-849e-4e73-b1b0-ef8167ddd383-maas-api-tls\") pod \"18e420ed-849e-4e73-b1b0-ef8167ddd383\" (UID: \"18e420ed-849e-4e73-b1b0-ef8167ddd383\") "
Nov 03 20:32:12.034986 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:12.034937    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18e420ed-849e-4e73-b1b0-ef8167ddd383-openshift-service-ca" pod "18e420ed-849e-4e73-b1b0-ef8167ddd383" (UID: "18e420ed-849e-4e73-b1b0-ef8167ddd383"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:32:12.035469 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:12.035146    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18e420ed-849e-4e73-b1b0-ef8167ddd383-kube-api-access-kmvz4\" (UniqueName: \"kubernetes.io/projected/18e420ed-849e-4e73-b1b0-ef8167ddd383-kube-api-access-kmvz4\") pod \"18e420ed-849e-4e73-b1b0-ef8167ddd383\" (UID: \"18e420ed-849e-4e73-b1b0-ef8167ddd383\") "
Nov 03 20:32:12.074803 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:12.052992    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18e420ed-849e-4e73-b1b0-ef8167ddd383-maas-api-tls" pod "18e420ed-849e-4e73-b1b0-ef8167ddd383" (UID: "18e420ed-849e-4e73-b1b0-ef8167ddd383"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:32:12.077294 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:12.076551    5570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18e420ed-849e-4e73-b1b0-ef8167ddd383-openshift-service-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:32:12.077294 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:12.076583    5570 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18e420ed-849e-4e73-b1b0-ef8167ddd383-maas-api-tls\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:32:12.081217 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:12.081186    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18e420ed-849e-4e73-b1b0-ef8167ddd383-metrics-certs" pod "18e420ed-849e-4e73-b1b0-ef8167ddd383" (UID: "18e420ed-849e-4e73-b1b0-ef8167ddd383"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:32:12.081322 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:12.081296    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18e420ed-849e-4e73-b1b0-ef8167ddd383-kube-api-access-kmvz4" pod "18e420ed-849e-4e73-b1b0-ef8167ddd383" (UID: "18e420ed-849e-4e73-b1b0-ef8167ddd383"). InnerVolumeSpecName "kube-api-access-kmvz4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:32:12.179127 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:12.179027    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kmvz4\" (UniqueName: \"kubernetes.io/projected/18e420ed-849e-4e73-b1b0-ef8167ddd383-kube-api-access-kmvz4\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:32:12.179127 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:12.179068    5570 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18e420ed-849e-4e73-b1b0-ef8167ddd383-metrics-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:32:12.836818 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:12.836743    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c4a71d-585549556c-x2sdc"
Nov 03 20:32:12.837347 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:12.836808    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c4a71d-585549556c-x2sdc" event={"ID":"18e420ed-849e-4e73-b1b0-ef8167ddd383","Type":"ContainerDied","Data":"950b85956f79d239ac6555dcc0934a356f0d459c94d8f627ad556f5e06185cd4"}
Nov 03 20:32:12.837347 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:12.836886    5570 scope.go:122] "RemoveContainer" containerID="4662eba89a9576abeccb320306f55399dd68ad648321d3b7748aa9e695abdb92"
Nov 03 20:32:12.909412 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:12.909365    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c4a71d-585549556c-x2sdc"]
Nov 03 20:32:12.922217 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:12.922186    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c4a71d-585549556c-x2sdc"]
Nov 03 20:32:13.972354 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:13.972321    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18e420ed-849e-4e73-b1b0-ef8167ddd383" path="/var/lib/kubelet/pods/18e420ed-849e-4e73-b1b0-ef8167ddd383/volumes"
Nov 03 20:32:13.972917 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:13.972895    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1da1c326-d687-48d8-aa52-42392ef0e1d2" path="/var/lib/kubelet/pods/1da1c326-d687-48d8-aa52-42392ef0e1d2/volumes"
Nov 03 20:32:13.973417 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:13.973401    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51d6f999-63b2-4461-a82f-65ff671546f3" path="/var/lib/kubelet/pods/51d6f999-63b2-4461-a82f-65ff671546f3/volumes"
Nov 03 20:32:18.406885 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:18.406719    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-bx2dr"]
Nov 03 20:32:18.407445 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:18.407044    5570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-bx2dr" podUID="ca63212a-77c0-4b97-8a23-5d0a518a55ef" containerName="istio-proxy" containerID="cri-o://2f27b981b1b00f2a6830b65b62ed19e37456b99a6f874e45cef4ab2c310ef747" gracePeriod=30
Nov 03 20:32:18.417865 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:18.417824    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-9pg4l"]
Nov 03 20:32:18.418142 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:18.418075    5570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-9pg4l" podUID="623f50bf-e25b-4647-9ffe-db4d163b2284" containerName="istio-proxy" containerID="cri-o://2a450d72048e8485b746d6c3a806cdc70b988882550514fbc8566ee1f05860a2" gracePeriod=30
Nov 03 20:32:18.930544 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:18.930501    5570 generic.go:356] "Generic (PLEG): container finished" podID="c6fb7442-714a-4e4a-a6cf-19326b50a929" containerID="4537e19896b788980331af4346ff30e28e6227d9410b794ab6558c526b1fc121" exitCode=0
Nov 03 20:32:18.930749 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:18.930573    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-60efdf-openshift-default-5dfc484849-k6445" event={"ID":"c6fb7442-714a-4e4a-a6cf-19326b50a929","Type":"ContainerDied","Data":"4537e19896b788980331af4346ff30e28e6227d9410b794ab6558c526b1fc121"}
Nov 03 20:32:18.931129 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:18.931109    5570 scope.go:122] "RemoveContainer" containerID="4537e19896b788980331af4346ff30e28e6227d9410b794ab6558c526b1fc121"
Nov 03 20:32:18.933014 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:18.932985    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:32:18.933113 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:18.933055    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:32:18.933113 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:18.933087    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:32:19.970716 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:19.970677    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-60efdf-openshift-default-5dfc484849-k6445" event={"ID":"c6fb7442-714a-4e4a-a6cf-19326b50a929","Type":"ContainerStarted","Data":"f7d8105e38072ca5fdd84f547f0b0fed18c8772f8ce6f4e0cc52c9f7cb80e806"}
Nov 03 20:32:20.230299 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:20.230210    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-60efdf-openshift-default-5dfc484849-k6445"
Nov 03 20:32:20.231310 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:20.231287    5570 patch_prober.go:29] interesting pod/e2e-inf-a-60efdf-openshift-default-5dfc484849-k6445 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body=
Nov 03 20:32:20.231383 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:20.231355    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-60efdf-openshift-default-5dfc484849-k6445" podUID="c6fb7442-714a-4e4a-a6cf-19326b50a929" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused"
Nov 03 20:32:20.571217 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:20.570084    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-1d159a-openshift-default-799c8f9695-b8h67"]
Nov 03 20:32:20.571217 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:20.571155    5570 cpu_manager.go:414] "containerMap: removing container" podUID="1b83e8f6-3b31-427c-8c6c-57e57d1bc4d4" containerName="payload-processing"
Nov 03 20:32:20.571217 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:20.571179    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b83e8f6-3b31-427c-8c6c-57e57d1bc4d4" containerName="payload-processing"
Nov 03 20:32:20.571217 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:20.571207    5570 cpu_manager.go:414] "containerMap: removing container" podUID="e5d67e4b-c834-426f-8df5-8d6dc4ae77ab" containerName="istio-proxy"
Nov 03 20:32:20.571217 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:20.571216    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5d67e4b-c834-426f-8df5-8d6dc4ae77ab" containerName="istio-proxy"
Nov 03 20:32:20.571618 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:20.571244    5570 cpu_manager.go:414] "containerMap: removing container" podUID="ef03b5c7-3817-4d77-b2c8-e450267276cc" containerName="payload-pre-processing"
Nov 03 20:32:20.571618 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:20.571253    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef03b5c7-3817-4d77-b2c8-e450267276cc" containerName="payload-pre-processing"
Nov 03 20:32:20.571618 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:20.571268    5570 cpu_manager.go:414] "containerMap: removing container" podUID="1da1c326-d687-48d8-aa52-42392ef0e1d2" containerName="payload-processing"
Nov 03 20:32:20.571618 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:20.571276    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1da1c326-d687-48d8-aa52-42392ef0e1d2" containerName="payload-processing"
Nov 03 20:32:20.571618 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:20.571301    5570 cpu_manager.go:414] "containerMap: removing container" podUID="5d450803-ade0-428d-9029-d71459b2fc22" containerName="maas-api"
Nov 03 20:32:20.571618 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:20.571308    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d450803-ade0-428d-9029-d71459b2fc22" containerName="maas-api"
Nov 03 20:32:20.571618 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:20.571319    5570 cpu_manager.go:414] "containerMap: removing container" podUID="ec1dc0b9-d778-48b7-986a-0d460bb6a52f" containerName="payload-processing"
Nov 03 20:32:20.571618 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:20.571327    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec1dc0b9-d778-48b7-986a-0d460bb6a52f" containerName="payload-processing"
Nov 03 20:32:20.571618 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:20.571340    5570 cpu_manager.go:414] "containerMap: removing container" podUID="7b57a22a-cf28-4099-8883-c04c61807dfa" containerName="istio-proxy"
Nov 03 20:32:20.571618 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:20.571346    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b57a22a-cf28-4099-8883-c04c61807dfa" containerName="istio-proxy"
Nov 03 20:32:20.571618 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:20.571365    5570 cpu_manager.go:414] "containerMap: removing container" podUID="18e420ed-849e-4e73-b1b0-ef8167ddd383" containerName="maas-api"
Nov 03 20:32:20.571618 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:20.571372    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18e420ed-849e-4e73-b1b0-ef8167ddd383" containerName="maas-api"
Nov 03 20:32:20.571618 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:20.571399    5570 cpu_manager.go:414] "containerMap: removing container" podUID="51d6f999-63b2-4461-a82f-65ff671546f3" containerName="payload-pre-processing"
Nov 03 20:32:20.571618 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:20.571406    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51d6f999-63b2-4461-a82f-65ff671546f3" containerName="payload-pre-processing"
Nov 03 20:32:20.571618 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:20.571552    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="51d6f999-63b2-4461-a82f-65ff671546f3" containerName="payload-pre-processing"
Nov 03 20:32:20.571618 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:20.571570    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b83e8f6-3b31-427c-8c6c-57e57d1bc4d4" containerName="payload-processing"
Nov 03 20:32:20.571618 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:20.571585    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5d67e4b-c834-426f-8df5-8d6dc4ae77ab" containerName="istio-proxy"
Nov 03 20:32:20.571618 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:20.571595    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef03b5c7-3817-4d77-b2c8-e450267276cc" containerName="payload-pre-processing"
Nov 03 20:32:20.571618 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:20.571604    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="1da1c326-d687-48d8-aa52-42392ef0e1d2" containerName="payload-processing"
Nov 03 20:32:20.571618 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:20.571615    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d450803-ade0-428d-9029-d71459b2fc22" containerName="maas-api"
Nov 03 20:32:20.571618 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:20.571625    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec1dc0b9-d778-48b7-986a-0d460bb6a52f" containerName="payload-processing"
Nov 03 20:32:20.572752 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:20.571636    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="18e420ed-849e-4e73-b1b0-ef8167ddd383" containerName="maas-api"
Nov 03 20:32:20.576882 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:20.576837    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-1d159a-openshift-default-799c8f9695-b8h67"
Nov 03 20:32:20.599530 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:20.599482    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-1d159a-openshift-default-799c8f9695-b8h67"]
Nov 03 20:32:20.607577 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:20.607547    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-1d159a-openshift-default-dockercfg-db82z\""
Nov 03 20:32:20.608066 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:20.608040    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40e4fc9c-c3c1-4aa0-98ad-eda347be89c5-istio-podinfo\") pod \"e2e-inf-b-1d159a-openshift-default-799c8f9695-b8h67\" (UID: \"40e4fc9c-c3c1-4aa0-98ad-eda347be89c5\") " pod="openshift-ingress/e2e-inf-b-1d159a-openshift-default-799c8f9695-b8h67"
Nov 03 20:32:20.608186 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:20.608098    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40e4fc9c-c3c1-4aa0-98ad-eda347be89c5-istiod-ca-cert\") pod \"e2e-inf-b-1d159a-openshift-default-799c8f9695-b8h67\" (UID: \"40e4fc9c-c3c1-4aa0-98ad-eda347be89c5\") " pod="openshift-ingress/e2e-inf-b-1d159a-openshift-default-799c8f9695-b8h67"
Nov 03 20:32:20.608186 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:20.608174    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40e4fc9c-c3c1-4aa0-98ad-eda347be89c5-workload-socket\") pod \"e2e-inf-b-1d159a-openshift-default-799c8f9695-b8h67\" (UID: \"40e4fc9c-c3c1-4aa0-98ad-eda347be89c5\") " pod="openshift-ingress/e2e-inf-b-1d159a-openshift-default-799c8f9695-b8h67"
Nov 03 20:32:20.608254 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:20.608225    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40e4fc9c-c3c1-4aa0-98ad-eda347be89c5-istio-data\") pod \"e2e-inf-b-1d159a-openshift-default-799c8f9695-b8h67\" (UID: \"40e4fc9c-c3c1-4aa0-98ad-eda347be89c5\") " pod="openshift-ingress/e2e-inf-b-1d159a-openshift-default-799c8f9695-b8h67"
Nov 03 20:32:20.608289 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:20.608247    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6jc5\" (UniqueName: \"kubernetes.io/projected/40e4fc9c-c3c1-4aa0-98ad-eda347be89c5-kube-api-access-k6jc5\") pod \"e2e-inf-b-1d159a-openshift-default-799c8f9695-b8h67\" (UID: \"40e4fc9c-c3c1-4aa0-98ad-eda347be89c5\") " pod="openshift-ingress/e2e-inf-b-1d159a-openshift-default-799c8f9695-b8h67"
Nov 03 20:32:20.608368 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:20.608355    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40e4fc9c-c3c1-4aa0-98ad-eda347be89c5-istio-envoy\") pod \"e2e-inf-b-1d159a-openshift-default-799c8f9695-b8h67\" (UID: \"40e4fc9c-c3c1-4aa0-98ad-eda347be89c5\") " pod="openshift-ingress/e2e-inf-b-1d159a-openshift-default-799c8f9695-b8h67"
Nov 03 20:32:20.608428 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:20.608417    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40e4fc9c-c3c1-4aa0-98ad-eda347be89c5-credential-socket\") pod \"e2e-inf-b-1d159a-openshift-default-799c8f9695-b8h67\" (UID: \"40e4fc9c-c3c1-4aa0-98ad-eda347be89c5\") " pod="openshift-ingress/e2e-inf-b-1d159a-openshift-default-799c8f9695-b8h67"
Nov 03 20:32:20.608476 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:20.608466    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40e4fc9c-c3c1-4aa0-98ad-eda347be89c5-workload-certs\") pod \"e2e-inf-b-1d159a-openshift-default-799c8f9695-b8h67\" (UID: \"40e4fc9c-c3c1-4aa0-98ad-eda347be89c5\") " pod="openshift-ingress/e2e-inf-b-1d159a-openshift-default-799c8f9695-b8h67"
Nov 03 20:32:20.608523 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:20.608513    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40e4fc9c-c3c1-4aa0-98ad-eda347be89c5-istio-token\") pod \"e2e-inf-b-1d159a-openshift-default-799c8f9695-b8h67\" (UID: \"40e4fc9c-c3c1-4aa0-98ad-eda347be89c5\") " pod="openshift-ingress/e2e-inf-b-1d159a-openshift-default-799c8f9695-b8h67"
Nov 03 20:32:20.710718 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:20.710675    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40e4fc9c-c3c1-4aa0-98ad-eda347be89c5-istiod-ca-cert\") pod \"e2e-inf-b-1d159a-openshift-default-799c8f9695-b8h67\" (UID: \"40e4fc9c-c3c1-4aa0-98ad-eda347be89c5\") " pod="openshift-ingress/e2e-inf-b-1d159a-openshift-default-799c8f9695-b8h67"
Nov 03 20:32:20.711089 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:20.711062    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40e4fc9c-c3c1-4aa0-98ad-eda347be89c5-workload-socket\") pod \"e2e-inf-b-1d159a-openshift-default-799c8f9695-b8h67\" (UID: \"40e4fc9c-c3c1-4aa0-98ad-eda347be89c5\") " pod="openshift-ingress/e2e-inf-b-1d159a-openshift-default-799c8f9695-b8h67"
Nov 03 20:32:20.711429 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:20.711399    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40e4fc9c-c3c1-4aa0-98ad-eda347be89c5-istio-data\") pod \"e2e-inf-b-1d159a-openshift-default-799c8f9695-b8h67\" (UID: \"40e4fc9c-c3c1-4aa0-98ad-eda347be89c5\") " pod="openshift-ingress/e2e-inf-b-1d159a-openshift-default-799c8f9695-b8h67"
Nov 03 20:32:20.711569 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:20.711550    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6jc5\" (UniqueName: \"kubernetes.io/projected/40e4fc9c-c3c1-4aa0-98ad-eda347be89c5-kube-api-access-k6jc5\") pod \"e2e-inf-b-1d159a-openshift-default-799c8f9695-b8h67\" (UID: \"40e4fc9c-c3c1-4aa0-98ad-eda347be89c5\") " pod="openshift-ingress/e2e-inf-b-1d159a-openshift-default-799c8f9695-b8h67"
Nov 03 20:32:20.711835 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:20.711802    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40e4fc9c-c3c1-4aa0-98ad-eda347be89c5-istio-envoy\") pod \"e2e-inf-b-1d159a-openshift-default-799c8f9695-b8h67\" (UID: \"40e4fc9c-c3c1-4aa0-98ad-eda347be89c5\") " pod="openshift-ingress/e2e-inf-b-1d159a-openshift-default-799c8f9695-b8h67"
Nov 03 20:32:20.712128 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:20.712109    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40e4fc9c-c3c1-4aa0-98ad-eda347be89c5-credential-socket\") pod \"e2e-inf-b-1d159a-openshift-default-799c8f9695-b8h67\" (UID: \"40e4fc9c-c3c1-4aa0-98ad-eda347be89c5\") " pod="openshift-ingress/e2e-inf-b-1d159a-openshift-default-799c8f9695-b8h67"
Nov 03 20:32:20.712977 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:20.712952    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40e4fc9c-c3c1-4aa0-98ad-eda347be89c5-workload-certs\") pod \"e2e-inf-b-1d159a-openshift-default-799c8f9695-b8h67\" (UID: \"40e4fc9c-c3c1-4aa0-98ad-eda347be89c5\") " pod="openshift-ingress/e2e-inf-b-1d159a-openshift-default-799c8f9695-b8h67"
Nov 03 20:32:20.713372 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:20.713338    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40e4fc9c-c3c1-4aa0-98ad-eda347be89c5-istio-token\") pod \"e2e-inf-b-1d159a-openshift-default-799c8f9695-b8h67\" (UID: \"40e4fc9c-c3c1-4aa0-98ad-eda347be89c5\") " pod="openshift-ingress/e2e-inf-b-1d159a-openshift-default-799c8f9695-b8h67"
Nov 03 20:32:20.713671 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:20.713650    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40e4fc9c-c3c1-4aa0-98ad-eda347be89c5-istio-podinfo\") pod \"e2e-inf-b-1d159a-openshift-default-799c8f9695-b8h67\" (UID: \"40e4fc9c-c3c1-4aa0-98ad-eda347be89c5\") " pod="openshift-ingress/e2e-inf-b-1d159a-openshift-default-799c8f9695-b8h67"
Nov 03 20:32:20.716045 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:20.712658    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40e4fc9c-c3c1-4aa0-98ad-eda347be89c5-istio-data\") pod \"e2e-inf-b-1d159a-openshift-default-799c8f9695-b8h67\" (UID: \"40e4fc9c-c3c1-4aa0-98ad-eda347be89c5\") " pod="openshift-ingress/e2e-inf-b-1d159a-openshift-default-799c8f9695-b8h67"
Nov 03 20:32:20.716610 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:20.716588    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40e4fc9c-c3c1-4aa0-98ad-eda347be89c5-credential-socket\") pod \"e2e-inf-b-1d159a-openshift-default-799c8f9695-b8h67\" (UID: \"40e4fc9c-c3c1-4aa0-98ad-eda347be89c5\") " pod="openshift-ingress/e2e-inf-b-1d159a-openshift-default-799c8f9695-b8h67"
Nov 03 20:32:20.754816 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:20.754722    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40e4fc9c-c3c1-4aa0-98ad-eda347be89c5-workload-certs\") pod \"e2e-inf-b-1d159a-openshift-default-799c8f9695-b8h67\" (UID: \"40e4fc9c-c3c1-4aa0-98ad-eda347be89c5\") " pod="openshift-ingress/e2e-inf-b-1d159a-openshift-default-799c8f9695-b8h67"
Nov 03 20:32:20.764466 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:20.764428    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40e4fc9c-c3c1-4aa0-98ad-eda347be89c5-workload-socket\") pod \"e2e-inf-b-1d159a-openshift-default-799c8f9695-b8h67\" (UID: \"40e4fc9c-c3c1-4aa0-98ad-eda347be89c5\") " pod="openshift-ingress/e2e-inf-b-1d159a-openshift-default-799c8f9695-b8h67"
Nov 03 20:32:20.792466 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:20.720787    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40e4fc9c-c3c1-4aa0-98ad-eda347be89c5-istiod-ca-cert\") pod \"e2e-inf-b-1d159a-openshift-default-799c8f9695-b8h67\" (UID: \"40e4fc9c-c3c1-4aa0-98ad-eda347be89c5\") " pod="openshift-ingress/e2e-inf-b-1d159a-openshift-default-799c8f9695-b8h67"
Nov 03 20:32:20.817798 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:20.803704    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40e4fc9c-c3c1-4aa0-98ad-eda347be89c5-istio-podinfo\") pod \"e2e-inf-b-1d159a-openshift-default-799c8f9695-b8h67\" (UID: \"40e4fc9c-c3c1-4aa0-98ad-eda347be89c5\") " pod="openshift-ingress/e2e-inf-b-1d159a-openshift-default-799c8f9695-b8h67"
Nov 03 20:32:20.818213 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:20.816026    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6jc5\" (UniqueName: \"kubernetes.io/projected/40e4fc9c-c3c1-4aa0-98ad-eda347be89c5-kube-api-access-k6jc5\") pod \"e2e-inf-b-1d159a-openshift-default-799c8f9695-b8h67\" (UID: \"40e4fc9c-c3c1-4aa0-98ad-eda347be89c5\") " pod="openshift-ingress/e2e-inf-b-1d159a-openshift-default-799c8f9695-b8h67"
Nov 03 20:32:20.818375 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:20.817367    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40e4fc9c-c3c1-4aa0-98ad-eda347be89c5-istio-token\") pod \"e2e-inf-b-1d159a-openshift-default-799c8f9695-b8h67\" (UID: \"40e4fc9c-c3c1-4aa0-98ad-eda347be89c5\") " pod="openshift-ingress/e2e-inf-b-1d159a-openshift-default-799c8f9695-b8h67"
Nov 03 20:32:20.847855 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:20.833625    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40e4fc9c-c3c1-4aa0-98ad-eda347be89c5-istio-envoy\") pod \"e2e-inf-b-1d159a-openshift-default-799c8f9695-b8h67\" (UID: \"40e4fc9c-c3c1-4aa0-98ad-eda347be89c5\") " pod="openshift-ingress/e2e-inf-b-1d159a-openshift-default-799c8f9695-b8h67"
Nov 03 20:32:20.909816 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:20.908094    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-1d159a-openshift-default-799c8f9695-b8h67"
Nov 03 20:32:21.231060 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:21.230524    5570 patch_prober.go:29] interesting pod/e2e-inf-a-60efdf-openshift-default-5dfc484849-k6445 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body=
Nov 03 20:32:21.231060 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:21.230602    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-60efdf-openshift-default-5dfc484849-k6445" podUID="c6fb7442-714a-4e4a-a6cf-19326b50a929" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused"
Nov 03 20:32:21.467598 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:21.467546    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-1d159a-openshift-default-799c8f9695-b8h67"]
Nov 03 20:32:21.473581 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:32:21.473522    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40e4fc9c_c3c1_4aa0_98ad_eda347be89c5.slice/crio-3861708181af8794b164b11647d15caf660b93d8c3f7983ec586b721c19d01f0 WatchSource:0}: Error finding container 3861708181af8794b164b11647d15caf660b93d8c3f7983ec586b721c19d01f0: Status 404 returned error can't find the container with id 3861708181af8794b164b11647d15caf660b93d8c3f7983ec586b721c19d01f0
Nov 03 20:32:21.477049 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:21.477006    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:32:21.477133 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:21.477104    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:32:21.477187 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:21.477157    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:32:22.096533 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:22.095238    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-1d159a-openshift-default-799c8f9695-b8h67" event={"ID":"40e4fc9c-c3c1-4aa0-98ad-eda347be89c5","Type":"ContainerStarted","Data":"e91fa2efdc7c6fd5b80b7258d1bd1d371bd0e89b53a388019b215aa898711527"}
Nov 03 20:32:22.096533 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:22.096466    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-1d159a-openshift-default-799c8f9695-b8h67" event={"ID":"40e4fc9c-c3c1-4aa0-98ad-eda347be89c5","Type":"ContainerStarted","Data":"3861708181af8794b164b11647d15caf660b93d8c3f7983ec586b721c19d01f0"}
Nov 03 20:32:22.172071 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:22.172000    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-1d159a-openshift-default-799c8f9695-b8h67" podStartSLOduration=2.171974884 podStartE2EDuration="2.171974884s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:32:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:32:22.164187094 +0000 UTC m=+1676.316848981" watchObservedRunningTime="2026-11-03 20:32:22.171974884 +0000 UTC m=+1676.324636770"
Nov 03 20:32:22.230913 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:22.230874    5570 patch_prober.go:29] interesting pod/e2e-inf-a-60efdf-openshift-default-5dfc484849-k6445 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body=
Nov 03 20:32:22.231114 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:22.230940    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-60efdf-openshift-default-5dfc484849-k6445" podUID="c6fb7442-714a-4e4a-a6cf-19326b50a929" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused"
Nov 03 20:32:22.909442 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:22.909404    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-1d159a-openshift-default-799c8f9695-b8h67"
Nov 03 20:32:22.916176 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:22.916144    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-1d159a-openshift-default-799c8f9695-b8h67"
Nov 03 20:32:22.974166 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:22.974131    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-60efdf-openshift-default-5dfc484849-k6445"
Nov 03 20:32:23.103448 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:23.103412    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-1d159a-openshift-default-799c8f9695-b8h67"
Nov 03 20:32:23.104442 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:23.104416    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-1d159a-openshift-default-799c8f9695-b8h67"
Nov 03 20:32:23.230828 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:23.230793    5570 patch_prober.go:29] interesting pod/e2e-inf-a-60efdf-openshift-default-5dfc484849-k6445 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body=
Nov 03 20:32:23.231029 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:23.230850    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-60efdf-openshift-default-5dfc484849-k6445" podUID="c6fb7442-714a-4e4a-a6cf-19326b50a929" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused"
Nov 03 20:32:23.792980 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:23.792572    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-bx2dr"
Nov 03 20:32:23.930716 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:23.921337    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca63212a-77c0-4b97-8a23-5d0a518a55ef-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca63212a-77c0-4b97-8a23-5d0a518a55ef-workload-socket\") pod \"ca63212a-77c0-4b97-8a23-5d0a518a55ef\" (UID: \"ca63212a-77c0-4b97-8a23-5d0a518a55ef\") "
Nov 03 20:32:23.930716 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:23.921403    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca63212a-77c0-4b97-8a23-5d0a518a55ef-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca63212a-77c0-4b97-8a23-5d0a518a55ef-istio-envoy\") pod \"ca63212a-77c0-4b97-8a23-5d0a518a55ef\" (UID: \"ca63212a-77c0-4b97-8a23-5d0a518a55ef\") "
Nov 03 20:32:23.930716 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:23.921510    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca63212a-77c0-4b97-8a23-5d0a518a55ef-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca63212a-77c0-4b97-8a23-5d0a518a55ef-workload-certs\") pod \"ca63212a-77c0-4b97-8a23-5d0a518a55ef\" (UID: \"ca63212a-77c0-4b97-8a23-5d0a518a55ef\") "
Nov 03 20:32:23.930716 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:23.921560    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca63212a-77c0-4b97-8a23-5d0a518a55ef-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca63212a-77c0-4b97-8a23-5d0a518a55ef-credential-socket\") pod \"ca63212a-77c0-4b97-8a23-5d0a518a55ef\" (UID: \"ca63212a-77c0-4b97-8a23-5d0a518a55ef\") "
Nov 03 20:32:23.930716 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:23.921653    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca63212a-77c0-4b97-8a23-5d0a518a55ef-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca63212a-77c0-4b97-8a23-5d0a518a55ef-istio-data\") pod \"ca63212a-77c0-4b97-8a23-5d0a518a55ef\" (UID: \"ca63212a-77c0-4b97-8a23-5d0a518a55ef\") "
Nov 03 20:32:23.930716 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:23.921721    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca63212a-77c0-4b97-8a23-5d0a518a55ef-kube-api-access-9w6sx\" (UniqueName: \"kubernetes.io/projected/ca63212a-77c0-4b97-8a23-5d0a518a55ef-kube-api-access-9w6sx\") pod \"ca63212a-77c0-4b97-8a23-5d0a518a55ef\" (UID: \"ca63212a-77c0-4b97-8a23-5d0a518a55ef\") "
Nov 03 20:32:23.930716 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:23.921831    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca63212a-77c0-4b97-8a23-5d0a518a55ef-istio-token\" (UniqueName: \"kubernetes.io/projected/ca63212a-77c0-4b97-8a23-5d0a518a55ef-istio-token\") pod \"ca63212a-77c0-4b97-8a23-5d0a518a55ef\" (UID: \"ca63212a-77c0-4b97-8a23-5d0a518a55ef\") "
Nov 03 20:32:23.930716 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:23.921894    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca63212a-77c0-4b97-8a23-5d0a518a55ef-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca63212a-77c0-4b97-8a23-5d0a518a55ef-istiod-ca-cert\") pod \"ca63212a-77c0-4b97-8a23-5d0a518a55ef\" (UID: \"ca63212a-77c0-4b97-8a23-5d0a518a55ef\") "
Nov 03 20:32:23.930716 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:23.921950    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ca63212a-77c0-4b97-8a23-5d0a518a55ef-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca63212a-77c0-4b97-8a23-5d0a518a55ef-istio-podinfo\") pod \"ca63212a-77c0-4b97-8a23-5d0a518a55ef\" (UID: \"ca63212a-77c0-4b97-8a23-5d0a518a55ef\") "
Nov 03 20:32:23.930716 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:23.927723    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-9pg4l"
Nov 03 20:32:23.953797 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:23.941917    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca63212a-77c0-4b97-8a23-5d0a518a55ef-credential-socket" pod "ca63212a-77c0-4b97-8a23-5d0a518a55ef" (UID: "ca63212a-77c0-4b97-8a23-5d0a518a55ef"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:32:23.953797 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:23.949900    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca63212a-77c0-4b97-8a23-5d0a518a55ef-workload-socket" pod "ca63212a-77c0-4b97-8a23-5d0a518a55ef" (UID: "ca63212a-77c0-4b97-8a23-5d0a518a55ef"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:32:23.985797 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:23.985118    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca63212a-77c0-4b97-8a23-5d0a518a55ef-istiod-ca-cert" pod "ca63212a-77c0-4b97-8a23-5d0a518a55ef" (UID: "ca63212a-77c0-4b97-8a23-5d0a518a55ef"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:32:23.985797 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:23.985431    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca63212a-77c0-4b97-8a23-5d0a518a55ef-workload-certs" pod "ca63212a-77c0-4b97-8a23-5d0a518a55ef" (UID: "ca63212a-77c0-4b97-8a23-5d0a518a55ef"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:32:23.985797 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:23.985434    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca63212a-77c0-4b97-8a23-5d0a518a55ef-istio-data" pod "ca63212a-77c0-4b97-8a23-5d0a518a55ef" (UID: "ca63212a-77c0-4b97-8a23-5d0a518a55ef"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:32:24.000253 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:24.000182    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca63212a-77c0-4b97-8a23-5d0a518a55ef-kube-api-access-9w6sx" pod "ca63212a-77c0-4b97-8a23-5d0a518a55ef" (UID: "ca63212a-77c0-4b97-8a23-5d0a518a55ef"). InnerVolumeSpecName "kube-api-access-9w6sx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:32:24.000714 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:24.000650    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ca63212a-77c0-4b97-8a23-5d0a518a55ef-istio-podinfo" pod "ca63212a-77c0-4b97-8a23-5d0a518a55ef" (UID: "ca63212a-77c0-4b97-8a23-5d0a518a55ef"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:32:24.003246 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:24.003212    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca63212a-77c0-4b97-8a23-5d0a518a55ef-istio-token" pod "ca63212a-77c0-4b97-8a23-5d0a518a55ef" (UID: "ca63212a-77c0-4b97-8a23-5d0a518a55ef"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:32:24.010270 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:24.010230    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca63212a-77c0-4b97-8a23-5d0a518a55ef-istio-envoy" pod "ca63212a-77c0-4b97-8a23-5d0a518a55ef" (UID: "ca63212a-77c0-4b97-8a23-5d0a518a55ef"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:32:24.023570 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:24.023534    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/623f50bf-e25b-4647-9ffe-db4d163b2284-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/623f50bf-e25b-4647-9ffe-db4d163b2284-istio-podinfo\") pod \"623f50bf-e25b-4647-9ffe-db4d163b2284\" (UID: \"623f50bf-e25b-4647-9ffe-db4d163b2284\") "
Nov 03 20:32:24.023735 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:24.023612    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/623f50bf-e25b-4647-9ffe-db4d163b2284-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/623f50bf-e25b-4647-9ffe-db4d163b2284-istio-data\") pod \"623f50bf-e25b-4647-9ffe-db4d163b2284\" (UID: \"623f50bf-e25b-4647-9ffe-db4d163b2284\") "
Nov 03 20:32:24.023735 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:24.023673    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/623f50bf-e25b-4647-9ffe-db4d163b2284-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/623f50bf-e25b-4647-9ffe-db4d163b2284-istiod-ca-cert\") pod \"623f50bf-e25b-4647-9ffe-db4d163b2284\" (UID: \"623f50bf-e25b-4647-9ffe-db4d163b2284\") "
Nov 03 20:32:24.023735 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:24.023718    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/623f50bf-e25b-4647-9ffe-db4d163b2284-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/623f50bf-e25b-4647-9ffe-db4d163b2284-workload-socket\") pod \"623f50bf-e25b-4647-9ffe-db4d163b2284\" (UID: \"623f50bf-e25b-4647-9ffe-db4d163b2284\") "
Nov 03 20:32:24.023867 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:24.023756    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/623f50bf-e25b-4647-9ffe-db4d163b2284-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/623f50bf-e25b-4647-9ffe-db4d163b2284-workload-certs\") pod \"623f50bf-e25b-4647-9ffe-db4d163b2284\" (UID: \"623f50bf-e25b-4647-9ffe-db4d163b2284\") "
Nov 03 20:32:24.023946 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:24.023930    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/623f50bf-e25b-4647-9ffe-db4d163b2284-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/623f50bf-e25b-4647-9ffe-db4d163b2284-istio-envoy\") pod \"623f50bf-e25b-4647-9ffe-db4d163b2284\" (UID: \"623f50bf-e25b-4647-9ffe-db4d163b2284\") "
Nov 03 20:32:24.024018 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:24.024003    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/623f50bf-e25b-4647-9ffe-db4d163b2284-istio-token\" (UniqueName: \"kubernetes.io/projected/623f50bf-e25b-4647-9ffe-db4d163b2284-istio-token\") pod \"623f50bf-e25b-4647-9ffe-db4d163b2284\" (UID: \"623f50bf-e25b-4647-9ffe-db4d163b2284\") "
Nov 03 20:32:24.024073 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:24.024056    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/623f50bf-e25b-4647-9ffe-db4d163b2284-kube-api-access-fv28w\" (UniqueName: \"kubernetes.io/projected/623f50bf-e25b-4647-9ffe-db4d163b2284-kube-api-access-fv28w\") pod \"623f50bf-e25b-4647-9ffe-db4d163b2284\" (UID: \"623f50bf-e25b-4647-9ffe-db4d163b2284\") "
Nov 03 20:32:24.024163 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:24.024148    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/623f50bf-e25b-4647-9ffe-db4d163b2284-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/623f50bf-e25b-4647-9ffe-db4d163b2284-credential-socket\") pod \"623f50bf-e25b-4647-9ffe-db4d163b2284\" (UID: \"623f50bf-e25b-4647-9ffe-db4d163b2284\") "
Nov 03 20:32:24.025133 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:24.025105    5570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca63212a-77c0-4b97-8a23-5d0a518a55ef-istio-token\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:32:24.025133 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:24.025135    5570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca63212a-77c0-4b97-8a23-5d0a518a55ef-istiod-ca-cert\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:32:24.025337 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:24.025150    5570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca63212a-77c0-4b97-8a23-5d0a518a55ef-istio-podinfo\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:32:24.025337 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:24.025164    5570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca63212a-77c0-4b97-8a23-5d0a518a55ef-workload-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:32:24.025337 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:24.025178    5570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca63212a-77c0-4b97-8a23-5d0a518a55ef-istio-envoy\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:32:24.025337 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:24.025191    5570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca63212a-77c0-4b97-8a23-5d0a518a55ef-workload-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:32:24.025337 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:24.025204    5570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca63212a-77c0-4b97-8a23-5d0a518a55ef-credential-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:32:24.025337 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:24.025216    5570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca63212a-77c0-4b97-8a23-5d0a518a55ef-istio-data\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:32:24.025337 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:24.025231    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9w6sx\" (UniqueName: \"kubernetes.io/projected/ca63212a-77c0-4b97-8a23-5d0a518a55ef-kube-api-access-9w6sx\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:32:24.025670 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:24.025502    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/623f50bf-e25b-4647-9ffe-db4d163b2284-credential-socket" pod "623f50bf-e25b-4647-9ffe-db4d163b2284" (UID: "623f50bf-e25b-4647-9ffe-db4d163b2284"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:32:24.025768 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:24.025749    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/623f50bf-e25b-4647-9ffe-db4d163b2284-istio-data" pod "623f50bf-e25b-4647-9ffe-db4d163b2284" (UID: "623f50bf-e25b-4647-9ffe-db4d163b2284"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:32:24.026067 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:24.026043    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/623f50bf-e25b-4647-9ffe-db4d163b2284-istio-podinfo" pod "623f50bf-e25b-4647-9ffe-db4d163b2284" (UID: "623f50bf-e25b-4647-9ffe-db4d163b2284"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:32:24.026128 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:24.026102    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/623f50bf-e25b-4647-9ffe-db4d163b2284-istiod-ca-cert" pod "623f50bf-e25b-4647-9ffe-db4d163b2284" (UID: "623f50bf-e25b-4647-9ffe-db4d163b2284"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:32:24.026919 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:24.026661    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/623f50bf-e25b-4647-9ffe-db4d163b2284-workload-certs" pod "623f50bf-e25b-4647-9ffe-db4d163b2284" (UID: "623f50bf-e25b-4647-9ffe-db4d163b2284"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:32:24.027297 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:24.027279    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/623f50bf-e25b-4647-9ffe-db4d163b2284-workload-socket" pod "623f50bf-e25b-4647-9ffe-db4d163b2284" (UID: "623f50bf-e25b-4647-9ffe-db4d163b2284"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:32:24.028894 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:24.028852    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/623f50bf-e25b-4647-9ffe-db4d163b2284-istio-envoy" pod "623f50bf-e25b-4647-9ffe-db4d163b2284" (UID: "623f50bf-e25b-4647-9ffe-db4d163b2284"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:32:24.030209 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:24.030188    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/623f50bf-e25b-4647-9ffe-db4d163b2284-istio-token" pod "623f50bf-e25b-4647-9ffe-db4d163b2284" (UID: "623f50bf-e25b-4647-9ffe-db4d163b2284"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:32:24.031107 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:24.031073    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/623f50bf-e25b-4647-9ffe-db4d163b2284-kube-api-access-fv28w" pod "623f50bf-e25b-4647-9ffe-db4d163b2284" (UID: "623f50bf-e25b-4647-9ffe-db4d163b2284"). InnerVolumeSpecName "kube-api-access-fv28w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:32:24.117232 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:24.117140    5570 generic.go:356] "Generic (PLEG): container finished" podID="ca63212a-77c0-4b97-8a23-5d0a518a55ef" containerID="2f27b981b1b00f2a6830b65b62ed19e37456b99a6f874e45cef4ab2c310ef747" exitCode=0
Nov 03 20:32:24.117463 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:24.117444    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-bx2dr" event={"ID":"ca63212a-77c0-4b97-8a23-5d0a518a55ef","Type":"ContainerDied","Data":"2f27b981b1b00f2a6830b65b62ed19e37456b99a6f874e45cef4ab2c310ef747"}
Nov 03 20:32:24.117586 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:24.117491    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-bx2dr" event={"ID":"ca63212a-77c0-4b97-8a23-5d0a518a55ef","Type":"ContainerDied","Data":"89018a843dc01fafbf9305a4f86a25539c93c57ae4b668a011c9899136f7671c"}
Nov 03 20:32:24.117586 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:24.117529    5570 scope.go:122] "RemoveContainer" containerID="2f27b981b1b00f2a6830b65b62ed19e37456b99a6f874e45cef4ab2c310ef747"
Nov 03 20:32:24.117705 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:24.117526    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-bx2dr"
Nov 03 20:32:24.119980 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:24.119954    5570 generic.go:356] "Generic (PLEG): container finished" podID="623f50bf-e25b-4647-9ffe-db4d163b2284" containerID="2a450d72048e8485b746d6c3a806cdc70b988882550514fbc8566ee1f05860a2" exitCode=0
Nov 03 20:32:24.120115 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:24.120028    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-9pg4l" event={"ID":"623f50bf-e25b-4647-9ffe-db4d163b2284","Type":"ContainerDied","Data":"2a450d72048e8485b746d6c3a806cdc70b988882550514fbc8566ee1f05860a2"}
Nov 03 20:32:24.120115 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:24.120044    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-9pg4l"
Nov 03 20:32:24.120115 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:24.120084    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-9pg4l" event={"ID":"623f50bf-e25b-4647-9ffe-db4d163b2284","Type":"ContainerDied","Data":"262b79c83c2bf9bcaac123740b14d31ea69c65209743767d859ca15cbcbc0f80"}
Nov 03 20:32:24.127537 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:24.127514    5570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/623f50bf-e25b-4647-9ffe-db4d163b2284-istio-envoy\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:32:24.127721 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:24.127709    5570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/623f50bf-e25b-4647-9ffe-db4d163b2284-istio-token\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:32:24.127804 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:24.127724    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fv28w\" (UniqueName: \"kubernetes.io/projected/623f50bf-e25b-4647-9ffe-db4d163b2284-kube-api-access-fv28w\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:32:24.127804 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:24.127733    5570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/623f50bf-e25b-4647-9ffe-db4d163b2284-credential-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:32:24.127804 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:24.127743    5570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/623f50bf-e25b-4647-9ffe-db4d163b2284-istio-podinfo\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:32:24.127804 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:24.127752    5570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/623f50bf-e25b-4647-9ffe-db4d163b2284-istio-data\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:32:24.127804 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:24.127760    5570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/623f50bf-e25b-4647-9ffe-db4d163b2284-istiod-ca-cert\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:32:24.127804 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:24.127767    5570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/623f50bf-e25b-4647-9ffe-db4d163b2284-workload-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:32:24.127804 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:24.127793    5570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/623f50bf-e25b-4647-9ffe-db4d163b2284-workload-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:32:24.134877 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:24.134576    5570 scope.go:122] "RemoveContainer" containerID="a4842be231f4b13a0568dacd707e44fd989a4a1f6d171db686a6aac8a246eb0f"
Nov 03 20:32:24.174518 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:24.174491    5570 scope.go:122] "RemoveContainer" containerID="2f27b981b1b00f2a6830b65b62ed19e37456b99a6f874e45cef4ab2c310ef747"
Nov 03 20:32:24.175060 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:32:24.174922    5570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f27b981b1b00f2a6830b65b62ed19e37456b99a6f874e45cef4ab2c310ef747\": container with ID starting with 2f27b981b1b00f2a6830b65b62ed19e37456b99a6f874e45cef4ab2c310ef747 not found: ID does not exist" containerID="2f27b981b1b00f2a6830b65b62ed19e37456b99a6f874e45cef4ab2c310ef747"
Nov 03 20:32:24.175060 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:24.174956    5570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f27b981b1b00f2a6830b65b62ed19e37456b99a6f874e45cef4ab2c310ef747"} err="failed to get container status \"2f27b981b1b00f2a6830b65b62ed19e37456b99a6f874e45cef4ab2c310ef747\": rpc error: code = NotFound desc = could not find container \"2f27b981b1b00f2a6830b65b62ed19e37456b99a6f874e45cef4ab2c310ef747\": container with ID starting with 2f27b981b1b00f2a6830b65b62ed19e37456b99a6f874e45cef4ab2c310ef747 not found: ID does not exist"
Nov 03 20:32:24.175060 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:24.174980    5570 scope.go:122] "RemoveContainer" containerID="a4842be231f4b13a0568dacd707e44fd989a4a1f6d171db686a6aac8a246eb0f"
Nov 03 20:32:24.175350 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:32:24.175332    5570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4842be231f4b13a0568dacd707e44fd989a4a1f6d171db686a6aac8a246eb0f\": container with ID starting with a4842be231f4b13a0568dacd707e44fd989a4a1f6d171db686a6aac8a246eb0f not found: ID does not exist" containerID="a4842be231f4b13a0568dacd707e44fd989a4a1f6d171db686a6aac8a246eb0f"
Nov 03 20:32:24.175403 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:24.175357    5570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4842be231f4b13a0568dacd707e44fd989a4a1f6d171db686a6aac8a246eb0f"} err="failed to get container status \"a4842be231f4b13a0568dacd707e44fd989a4a1f6d171db686a6aac8a246eb0f\": rpc error: code = NotFound desc = could not find container \"a4842be231f4b13a0568dacd707e44fd989a4a1f6d171db686a6aac8a246eb0f\": container with ID starting with a4842be231f4b13a0568dacd707e44fd989a4a1f6d171db686a6aac8a246eb0f not found: ID does not exist"
Nov 03 20:32:24.175403 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:24.175372    5570 scope.go:122] "RemoveContainer" containerID="2a450d72048e8485b746d6c3a806cdc70b988882550514fbc8566ee1f05860a2"
Nov 03 20:32:24.186282 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:24.186250    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-bx2dr"]
Nov 03 20:32:24.188979 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:24.188960    5570 scope.go:122] "RemoveContainer" containerID="2a450d72048e8485b746d6c3a806cdc70b988882550514fbc8566ee1f05860a2"
Nov 03 20:32:24.189292 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:32:24.189272    5570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a450d72048e8485b746d6c3a806cdc70b988882550514fbc8566ee1f05860a2\": container with ID starting with 2a450d72048e8485b746d6c3a806cdc70b988882550514fbc8566ee1f05860a2 not found: ID does not exist" containerID="2a450d72048e8485b746d6c3a806cdc70b988882550514fbc8566ee1f05860a2"
Nov 03 20:32:24.189358 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:24.189301    5570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a450d72048e8485b746d6c3a806cdc70b988882550514fbc8566ee1f05860a2"} err="failed to get container status \"2a450d72048e8485b746d6c3a806cdc70b988882550514fbc8566ee1f05860a2\": rpc error: code = NotFound desc = could not find container \"2a450d72048e8485b746d6c3a806cdc70b988882550514fbc8566ee1f05860a2\": container with ID starting with 2a450d72048e8485b746d6c3a806cdc70b988882550514fbc8566ee1f05860a2 not found: ID does not exist"
Nov 03 20:32:24.189663 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:24.189641    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-bx2dr"]
Nov 03 20:32:24.230141 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:24.230107    5570 patch_prober.go:29] interesting pod/e2e-inf-a-60efdf-openshift-default-5dfc484849-k6445 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body=
Nov 03 20:32:24.230331 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:24.230157    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-60efdf-openshift-default-5dfc484849-k6445" podUID="c6fb7442-714a-4e4a-a6cf-19326b50a929" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused"
Nov 03 20:32:24.266919 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:24.266885    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-9pg4l"]
Nov 03 20:32:24.270120 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:24.270094    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-c4a71d-openshift-default-7647f5cdbf-9pg4l"]
Nov 03 20:32:25.231016 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:25.230968    5570 patch_prober.go:29] interesting pod/e2e-inf-a-60efdf-openshift-default-5dfc484849-k6445 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body=
Nov 03 20:32:25.231415 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:25.231046    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-60efdf-openshift-default-5dfc484849-k6445" podUID="c6fb7442-714a-4e4a-a6cf-19326b50a929" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused"
Nov 03 20:32:25.994455 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:25.994415    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="623f50bf-e25b-4647-9ffe-db4d163b2284" path="/var/lib/kubelet/pods/623f50bf-e25b-4647-9ffe-db4d163b2284/volumes"
Nov 03 20:32:25.995149 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:25.995129    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca63212a-77c0-4b97-8a23-5d0a518a55ef" path="/var/lib/kubelet/pods/ca63212a-77c0-4b97-8a23-5d0a518a55ef/volumes"
Nov 03 20:32:26.235216 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:26.235185    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-60efdf-openshift-default-5dfc484849-k6445"
Nov 03 20:32:26.236525 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:26.236501    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-60efdf-openshift-default-5dfc484849-k6445"
Nov 03 20:32:26.524045 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:26.524005    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-1d159a-5c484658dc-8qvxq"]
Nov 03 20:32:26.524691 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:26.524676    5570 cpu_manager.go:414] "containerMap: removing container" podUID="e5d67e4b-c834-426f-8df5-8d6dc4ae77ab" containerName="istio-proxy"
Nov 03 20:32:26.524737 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:26.524691    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5d67e4b-c834-426f-8df5-8d6dc4ae77ab" containerName="istio-proxy"
Nov 03 20:32:26.524737 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:26.524704    5570 cpu_manager.go:414] "containerMap: removing container" podUID="ca63212a-77c0-4b97-8a23-5d0a518a55ef" containerName="istio-proxy"
Nov 03 20:32:26.524737 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:26.524709    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca63212a-77c0-4b97-8a23-5d0a518a55ef" containerName="istio-proxy"
Nov 03 20:32:26.524892 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:26.524761    5570 cpu_manager.go:414] "containerMap: removing container" podUID="623f50bf-e25b-4647-9ffe-db4d163b2284" containerName="istio-proxy"
Nov 03 20:32:26.524892 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:26.524768    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="623f50bf-e25b-4647-9ffe-db4d163b2284" containerName="istio-proxy"
Nov 03 20:32:26.524892 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:26.524791    5570 cpu_manager.go:414] "containerMap: removing container" podUID="ca63212a-77c0-4b97-8a23-5d0a518a55ef" containerName="istio-proxy"
Nov 03 20:32:26.524892 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:26.524798    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca63212a-77c0-4b97-8a23-5d0a518a55ef" containerName="istio-proxy"
Nov 03 20:32:26.525014 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:26.524907    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="623f50bf-e25b-4647-9ffe-db4d163b2284" containerName="istio-proxy"
Nov 03 20:32:26.525014 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:26.524916    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca63212a-77c0-4b97-8a23-5d0a518a55ef" containerName="istio-proxy"
Nov 03 20:32:26.525014 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:26.524927    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5d67e4b-c834-426f-8df5-8d6dc4ae77ab" containerName="istio-proxy"
Nov 03 20:32:26.525014 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:26.524937    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca63212a-77c0-4b97-8a23-5d0a518a55ef" containerName="istio-proxy"
Nov 03 20:32:26.530187 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:26.530164    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1d159a-5c484658dc-8qvxq"
Nov 03 20:32:26.551123 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:26.551090    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-1d159a-dockercfg-t7pmh\""
Nov 03 20:32:26.552565 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:26.552533    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-1d159a-5c484658dc-8qvxq"]
Nov 03 20:32:26.554791 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:26.554750    5570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-1d159a\""
Nov 03 20:32:26.586123 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:26.586066    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ed630bf-cf60-4e37-9113-a1707b99937f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-1d159a-5c484658dc-8qvxq\" (UID: \"1ed630bf-cf60-4e37-9113-a1707b99937f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1d159a-5c484658dc-8qvxq"
Nov 03 20:32:26.586342 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:26.586287    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rp6wq\" (UniqueName: \"kubernetes.io/projected/1ed630bf-cf60-4e37-9113-a1707b99937f-kube-api-access-rp6wq\") pod \"payload-pre-processing-e2e-inf-b-1d159a-5c484658dc-8qvxq\" (UID: \"1ed630bf-cf60-4e37-9113-a1707b99937f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1d159a-5c484658dc-8qvxq"
Nov 03 20:32:26.746798 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:26.725834    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-1d159a-5497649584-zvjng"]
Nov 03 20:32:26.747793 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:26.747734    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rp6wq\" (UniqueName: \"kubernetes.io/projected/1ed630bf-cf60-4e37-9113-a1707b99937f-kube-api-access-rp6wq\") pod \"payload-pre-processing-e2e-inf-b-1d159a-5c484658dc-8qvxq\" (UID: \"1ed630bf-cf60-4e37-9113-a1707b99937f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1d159a-5c484658dc-8qvxq"
Nov 03 20:32:26.748308 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:26.748284    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ed630bf-cf60-4e37-9113-a1707b99937f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-1d159a-5c484658dc-8qvxq\" (UID: \"1ed630bf-cf60-4e37-9113-a1707b99937f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1d159a-5c484658dc-8qvxq"
Nov 03 20:32:26.764797 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:26.748353    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-1d159a-5497649584-zvjng"
Nov 03 20:32:26.765576 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:26.749477    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ed630bf-cf60-4e37-9113-a1707b99937f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-1d159a-5c484658dc-8qvxq\" (UID: \"1ed630bf-cf60-4e37-9113-a1707b99937f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1d159a-5c484658dc-8qvxq"
Nov 03 20:32:26.872982 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:26.872884    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-1d159a-5497649584-zvjng"]
Nov 03 20:32:26.892483 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:26.873978    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96055efa-780d-46b1-8d17-31587157b508-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-1d159a-5497649584-zvjng\" (UID: \"96055efa-780d-46b1-8d17-31587157b508\") " pod="openshift-ingress/payload-processing-e2e-inf-b-1d159a-5497649584-zvjng"
Nov 03 20:32:26.907737 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:26.907695    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qk6mv\" (UniqueName: \"kubernetes.io/projected/96055efa-780d-46b1-8d17-31587157b508-kube-api-access-qk6mv\") pod \"payload-processing-e2e-inf-b-1d159a-5497649584-zvjng\" (UID: \"96055efa-780d-46b1-8d17-31587157b508\") " pod="openshift-ingress/payload-processing-e2e-inf-b-1d159a-5497649584-zvjng"
Nov 03 20:32:26.909170 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:26.893469    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rp6wq\" (UniqueName: \"kubernetes.io/projected/1ed630bf-cf60-4e37-9113-a1707b99937f-kube-api-access-rp6wq\") pod \"payload-pre-processing-e2e-inf-b-1d159a-5c484658dc-8qvxq\" (UID: \"1ed630bf-cf60-4e37-9113-a1707b99937f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1d159a-5c484658dc-8qvxq"
Nov 03 20:32:27.010391 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:27.010354    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qk6mv\" (UniqueName: \"kubernetes.io/projected/96055efa-780d-46b1-8d17-31587157b508-kube-api-access-qk6mv\") pod \"payload-processing-e2e-inf-b-1d159a-5497649584-zvjng\" (UID: \"96055efa-780d-46b1-8d17-31587157b508\") " pod="openshift-ingress/payload-processing-e2e-inf-b-1d159a-5497649584-zvjng"
Nov 03 20:32:27.010555 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:27.010416    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96055efa-780d-46b1-8d17-31587157b508-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-1d159a-5497649584-zvjng\" (UID: \"96055efa-780d-46b1-8d17-31587157b508\") " pod="openshift-ingress/payload-processing-e2e-inf-b-1d159a-5497649584-zvjng"
Nov 03 20:32:27.011148 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:27.011128    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96055efa-780d-46b1-8d17-31587157b508-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-1d159a-5497649584-zvjng\" (UID: \"96055efa-780d-46b1-8d17-31587157b508\") " pod="openshift-ingress/payload-processing-e2e-inf-b-1d159a-5497649584-zvjng"
Nov 03 20:32:27.030943 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:27.030915    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qk6mv\" (UniqueName: \"kubernetes.io/projected/96055efa-780d-46b1-8d17-31587157b508-kube-api-access-qk6mv\") pod \"payload-processing-e2e-inf-b-1d159a-5497649584-zvjng\" (UID: \"96055efa-780d-46b1-8d17-31587157b508\") " pod="openshift-ingress/payload-processing-e2e-inf-b-1d159a-5497649584-zvjng"
Nov 03 20:32:27.145505 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:27.145414    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1d159a-5c484658dc-8qvxq"
Nov 03 20:32:27.227458 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:27.227348    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-1d159a-5497649584-zvjng"
Nov 03 20:32:27.252047 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:32:27.251979    5570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8bwhz" containerName="extractor"
Nov 03 20:32:27.450498 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:27.450468    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-1d159a-5c484658dc-8qvxq"]
Nov 03 20:32:27.451611 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:32:27.451584    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ed630bf_cf60_4e37_9113_a1707b99937f.slice/crio-91798eb4a6b2d791273fd5918f34a359436bcbbf04374c5c7fb7d5abfc34c717 WatchSource:0}: Error finding container 91798eb4a6b2d791273fd5918f34a359436bcbbf04374c5c7fb7d5abfc34c717: Status 404 returned error can't find the container with id 91798eb4a6b2d791273fd5918f34a359436bcbbf04374c5c7fb7d5abfc34c717
Nov 03 20:32:27.555213 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:32:27.555156    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96055efa_780d_46b1_8d17_31587157b508.slice/crio-1f4419d15f67647080e121042acf1150dfd460859516e223f90a74c2f4e8a6eb WatchSource:0}: Error finding container 1f4419d15f67647080e121042acf1150dfd460859516e223f90a74c2f4e8a6eb: Status 404 returned error can't find the container with id 1f4419d15f67647080e121042acf1150dfd460859516e223f90a74c2f4e8a6eb
Nov 03 20:32:27.556135 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:27.556113    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-1d159a-5497649584-zvjng"]
Nov 03 20:32:28.152505 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:28.152456    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1d159a-5c484658dc-8qvxq" event={"ID":"1ed630bf-cf60-4e37-9113-a1707b99937f","Type":"ContainerStarted","Data":"dd61f44cdbdbff9a9b8c70ea8c8cc2997d1eae214f81d484aaf387a2325fffee"}
Nov 03 20:32:28.152505 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:28.152512    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1d159a-5c484658dc-8qvxq" event={"ID":"1ed630bf-cf60-4e37-9113-a1707b99937f","Type":"ContainerStarted","Data":"91798eb4a6b2d791273fd5918f34a359436bcbbf04374c5c7fb7d5abfc34c717"}
Nov 03 20:32:28.152788 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:28.152536    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1d159a-5c484658dc-8qvxq"
Nov 03 20:32:28.154153 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:28.154121    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-1d159a-5497649584-zvjng" event={"ID":"96055efa-780d-46b1-8d17-31587157b508","Type":"ContainerStarted","Data":"4acb55c63c600be5f8c437ba91538b91b5cdd1506ef232360c71451b0e035676"}
Nov 03 20:32:28.154274 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:28.154172    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-1d159a-5497649584-zvjng" event={"ID":"96055efa-780d-46b1-8d17-31587157b508","Type":"ContainerStarted","Data":"1f4419d15f67647080e121042acf1150dfd460859516e223f90a74c2f4e8a6eb"}
Nov 03 20:32:28.154274 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:28.154193    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-1d159a-5497649584-zvjng"
Nov 03 20:32:28.195319 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:28.195260    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1d159a-5c484658dc-8qvxq" podStartSLOduration=2.195244152 podStartE2EDuration="2.195244152s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:32:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:32:28.194424247 +0000 UTC m=+1682.347086170" watchObservedRunningTime="2026-11-03 20:32:28.195244152 +0000 UTC m=+1682.347906090"
Nov 03 20:32:28.239509 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:28.239448    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-1d159a-5497649584-zvjng" podStartSLOduration=2.239432513 podStartE2EDuration="2.239432513s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:32:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:32:28.237273358 +0000 UTC m=+1682.389935244" watchObservedRunningTime="2026-11-03 20:32:28.239432513 +0000 UTC m=+1682.392094399"
Nov 03 20:32:39.165154 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:39.165115    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-1d159a-5497649584-zvjng"
Nov 03 20:32:39.165602 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:39.165192    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1d159a-5c484658dc-8qvxq"
Nov 03 20:32:49.717371 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:49.717331    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-fab871-openshift-default-698b94dc5f-l6ggn"]
Nov 03 20:32:49.718078 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:49.717612    5570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-fab871-openshift-default-698b94dc5f-l6ggn" podUID="46af9c86-b058-4323-97a2-af8d89d3dc8f" containerName="istio-proxy" containerID="cri-o://a80861982cfdef69f6febbf07cdbb95484719e8d26e334dfd35ae52132567687" gracePeriod=30
Nov 03 20:32:51.627786 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:51.627731    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-e3ba87-7c7fd945bd-dzmfm"]
Nov 03 20:32:55.059047 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:55.059016    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-fab871-openshift-default-698b94dc5f-l6ggn"
Nov 03 20:32:55.101142 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:55.101089    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46af9c86-b058-4323-97a2-af8d89d3dc8f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46af9c86-b058-4323-97a2-af8d89d3dc8f-workload-certs\") pod \"46af9c86-b058-4323-97a2-af8d89d3dc8f\" (UID: \"46af9c86-b058-4323-97a2-af8d89d3dc8f\") "
Nov 03 20:32:55.101347 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:55.101297    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46af9c86-b058-4323-97a2-af8d89d3dc8f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46af9c86-b058-4323-97a2-af8d89d3dc8f-workload-socket\") pod \"46af9c86-b058-4323-97a2-af8d89d3dc8f\" (UID: \"46af9c86-b058-4323-97a2-af8d89d3dc8f\") "
Nov 03 20:32:55.101347 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:55.101337    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46af9c86-b058-4323-97a2-af8d89d3dc8f-kube-api-access-ztn88\" (UniqueName: \"kubernetes.io/projected/46af9c86-b058-4323-97a2-af8d89d3dc8f-kube-api-access-ztn88\") pod \"46af9c86-b058-4323-97a2-af8d89d3dc8f\" (UID: \"46af9c86-b058-4323-97a2-af8d89d3dc8f\") "
Nov 03 20:32:55.101477 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:55.101370    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46af9c86-b058-4323-97a2-af8d89d3dc8f-workload-certs" pod "46af9c86-b058-4323-97a2-af8d89d3dc8f" (UID: "46af9c86-b058-4323-97a2-af8d89d3dc8f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:32:55.101477 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:55.101385    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46af9c86-b058-4323-97a2-af8d89d3dc8f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46af9c86-b058-4323-97a2-af8d89d3dc8f-istio-envoy\") pod \"46af9c86-b058-4323-97a2-af8d89d3dc8f\" (UID: \"46af9c86-b058-4323-97a2-af8d89d3dc8f\") "
Nov 03 20:32:55.101477 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:55.101467    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46af9c86-b058-4323-97a2-af8d89d3dc8f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46af9c86-b058-4323-97a2-af8d89d3dc8f-istio-data\") pod \"46af9c86-b058-4323-97a2-af8d89d3dc8f\" (UID: \"46af9c86-b058-4323-97a2-af8d89d3dc8f\") "
Nov 03 20:32:55.101628 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:55.101527    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/46af9c86-b058-4323-97a2-af8d89d3dc8f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46af9c86-b058-4323-97a2-af8d89d3dc8f-istio-podinfo\") pod \"46af9c86-b058-4323-97a2-af8d89d3dc8f\" (UID: \"46af9c86-b058-4323-97a2-af8d89d3dc8f\") "
Nov 03 20:32:55.101628 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:55.101595    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46af9c86-b058-4323-97a2-af8d89d3dc8f-workload-socket" pod "46af9c86-b058-4323-97a2-af8d89d3dc8f" (UID: "46af9c86-b058-4323-97a2-af8d89d3dc8f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:32:55.101733 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:55.101686    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46af9c86-b058-4323-97a2-af8d89d3dc8f-istio-token\" (UniqueName: \"kubernetes.io/projected/46af9c86-b058-4323-97a2-af8d89d3dc8f-istio-token\") pod \"46af9c86-b058-4323-97a2-af8d89d3dc8f\" (UID: \"46af9c86-b058-4323-97a2-af8d89d3dc8f\") "
Nov 03 20:32:55.101733 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:55.101716    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46af9c86-b058-4323-97a2-af8d89d3dc8f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46af9c86-b058-4323-97a2-af8d89d3dc8f-istiod-ca-cert\") pod \"46af9c86-b058-4323-97a2-af8d89d3dc8f\" (UID: \"46af9c86-b058-4323-97a2-af8d89d3dc8f\") "
Nov 03 20:32:55.101866 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:55.101750    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46af9c86-b058-4323-97a2-af8d89d3dc8f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46af9c86-b058-4323-97a2-af8d89d3dc8f-credential-socket\") pod \"46af9c86-b058-4323-97a2-af8d89d3dc8f\" (UID: \"46af9c86-b058-4323-97a2-af8d89d3dc8f\") "
Nov 03 20:32:55.101866 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:55.101805    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46af9c86-b058-4323-97a2-af8d89d3dc8f-istio-data" pod "46af9c86-b058-4323-97a2-af8d89d3dc8f" (UID: "46af9c86-b058-4323-97a2-af8d89d3dc8f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:32:55.102162 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:55.102110    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46af9c86-b058-4323-97a2-af8d89d3dc8f-credential-socket" pod "46af9c86-b058-4323-97a2-af8d89d3dc8f" (UID: "46af9c86-b058-4323-97a2-af8d89d3dc8f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:32:55.102162 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:55.102134    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46af9c86-b058-4323-97a2-af8d89d3dc8f-istiod-ca-cert" pod "46af9c86-b058-4323-97a2-af8d89d3dc8f" (UID: "46af9c86-b058-4323-97a2-af8d89d3dc8f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:32:55.102554 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:55.102535    5570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46af9c86-b058-4323-97a2-af8d89d3dc8f-istio-data\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:32:55.102633 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:55.102563    5570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46af9c86-b058-4323-97a2-af8d89d3dc8f-istiod-ca-cert\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:32:55.102633 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:55.102578    5570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46af9c86-b058-4323-97a2-af8d89d3dc8f-credential-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:32:55.102633 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:55.102592    5570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46af9c86-b058-4323-97a2-af8d89d3dc8f-workload-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:32:55.102633 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:55.102605    5570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46af9c86-b058-4323-97a2-af8d89d3dc8f-workload-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:32:55.103911 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:55.103868    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46af9c86-b058-4323-97a2-af8d89d3dc8f-kube-api-access-ztn88" pod "46af9c86-b058-4323-97a2-af8d89d3dc8f" (UID: "46af9c86-b058-4323-97a2-af8d89d3dc8f"). InnerVolumeSpecName "kube-api-access-ztn88". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:32:55.104002 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:55.103910    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/46af9c86-b058-4323-97a2-af8d89d3dc8f-istio-podinfo" pod "46af9c86-b058-4323-97a2-af8d89d3dc8f" (UID: "46af9c86-b058-4323-97a2-af8d89d3dc8f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:32:55.104089 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:55.104063    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46af9c86-b058-4323-97a2-af8d89d3dc8f-istio-envoy" pod "46af9c86-b058-4323-97a2-af8d89d3dc8f" (UID: "46af9c86-b058-4323-97a2-af8d89d3dc8f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:32:55.104459 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:55.104436    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46af9c86-b058-4323-97a2-af8d89d3dc8f-istio-token" pod "46af9c86-b058-4323-97a2-af8d89d3dc8f" (UID: "46af9c86-b058-4323-97a2-af8d89d3dc8f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:32:55.203995 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:55.203958    5570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46af9c86-b058-4323-97a2-af8d89d3dc8f-istio-token\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:32:55.203995 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:55.203993    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ztn88\" (UniqueName: \"kubernetes.io/projected/46af9c86-b058-4323-97a2-af8d89d3dc8f-kube-api-access-ztn88\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:32:55.203995 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:55.204003    5570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46af9c86-b058-4323-97a2-af8d89d3dc8f-istio-envoy\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:32:55.204261 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:55.204012    5570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46af9c86-b058-4323-97a2-af8d89d3dc8f-istio-podinfo\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:32:55.564208 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:55.564172    5570 generic.go:356] "Generic (PLEG): container finished" podID="46af9c86-b058-4323-97a2-af8d89d3dc8f" containerID="a80861982cfdef69f6febbf07cdbb95484719e8d26e334dfd35ae52132567687" exitCode=0
Nov 03 20:32:55.564397 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:55.564222    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-fab871-openshift-default-698b94dc5f-l6ggn"
Nov 03 20:32:55.564397 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:55.564251    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-fab871-openshift-default-698b94dc5f-l6ggn" event={"ID":"46af9c86-b058-4323-97a2-af8d89d3dc8f","Type":"ContainerDied","Data":"a80861982cfdef69f6febbf07cdbb95484719e8d26e334dfd35ae52132567687"}
Nov 03 20:32:55.564397 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:55.564306    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-fab871-openshift-default-698b94dc5f-l6ggn" event={"ID":"46af9c86-b058-4323-97a2-af8d89d3dc8f","Type":"ContainerDied","Data":"79aac0184b8eb138f726e3ecd811259ca798561e8b42ade7acc18c7218a9f64b"}
Nov 03 20:32:55.564397 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:55.564332    5570 scope.go:122] "RemoveContainer" containerID="a80861982cfdef69f6febbf07cdbb95484719e8d26e334dfd35ae52132567687"
Nov 03 20:32:55.575787 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:55.575754    5570 scope.go:122] "RemoveContainer" containerID="3fef9e5c8d3757cb3a7f4684106e6b30c1b1f324ad8bb942240b0cf6e9a436d8"
Nov 03 20:32:55.586333 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:55.586311    5570 scope.go:122] "RemoveContainer" containerID="a80861982cfdef69f6febbf07cdbb95484719e8d26e334dfd35ae52132567687"
Nov 03 20:32:55.586645 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:32:55.586624    5570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a80861982cfdef69f6febbf07cdbb95484719e8d26e334dfd35ae52132567687\": container with ID starting with a80861982cfdef69f6febbf07cdbb95484719e8d26e334dfd35ae52132567687 not found: ID does not exist" containerID="a80861982cfdef69f6febbf07cdbb95484719e8d26e334dfd35ae52132567687"
Nov 03 20:32:55.586707 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:55.586654    5570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a80861982cfdef69f6febbf07cdbb95484719e8d26e334dfd35ae52132567687"} err="failed to get container status \"a80861982cfdef69f6febbf07cdbb95484719e8d26e334dfd35ae52132567687\": rpc error: code = NotFound desc = could not find container \"a80861982cfdef69f6febbf07cdbb95484719e8d26e334dfd35ae52132567687\": container with ID starting with a80861982cfdef69f6febbf07cdbb95484719e8d26e334dfd35ae52132567687 not found: ID does not exist"
Nov 03 20:32:55.586707 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:55.586675    5570 scope.go:122] "RemoveContainer" containerID="3fef9e5c8d3757cb3a7f4684106e6b30c1b1f324ad8bb942240b0cf6e9a436d8"
Nov 03 20:32:55.587001 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:32:55.586974    5570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3fef9e5c8d3757cb3a7f4684106e6b30c1b1f324ad8bb942240b0cf6e9a436d8\": container with ID starting with 3fef9e5c8d3757cb3a7f4684106e6b30c1b1f324ad8bb942240b0cf6e9a436d8 not found: ID does not exist" containerID="3fef9e5c8d3757cb3a7f4684106e6b30c1b1f324ad8bb942240b0cf6e9a436d8"
Nov 03 20:32:55.587081 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:55.587010    5570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3fef9e5c8d3757cb3a7f4684106e6b30c1b1f324ad8bb942240b0cf6e9a436d8"} err="failed to get container status \"3fef9e5c8d3757cb3a7f4684106e6b30c1b1f324ad8bb942240b0cf6e9a436d8\": rpc error: code = NotFound desc = could not find container \"3fef9e5c8d3757cb3a7f4684106e6b30c1b1f324ad8bb942240b0cf6e9a436d8\": container with ID starting with 3fef9e5c8d3757cb3a7f4684106e6b30c1b1f324ad8bb942240b0cf6e9a436d8 not found: ID does not exist"
Nov 03 20:32:55.624691 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:55.624656    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-fab871-openshift-default-698b94dc5f-l6ggn"]
Nov 03 20:32:55.637757 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:55.637722    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-fab871-openshift-default-698b94dc5f-l6ggn"]
Nov 03 20:32:55.967116 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:55.967081    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46af9c86-b058-4323-97a2-af8d89d3dc8f" path="/var/lib/kubelet/pods/46af9c86-b058-4323-97a2-af8d89d3dc8f/volumes"
Nov 03 20:32:56.628143 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:56.628099    5570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-e3ba87-7c7fd945bd-dzmfm" podUID="5046089f-f2fa-4ed2-b424-7661fb3ea8c2" containerName="payload-processing" containerID="cri-o://28ec97b6355ca6177efbbf8aa5e614a1eaa02acdfec2e882ee20ad66e948eaad" gracePeriod=25
Nov 03 20:32:56.918250 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:56.918223    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-e3ba87-7c7fd945bd-dzmfm"
Nov 03 20:32:57.059950 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:57.059916    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5046089f-f2fa-4ed2-b424-7661fb3ea8c2-kube-api-access-qj9s9\" (UniqueName: \"kubernetes.io/projected/5046089f-f2fa-4ed2-b424-7661fb3ea8c2-kube-api-access-qj9s9\") pod \"5046089f-f2fa-4ed2-b424-7661fb3ea8c2\" (UID: \"5046089f-f2fa-4ed2-b424-7661fb3ea8c2\") "
Nov 03 20:32:57.060136 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:57.060101    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5046089f-f2fa-4ed2-b424-7661fb3ea8c2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5046089f-f2fa-4ed2-b424-7661fb3ea8c2-plugins-config-volume\") pod \"5046089f-f2fa-4ed2-b424-7661fb3ea8c2\" (UID: \"5046089f-f2fa-4ed2-b424-7661fb3ea8c2\") "
Nov 03 20:32:57.060551 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:57.060521    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5046089f-f2fa-4ed2-b424-7661fb3ea8c2-plugins-config-volume" pod "5046089f-f2fa-4ed2-b424-7661fb3ea8c2" (UID: "5046089f-f2fa-4ed2-b424-7661fb3ea8c2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:32:57.060789 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:57.060762    5570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5046089f-f2fa-4ed2-b424-7661fb3ea8c2-plugins-config-volume\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:32:57.062133 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:57.062108    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5046089f-f2fa-4ed2-b424-7661fb3ea8c2-kube-api-access-qj9s9" pod "5046089f-f2fa-4ed2-b424-7661fb3ea8c2" (UID: "5046089f-f2fa-4ed2-b424-7661fb3ea8c2"). InnerVolumeSpecName "kube-api-access-qj9s9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:32:57.162638 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:57.162541    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qj9s9\" (UniqueName: \"kubernetes.io/projected/5046089f-f2fa-4ed2-b424-7661fb3ea8c2-kube-api-access-qj9s9\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:32:57.229823 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:32:57.229767    5570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8bwhz" containerName="extractor"
Nov 03 20:32:57.580649 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:57.580613    5570 generic.go:356] "Generic (PLEG): container finished" podID="5046089f-f2fa-4ed2-b424-7661fb3ea8c2" containerID="28ec97b6355ca6177efbbf8aa5e614a1eaa02acdfec2e882ee20ad66e948eaad" exitCode=0
Nov 03 20:32:57.580865 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:57.580665    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-e3ba87-7c7fd945bd-dzmfm"
Nov 03 20:32:57.580865 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:57.580677    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-e3ba87-7c7fd945bd-dzmfm" event={"ID":"5046089f-f2fa-4ed2-b424-7661fb3ea8c2","Type":"ContainerDied","Data":"28ec97b6355ca6177efbbf8aa5e614a1eaa02acdfec2e882ee20ad66e948eaad"}
Nov 03 20:32:57.580865 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:57.580723    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-e3ba87-7c7fd945bd-dzmfm" event={"ID":"5046089f-f2fa-4ed2-b424-7661fb3ea8c2","Type":"ContainerDied","Data":"adf23f87b70594cd11222dd65534aac6826201d0db6fddeb642a2a8380af7075"}
Nov 03 20:32:57.580865 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:57.580765    5570 scope.go:122] "RemoveContainer" containerID="28ec97b6355ca6177efbbf8aa5e614a1eaa02acdfec2e882ee20ad66e948eaad"
Nov 03 20:32:57.592611 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:57.592580    5570 scope.go:122] "RemoveContainer" containerID="28ec97b6355ca6177efbbf8aa5e614a1eaa02acdfec2e882ee20ad66e948eaad"
Nov 03 20:32:57.592938 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:32:57.592918    5570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28ec97b6355ca6177efbbf8aa5e614a1eaa02acdfec2e882ee20ad66e948eaad\": container with ID starting with 28ec97b6355ca6177efbbf8aa5e614a1eaa02acdfec2e882ee20ad66e948eaad not found: ID does not exist" containerID="28ec97b6355ca6177efbbf8aa5e614a1eaa02acdfec2e882ee20ad66e948eaad"
Nov 03 20:32:57.593032 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:57.592946    5570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28ec97b6355ca6177efbbf8aa5e614a1eaa02acdfec2e882ee20ad66e948eaad"} err="failed to get container status \"28ec97b6355ca6177efbbf8aa5e614a1eaa02acdfec2e882ee20ad66e948eaad\": rpc error: code = NotFound desc = could not find container \"28ec97b6355ca6177efbbf8aa5e614a1eaa02acdfec2e882ee20ad66e948eaad\": container with ID starting with 28ec97b6355ca6177efbbf8aa5e614a1eaa02acdfec2e882ee20ad66e948eaad not found: ID does not exist"
Nov 03 20:32:57.638663 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:57.638623    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-e3ba87-7c7fd945bd-dzmfm"]
Nov 03 20:32:57.651846 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:57.651793    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-e3ba87-7c7fd945bd-dzmfm"]
Nov 03 20:32:57.967278 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:32:57.967241    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5046089f-f2fa-4ed2-b424-7661fb3ea8c2" path="/var/lib/kubelet/pods/5046089f-f2fa-4ed2-b424-7661fb3ea8c2/volumes"
Nov 03 20:33:00.784937 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:00.784901    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d05604-5fb56668c4-24x95"]
Nov 03 20:33:00.814608 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:00.814572    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d05604-7799fccfc9-m4xqb"]
Nov 03 20:33:03.288003 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:03.287958    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-d1aa66-openshift-default-8457c96657-lt2kw"]
Nov 03 20:33:03.288637 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:03.288254    5570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-d1aa66-openshift-default-8457c96657-lt2kw" podUID="5a701695-c67d-4cda-a46d-7834b22750ed" containerName="istio-proxy" containerID="cri-o://7d6407f3a09e7889871e501625f155ce4ccdfbd5bf75f845f25ed94d35a2e75f" gracePeriod=30
Nov 03 20:33:04.201369 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:04.201330    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-a9ba83-openshift-default-8c675fb59-486wc"]
Nov 03 20:33:04.202433 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:04.202402    5570 cpu_manager.go:414] "containerMap: removing container" podUID="46af9c86-b058-4323-97a2-af8d89d3dc8f" containerName="istio-proxy"
Nov 03 20:33:04.202433 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:04.202431    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46af9c86-b058-4323-97a2-af8d89d3dc8f" containerName="istio-proxy"
Nov 03 20:33:04.202618 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:04.202527    5570 cpu_manager.go:414] "containerMap: removing container" podUID="5046089f-f2fa-4ed2-b424-7661fb3ea8c2" containerName="payload-processing"
Nov 03 20:33:04.202618 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:04.202536    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5046089f-f2fa-4ed2-b424-7661fb3ea8c2" containerName="payload-processing"
Nov 03 20:33:04.202708 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:04.202695    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="5046089f-f2fa-4ed2-b424-7661fb3ea8c2" containerName="payload-processing"
Nov 03 20:33:04.202743 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:04.202712    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="46af9c86-b058-4323-97a2-af8d89d3dc8f" containerName="istio-proxy"
Nov 03 20:33:04.208625 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:04.208592    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a9ba83-openshift-default-8c675fb59-486wc"
Nov 03 20:33:04.233705 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:04.233668    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-a9ba83-openshift-default-dockercfg-6phwj\""
Nov 03 20:33:04.233981 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:04.233955    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-a9ba83-openshift-default-8c675fb59-486wc"]
Nov 03 20:33:04.370799 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:04.367158    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f913fe36-7256-4171-ad6f-db25850da8ce-workload-certs\") pod \"e2e-inf-a-a9ba83-openshift-default-8c675fb59-486wc\" (UID: \"f913fe36-7256-4171-ad6f-db25850da8ce\") " pod="openshift-ingress/e2e-inf-a-a9ba83-openshift-default-8c675fb59-486wc"
Nov 03 20:33:04.370799 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:04.367221    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f913fe36-7256-4171-ad6f-db25850da8ce-istio-podinfo\") pod \"e2e-inf-a-a9ba83-openshift-default-8c675fb59-486wc\" (UID: \"f913fe36-7256-4171-ad6f-db25850da8ce\") " pod="openshift-ingress/e2e-inf-a-a9ba83-openshift-default-8c675fb59-486wc"
Nov 03 20:33:04.370799 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:04.367298    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86clh\" (UniqueName: \"kubernetes.io/projected/f913fe36-7256-4171-ad6f-db25850da8ce-kube-api-access-86clh\") pod \"e2e-inf-a-a9ba83-openshift-default-8c675fb59-486wc\" (UID: \"f913fe36-7256-4171-ad6f-db25850da8ce\") " pod="openshift-ingress/e2e-inf-a-a9ba83-openshift-default-8c675fb59-486wc"
Nov 03 20:33:04.370799 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:04.367373    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f913fe36-7256-4171-ad6f-db25850da8ce-istio-token\") pod \"e2e-inf-a-a9ba83-openshift-default-8c675fb59-486wc\" (UID: \"f913fe36-7256-4171-ad6f-db25850da8ce\") " pod="openshift-ingress/e2e-inf-a-a9ba83-openshift-default-8c675fb59-486wc"
Nov 03 20:33:04.370799 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:04.367549    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f913fe36-7256-4171-ad6f-db25850da8ce-credential-socket\") pod \"e2e-inf-a-a9ba83-openshift-default-8c675fb59-486wc\" (UID: \"f913fe36-7256-4171-ad6f-db25850da8ce\") " pod="openshift-ingress/e2e-inf-a-a9ba83-openshift-default-8c675fb59-486wc"
Nov 03 20:33:04.370799 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:04.367615    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f913fe36-7256-4171-ad6f-db25850da8ce-istio-envoy\") pod \"e2e-inf-a-a9ba83-openshift-default-8c675fb59-486wc\" (UID: \"f913fe36-7256-4171-ad6f-db25850da8ce\") " pod="openshift-ingress/e2e-inf-a-a9ba83-openshift-default-8c675fb59-486wc"
Nov 03 20:33:04.370799 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:04.367757    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f913fe36-7256-4171-ad6f-db25850da8ce-workload-socket\") pod \"e2e-inf-a-a9ba83-openshift-default-8c675fb59-486wc\" (UID: \"f913fe36-7256-4171-ad6f-db25850da8ce\") " pod="openshift-ingress/e2e-inf-a-a9ba83-openshift-default-8c675fb59-486wc"
Nov 03 20:33:04.370799 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:04.367839    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f913fe36-7256-4171-ad6f-db25850da8ce-istiod-ca-cert\") pod \"e2e-inf-a-a9ba83-openshift-default-8c675fb59-486wc\" (UID: \"f913fe36-7256-4171-ad6f-db25850da8ce\") " pod="openshift-ingress/e2e-inf-a-a9ba83-openshift-default-8c675fb59-486wc"
Nov 03 20:33:04.370799 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:04.367923    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f913fe36-7256-4171-ad6f-db25850da8ce-istio-data\") pod \"e2e-inf-a-a9ba83-openshift-default-8c675fb59-486wc\" (UID: \"f913fe36-7256-4171-ad6f-db25850da8ce\") " pod="openshift-ingress/e2e-inf-a-a9ba83-openshift-default-8c675fb59-486wc"
Nov 03 20:33:04.486846 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:04.478446    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f913fe36-7256-4171-ad6f-db25850da8ce-credential-socket\") pod \"e2e-inf-a-a9ba83-openshift-default-8c675fb59-486wc\" (UID: \"f913fe36-7256-4171-ad6f-db25850da8ce\") " pod="openshift-ingress/e2e-inf-a-a9ba83-openshift-default-8c675fb59-486wc"
Nov 03 20:33:04.486846 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:04.478532    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f913fe36-7256-4171-ad6f-db25850da8ce-istio-envoy\") pod \"e2e-inf-a-a9ba83-openshift-default-8c675fb59-486wc\" (UID: \"f913fe36-7256-4171-ad6f-db25850da8ce\") " pod="openshift-ingress/e2e-inf-a-a9ba83-openshift-default-8c675fb59-486wc"
Nov 03 20:33:04.486846 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:04.478653    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f913fe36-7256-4171-ad6f-db25850da8ce-workload-socket\") pod \"e2e-inf-a-a9ba83-openshift-default-8c675fb59-486wc\" (UID: \"f913fe36-7256-4171-ad6f-db25850da8ce\") " pod="openshift-ingress/e2e-inf-a-a9ba83-openshift-default-8c675fb59-486wc"
Nov 03 20:33:04.486846 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:04.478692    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f913fe36-7256-4171-ad6f-db25850da8ce-istiod-ca-cert\") pod \"e2e-inf-a-a9ba83-openshift-default-8c675fb59-486wc\" (UID: \"f913fe36-7256-4171-ad6f-db25850da8ce\") " pod="openshift-ingress/e2e-inf-a-a9ba83-openshift-default-8c675fb59-486wc"
Nov 03 20:33:04.486846 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:04.478734    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f913fe36-7256-4171-ad6f-db25850da8ce-istio-data\") pod \"e2e-inf-a-a9ba83-openshift-default-8c675fb59-486wc\" (UID: \"f913fe36-7256-4171-ad6f-db25850da8ce\") " pod="openshift-ingress/e2e-inf-a-a9ba83-openshift-default-8c675fb59-486wc"
Nov 03 20:33:04.486846 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:04.478815    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f913fe36-7256-4171-ad6f-db25850da8ce-workload-certs\") pod \"e2e-inf-a-a9ba83-openshift-default-8c675fb59-486wc\" (UID: \"f913fe36-7256-4171-ad6f-db25850da8ce\") " pod="openshift-ingress/e2e-inf-a-a9ba83-openshift-default-8c675fb59-486wc"
Nov 03 20:33:04.486846 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:04.478842    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f913fe36-7256-4171-ad6f-db25850da8ce-istio-podinfo\") pod \"e2e-inf-a-a9ba83-openshift-default-8c675fb59-486wc\" (UID: \"f913fe36-7256-4171-ad6f-db25850da8ce\") " pod="openshift-ingress/e2e-inf-a-a9ba83-openshift-default-8c675fb59-486wc"
Nov 03 20:33:04.486846 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:04.478925    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86clh\" (UniqueName: \"kubernetes.io/projected/f913fe36-7256-4171-ad6f-db25850da8ce-kube-api-access-86clh\") pod \"e2e-inf-a-a9ba83-openshift-default-8c675fb59-486wc\" (UID: \"f913fe36-7256-4171-ad6f-db25850da8ce\") " pod="openshift-ingress/e2e-inf-a-a9ba83-openshift-default-8c675fb59-486wc"
Nov 03 20:33:04.486846 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:04.478993    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f913fe36-7256-4171-ad6f-db25850da8ce-istio-token\") pod \"e2e-inf-a-a9ba83-openshift-default-8c675fb59-486wc\" (UID: \"f913fe36-7256-4171-ad6f-db25850da8ce\") " pod="openshift-ingress/e2e-inf-a-a9ba83-openshift-default-8c675fb59-486wc"
Nov 03 20:33:04.486846 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:04.480010    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f913fe36-7256-4171-ad6f-db25850da8ce-credential-socket\") pod \"e2e-inf-a-a9ba83-openshift-default-8c675fb59-486wc\" (UID: \"f913fe36-7256-4171-ad6f-db25850da8ce\") " pod="openshift-ingress/e2e-inf-a-a9ba83-openshift-default-8c675fb59-486wc"
Nov 03 20:33:04.486846 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:04.482045    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f913fe36-7256-4171-ad6f-db25850da8ce-workload-certs\") pod \"e2e-inf-a-a9ba83-openshift-default-8c675fb59-486wc\" (UID: \"f913fe36-7256-4171-ad6f-db25850da8ce\") " pod="openshift-ingress/e2e-inf-a-a9ba83-openshift-default-8c675fb59-486wc"
Nov 03 20:33:04.486846 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:04.482282    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f913fe36-7256-4171-ad6f-db25850da8ce-workload-socket\") pod \"e2e-inf-a-a9ba83-openshift-default-8c675fb59-486wc\" (UID: \"f913fe36-7256-4171-ad6f-db25850da8ce\") " pod="openshift-ingress/e2e-inf-a-a9ba83-openshift-default-8c675fb59-486wc"
Nov 03 20:33:04.486846 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:04.483158    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f913fe36-7256-4171-ad6f-db25850da8ce-istio-data\") pod \"e2e-inf-a-a9ba83-openshift-default-8c675fb59-486wc\" (UID: \"f913fe36-7256-4171-ad6f-db25850da8ce\") " pod="openshift-ingress/e2e-inf-a-a9ba83-openshift-default-8c675fb59-486wc"
Nov 03 20:33:04.508796 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:04.500144    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f913fe36-7256-4171-ad6f-db25850da8ce-istiod-ca-cert\") pod \"e2e-inf-a-a9ba83-openshift-default-8c675fb59-486wc\" (UID: \"f913fe36-7256-4171-ad6f-db25850da8ce\") " pod="openshift-ingress/e2e-inf-a-a9ba83-openshift-default-8c675fb59-486wc"
Nov 03 20:33:04.508796 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:04.503324    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f913fe36-7256-4171-ad6f-db25850da8ce-istio-podinfo\") pod \"e2e-inf-a-a9ba83-openshift-default-8c675fb59-486wc\" (UID: \"f913fe36-7256-4171-ad6f-db25850da8ce\") " pod="openshift-ingress/e2e-inf-a-a9ba83-openshift-default-8c675fb59-486wc"
Nov 03 20:33:04.526212 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:04.520327    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f913fe36-7256-4171-ad6f-db25850da8ce-istio-token\") pod \"e2e-inf-a-a9ba83-openshift-default-8c675fb59-486wc\" (UID: \"f913fe36-7256-4171-ad6f-db25850da8ce\") " pod="openshift-ingress/e2e-inf-a-a9ba83-openshift-default-8c675fb59-486wc"
Nov 03 20:33:04.527751 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:04.527719    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f913fe36-7256-4171-ad6f-db25850da8ce-istio-envoy\") pod \"e2e-inf-a-a9ba83-openshift-default-8c675fb59-486wc\" (UID: \"f913fe36-7256-4171-ad6f-db25850da8ce\") " pod="openshift-ingress/e2e-inf-a-a9ba83-openshift-default-8c675fb59-486wc"
Nov 03 20:33:04.536721 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:04.536690    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86clh\" (UniqueName: \"kubernetes.io/projected/f913fe36-7256-4171-ad6f-db25850da8ce-kube-api-access-86clh\") pod \"e2e-inf-a-a9ba83-openshift-default-8c675fb59-486wc\" (UID: \"f913fe36-7256-4171-ad6f-db25850da8ce\") " pod="openshift-ingress/e2e-inf-a-a9ba83-openshift-default-8c675fb59-486wc"
Nov 03 20:33:04.834505 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:04.834407    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a9ba83-openshift-default-8c675fb59-486wc"
Nov 03 20:33:05.155729 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:05.155696    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-a9ba83-openshift-default-8c675fb59-486wc"]
Nov 03 20:33:05.156658 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:33:05.156630    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf913fe36_7256_4171_ad6f_db25850da8ce.slice/crio-a148340f62f3f9ce63f99870772c1b10dd09ef6a238a2187a61e65d1fa1e026d WatchSource:0}: Error finding container a148340f62f3f9ce63f99870772c1b10dd09ef6a238a2187a61e65d1fa1e026d: Status 404 returned error can't find the container with id a148340f62f3f9ce63f99870772c1b10dd09ef6a238a2187a61e65d1fa1e026d
Nov 03 20:33:05.159689 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:05.159650    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:33:05.159848 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:05.159736    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:33:05.159848 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:05.159803    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:33:05.698088 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:05.698038    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a9ba83-openshift-default-8c675fb59-486wc" event={"ID":"f913fe36-7256-4171-ad6f-db25850da8ce","Type":"ContainerStarted","Data":"edeb6af257049566a198a0ed1235fc9c416060dc50a5fd85246d010da5f12ea0"}
Nov 03 20:33:05.698539 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:05.698120    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a9ba83-openshift-default-8c675fb59-486wc" event={"ID":"f913fe36-7256-4171-ad6f-db25850da8ce","Type":"ContainerStarted","Data":"a148340f62f3f9ce63f99870772c1b10dd09ef6a238a2187a61e65d1fa1e026d"}
Nov 03 20:33:05.772786 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:05.772718    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-a9ba83-openshift-default-8c675fb59-486wc" podStartSLOduration=1.772702095 podStartE2EDuration="1.772702095s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:33:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:33:05.757293643 +0000 UTC m=+1719.909955528" watchObservedRunningTime="2026-11-03 20:33:05.772702095 +0000 UTC m=+1719.925363982"
Nov 03 20:33:05.785838 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:05.785759    5570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-d05604-5fb56668c4-24x95" podUID="6216548e-093f-4c34-bb8a-01fe1ad91aa8" containerName="payload-processing" containerID="cri-o://26aad60548fb9ae9dc303f7839e88dc85f6e3efae94f71132f80941b40cc45ff" gracePeriod=25
Nov 03 20:33:05.815476 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:05.815439    5570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d05604-7799fccfc9-m4xqb" podUID="f9114ccd-117b-4fb8-9314-434a65aeabe6" containerName="payload-pre-processing" containerID="cri-o://596bfe0429d65cc33f5a9a1b08e2d9f0fa5b38e1da423abdc5f62b353dc81499" gracePeriod=25
Nov 03 20:33:05.834712 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:05.834677    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-a9ba83-openshift-default-8c675fb59-486wc"
Nov 03 20:33:06.153371 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:06.153341    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-d05604-5fb56668c4-24x95"
Nov 03 20:33:06.252443 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:06.252409    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d05604-7799fccfc9-m4xqb"
Nov 03 20:33:06.325055 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:06.325021    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6216548e-093f-4c34-bb8a-01fe1ad91aa8-kube-api-access-2clp9\" (UniqueName: \"kubernetes.io/projected/6216548e-093f-4c34-bb8a-01fe1ad91aa8-kube-api-access-2clp9\") pod \"6216548e-093f-4c34-bb8a-01fe1ad91aa8\" (UID: \"6216548e-093f-4c34-bb8a-01fe1ad91aa8\") "
Nov 03 20:33:06.325279 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:06.325152    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9114ccd-117b-4fb8-9314-434a65aeabe6-kube-api-access-pwvwb\" (UniqueName: \"kubernetes.io/projected/f9114ccd-117b-4fb8-9314-434a65aeabe6-kube-api-access-pwvwb\") pod \"f9114ccd-117b-4fb8-9314-434a65aeabe6\" (UID: \"f9114ccd-117b-4fb8-9314-434a65aeabe6\") "
Nov 03 20:33:06.325279 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:06.325252    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6216548e-093f-4c34-bb8a-01fe1ad91aa8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6216548e-093f-4c34-bb8a-01fe1ad91aa8-plugins-config-volume\") pod \"6216548e-093f-4c34-bb8a-01fe1ad91aa8\" (UID: \"6216548e-093f-4c34-bb8a-01fe1ad91aa8\") "
Nov 03 20:33:06.325417 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:06.325354    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9114ccd-117b-4fb8-9314-434a65aeabe6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9114ccd-117b-4fb8-9314-434a65aeabe6-plugins-config-volume\") pod \"f9114ccd-117b-4fb8-9314-434a65aeabe6\" (UID: \"f9114ccd-117b-4fb8-9314-434a65aeabe6\") "
Nov 03 20:33:06.328823 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:06.325902    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6216548e-093f-4c34-bb8a-01fe1ad91aa8-plugins-config-volume" pod "6216548e-093f-4c34-bb8a-01fe1ad91aa8" (UID: "6216548e-093f-4c34-bb8a-01fe1ad91aa8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:33:06.328823 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:06.325963    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9114ccd-117b-4fb8-9314-434a65aeabe6-plugins-config-volume" pod "f9114ccd-117b-4fb8-9314-434a65aeabe6" (UID: "f9114ccd-117b-4fb8-9314-434a65aeabe6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:33:06.328823 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:06.327957    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6216548e-093f-4c34-bb8a-01fe1ad91aa8-kube-api-access-2clp9" pod "6216548e-093f-4c34-bb8a-01fe1ad91aa8" (UID: "6216548e-093f-4c34-bb8a-01fe1ad91aa8"). InnerVolumeSpecName "kube-api-access-2clp9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:33:06.328823 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:06.327989    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9114ccd-117b-4fb8-9314-434a65aeabe6-kube-api-access-pwvwb" pod "f9114ccd-117b-4fb8-9314-434a65aeabe6" (UID: "f9114ccd-117b-4fb8-9314-434a65aeabe6"). InnerVolumeSpecName "kube-api-access-pwvwb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:33:06.427416 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:06.427380    5570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6216548e-093f-4c34-bb8a-01fe1ad91aa8-plugins-config-volume\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:33:06.427416 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:06.427412    5570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9114ccd-117b-4fb8-9314-434a65aeabe6-plugins-config-volume\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:33:06.427416 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:06.427422    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2clp9\" (UniqueName: \"kubernetes.io/projected/6216548e-093f-4c34-bb8a-01fe1ad91aa8-kube-api-access-2clp9\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:33:06.427758 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:06.427433    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwvwb\" (UniqueName: \"kubernetes.io/projected/f9114ccd-117b-4fb8-9314-434a65aeabe6-kube-api-access-pwvwb\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:33:06.706932 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:06.706898    5570 generic.go:356] "Generic (PLEG): container finished" podID="f9114ccd-117b-4fb8-9314-434a65aeabe6" containerID="596bfe0429d65cc33f5a9a1b08e2d9f0fa5b38e1da423abdc5f62b353dc81499" exitCode=0
Nov 03 20:33:06.707415 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:06.706956    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d05604-7799fccfc9-m4xqb"
Nov 03 20:33:06.707415 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:06.706976    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d05604-7799fccfc9-m4xqb" event={"ID":"f9114ccd-117b-4fb8-9314-434a65aeabe6","Type":"ContainerDied","Data":"596bfe0429d65cc33f5a9a1b08e2d9f0fa5b38e1da423abdc5f62b353dc81499"}
Nov 03 20:33:06.707415 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:06.707027    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d05604-7799fccfc9-m4xqb" event={"ID":"f9114ccd-117b-4fb8-9314-434a65aeabe6","Type":"ContainerDied","Data":"fcb8c1cc1a67e43785dcf1b2d2129218b5f68f8fda7cf3386774915c0f7eb728"}
Nov 03 20:33:06.707415 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:06.707054    5570 scope.go:122] "RemoveContainer" containerID="596bfe0429d65cc33f5a9a1b08e2d9f0fa5b38e1da423abdc5f62b353dc81499"
Nov 03 20:33:06.708628 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:06.708601    5570 generic.go:356] "Generic (PLEG): container finished" podID="6216548e-093f-4c34-bb8a-01fe1ad91aa8" containerID="26aad60548fb9ae9dc303f7839e88dc85f6e3efae94f71132f80941b40cc45ff" exitCode=0
Nov 03 20:33:06.708757 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:06.708643    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-d05604-5fb56668c4-24x95"
Nov 03 20:33:06.708757 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:06.708686    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-d05604-5fb56668c4-24x95" event={"ID":"6216548e-093f-4c34-bb8a-01fe1ad91aa8","Type":"ContainerDied","Data":"26aad60548fb9ae9dc303f7839e88dc85f6e3efae94f71132f80941b40cc45ff"}
Nov 03 20:33:06.708757 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:06.708743    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-d05604-5fb56668c4-24x95" event={"ID":"6216548e-093f-4c34-bb8a-01fe1ad91aa8","Type":"ContainerDied","Data":"4a9dab3bf88c4a0e6955db31b2485b4a977aea3e89c5659ceb8e60b51a203970"}
Nov 03 20:33:06.723362 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:06.723338    5570 scope.go:122] "RemoveContainer" containerID="596bfe0429d65cc33f5a9a1b08e2d9f0fa5b38e1da423abdc5f62b353dc81499"
Nov 03 20:33:06.724002 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:33:06.723733    5570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"596bfe0429d65cc33f5a9a1b08e2d9f0fa5b38e1da423abdc5f62b353dc81499\": container with ID starting with 596bfe0429d65cc33f5a9a1b08e2d9f0fa5b38e1da423abdc5f62b353dc81499 not found: ID does not exist" containerID="596bfe0429d65cc33f5a9a1b08e2d9f0fa5b38e1da423abdc5f62b353dc81499"
Nov 03 20:33:06.724002 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:06.723785    5570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"596bfe0429d65cc33f5a9a1b08e2d9f0fa5b38e1da423abdc5f62b353dc81499"} err="failed to get container status \"596bfe0429d65cc33f5a9a1b08e2d9f0fa5b38e1da423abdc5f62b353dc81499\": rpc error: code = NotFound desc = could not find container \"596bfe0429d65cc33f5a9a1b08e2d9f0fa5b38e1da423abdc5f62b353dc81499\": container with ID starting with 596bfe0429d65cc33f5a9a1b08e2d9f0fa5b38e1da423abdc5f62b353dc81499 not found: ID does not exist"
Nov 03 20:33:06.726056 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:06.726017    5570 scope.go:122] "RemoveContainer" containerID="26aad60548fb9ae9dc303f7839e88dc85f6e3efae94f71132f80941b40cc45ff"
Nov 03 20:33:06.818196 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:06.812967    5570 scope.go:122] "RemoveContainer" containerID="26aad60548fb9ae9dc303f7839e88dc85f6e3efae94f71132f80941b40cc45ff"
Nov 03 20:33:06.818196 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:06.813082    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d05604-7799fccfc9-m4xqb"]
Nov 03 20:33:06.818196 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:33:06.816464    5570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26aad60548fb9ae9dc303f7839e88dc85f6e3efae94f71132f80941b40cc45ff\": container with ID starting with 26aad60548fb9ae9dc303f7839e88dc85f6e3efae94f71132f80941b40cc45ff not found: ID does not exist" containerID="26aad60548fb9ae9dc303f7839e88dc85f6e3efae94f71132f80941b40cc45ff"
Nov 03 20:33:06.818196 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:06.816511    5570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26aad60548fb9ae9dc303f7839e88dc85f6e3efae94f71132f80941b40cc45ff"} err="failed to get container status \"26aad60548fb9ae9dc303f7839e88dc85f6e3efae94f71132f80941b40cc45ff\": rpc error: code = NotFound desc = could not find container \"26aad60548fb9ae9dc303f7839e88dc85f6e3efae94f71132f80941b40cc45ff\": container with ID starting with 26aad60548fb9ae9dc303f7839e88dc85f6e3efae94f71132f80941b40cc45ff not found: ID does not exist"
Nov 03 20:33:06.820040 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:06.818634    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d05604-7799fccfc9-m4xqb"]
Nov 03 20:33:06.838211 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:06.838133    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d05604-5fb56668c4-24x95"]
Nov 03 20:33:06.841358 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:06.841335    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-a9ba83-openshift-default-8c675fb59-486wc"
Nov 03 20:33:06.842104 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:06.842086    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d05604-5fb56668c4-24x95"]
Nov 03 20:33:07.718630 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:07.718596    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-a9ba83-openshift-default-8c675fb59-486wc"
Nov 03 20:33:07.719741 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:07.719724    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-a9ba83-openshift-default-8c675fb59-486wc"
Nov 03 20:33:07.966415 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:07.966379    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6216548e-093f-4c34-bb8a-01fe1ad91aa8" path="/var/lib/kubelet/pods/6216548e-093f-4c34-bb8a-01fe1ad91aa8/volumes"
Nov 03 20:33:07.966879 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:07.966864    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9114ccd-117b-4fb8-9314-434a65aeabe6" path="/var/lib/kubelet/pods/f9114ccd-117b-4fb8-9314-434a65aeabe6/volumes"
Nov 03 20:33:08.333677 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:08.333631    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-9e1bcb/distinct-2-w6-9e1bcb-kserve-7d96d9f6f6-4f7fl"]
Nov 03 20:33:08.344855 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:08.344824    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-9e1bcb/premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh"]
Nov 03 20:33:08.639440 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:08.639412    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d1aa66-openshift-default-8457c96657-lt2kw"
Nov 03 20:33:08.683023 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:08.675370    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a701695-c67d-4cda-a46d-7834b22750ed-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a701695-c67d-4cda-a46d-7834b22750ed-workload-certs\") pod \"5a701695-c67d-4cda-a46d-7834b22750ed\" (UID: \"5a701695-c67d-4cda-a46d-7834b22750ed\") "
Nov 03 20:33:08.683023 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:08.675503    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a701695-c67d-4cda-a46d-7834b22750ed-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a701695-c67d-4cda-a46d-7834b22750ed-credential-socket\") pod \"5a701695-c67d-4cda-a46d-7834b22750ed\" (UID: \"5a701695-c67d-4cda-a46d-7834b22750ed\") "
Nov 03 20:33:08.683023 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:08.675548    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a701695-c67d-4cda-a46d-7834b22750ed-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a701695-c67d-4cda-a46d-7834b22750ed-workload-socket\") pod \"5a701695-c67d-4cda-a46d-7834b22750ed\" (UID: \"5a701695-c67d-4cda-a46d-7834b22750ed\") "
Nov 03 20:33:08.683023 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:08.675677    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5a701695-c67d-4cda-a46d-7834b22750ed-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a701695-c67d-4cda-a46d-7834b22750ed-istio-podinfo\") pod \"5a701695-c67d-4cda-a46d-7834b22750ed\" (UID: \"5a701695-c67d-4cda-a46d-7834b22750ed\") "
Nov 03 20:33:08.683023 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:08.675787    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a701695-c67d-4cda-a46d-7834b22750ed-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a701695-c67d-4cda-a46d-7834b22750ed-istio-envoy\") pod \"5a701695-c67d-4cda-a46d-7834b22750ed\" (UID: \"5a701695-c67d-4cda-a46d-7834b22750ed\") "
Nov 03 20:33:08.683023 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:08.675844    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a701695-c67d-4cda-a46d-7834b22750ed-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a701695-c67d-4cda-a46d-7834b22750ed-istio-data\") pod \"5a701695-c67d-4cda-a46d-7834b22750ed\" (UID: \"5a701695-c67d-4cda-a46d-7834b22750ed\") "
Nov 03 20:33:08.683023 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:08.675927    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a701695-c67d-4cda-a46d-7834b22750ed-kube-api-access-bbcw7\" (UniqueName: \"kubernetes.io/projected/5a701695-c67d-4cda-a46d-7834b22750ed-kube-api-access-bbcw7\") pod \"5a701695-c67d-4cda-a46d-7834b22750ed\" (UID: \"5a701695-c67d-4cda-a46d-7834b22750ed\") "
Nov 03 20:33:08.723800 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:08.704521    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a701695-c67d-4cda-a46d-7834b22750ed-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a701695-c67d-4cda-a46d-7834b22750ed-istiod-ca-cert\") pod \"5a701695-c67d-4cda-a46d-7834b22750ed\" (UID: \"5a701695-c67d-4cda-a46d-7834b22750ed\") "
Nov 03 20:33:08.723800 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:08.704635    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a701695-c67d-4cda-a46d-7834b22750ed-istio-token\" (UniqueName: \"kubernetes.io/projected/5a701695-c67d-4cda-a46d-7834b22750ed-istio-token\") pod \"5a701695-c67d-4cda-a46d-7834b22750ed\" (UID: \"5a701695-c67d-4cda-a46d-7834b22750ed\") "
Nov 03 20:33:08.723800 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:08.706137    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a701695-c67d-4cda-a46d-7834b22750ed-workload-socket" pod "5a701695-c67d-4cda-a46d-7834b22750ed" (UID: "5a701695-c67d-4cda-a46d-7834b22750ed"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:33:08.723800 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:08.706359    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a701695-c67d-4cda-a46d-7834b22750ed-workload-certs" pod "5a701695-c67d-4cda-a46d-7834b22750ed" (UID: "5a701695-c67d-4cda-a46d-7834b22750ed"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:33:08.723800 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:08.709834    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a701695-c67d-4cda-a46d-7834b22750ed-istio-data" pod "5a701695-c67d-4cda-a46d-7834b22750ed" (UID: "5a701695-c67d-4cda-a46d-7834b22750ed"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:33:08.723800 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:08.710373    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a701695-c67d-4cda-a46d-7834b22750ed-istiod-ca-cert" pod "5a701695-c67d-4cda-a46d-7834b22750ed" (UID: "5a701695-c67d-4cda-a46d-7834b22750ed"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:33:08.723800 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:08.714891    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a701695-c67d-4cda-a46d-7834b22750ed-credential-socket" pod "5a701695-c67d-4cda-a46d-7834b22750ed" (UID: "5a701695-c67d-4cda-a46d-7834b22750ed"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:33:08.743804 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:08.743340    5570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a701695-c67d-4cda-a46d-7834b22750ed-istiod-ca-cert\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:33:08.743804 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:08.743382    5570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a701695-c67d-4cda-a46d-7834b22750ed-workload-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:33:08.743804 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:08.743397    5570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a701695-c67d-4cda-a46d-7834b22750ed-credential-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:33:08.743804 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:08.743411    5570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a701695-c67d-4cda-a46d-7834b22750ed-workload-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:33:08.743804 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:08.743425    5570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a701695-c67d-4cda-a46d-7834b22750ed-istio-data\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:33:08.755624 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:08.755579    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a701695-c67d-4cda-a46d-7834b22750ed-istio-envoy" pod "5a701695-c67d-4cda-a46d-7834b22750ed" (UID: "5a701695-c67d-4cda-a46d-7834b22750ed"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:33:08.762266 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:08.762232    5570 generic.go:356] "Generic (PLEG): container finished" podID="5a701695-c67d-4cda-a46d-7834b22750ed" containerID="7d6407f3a09e7889871e501625f155ce4ccdfbd5bf75f845f25ed94d35a2e75f" exitCode=0
Nov 03 20:33:08.762465 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:08.762362    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d1aa66-openshift-default-8457c96657-lt2kw"
Nov 03 20:33:08.762627 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:08.762595    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d1aa66-openshift-default-8457c96657-lt2kw" event={"ID":"5a701695-c67d-4cda-a46d-7834b22750ed","Type":"ContainerDied","Data":"7d6407f3a09e7889871e501625f155ce4ccdfbd5bf75f845f25ed94d35a2e75f"}
Nov 03 20:33:08.762715 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:08.762682    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d1aa66-openshift-default-8457c96657-lt2kw" event={"ID":"5a701695-c67d-4cda-a46d-7834b22750ed","Type":"ContainerDied","Data":"251e355a12f83f023b75ede730b776523c78acdb5d02bcb71d3e83a86b354ffe"}
Nov 03 20:33:08.762715 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:08.762711    5570 scope.go:122] "RemoveContainer" containerID="7d6407f3a09e7889871e501625f155ce4ccdfbd5bf75f845f25ed94d35a2e75f"
Nov 03 20:33:08.763654 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:08.763624    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5a701695-c67d-4cda-a46d-7834b22750ed-istio-podinfo" pod "5a701695-c67d-4cda-a46d-7834b22750ed" (UID: "5a701695-c67d-4cda-a46d-7834b22750ed"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:33:08.766025 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:08.765993    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a701695-c67d-4cda-a46d-7834b22750ed-kube-api-access-bbcw7" pod "5a701695-c67d-4cda-a46d-7834b22750ed" (UID: "5a701695-c67d-4cda-a46d-7834b22750ed"). InnerVolumeSpecName "kube-api-access-bbcw7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:33:08.768407 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:08.768378    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a701695-c67d-4cda-a46d-7834b22750ed-istio-token" pod "5a701695-c67d-4cda-a46d-7834b22750ed" (UID: "5a701695-c67d-4cda-a46d-7834b22750ed"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:33:08.782012 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:08.781986    5570 scope.go:122] "RemoveContainer" containerID="8aaebcb8af28523aa8fb9da47a8dd3c1e0d18b4d1ea123656dcb34aa2d36d88c"
Nov 03 20:33:08.801025 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:08.801000    5570 scope.go:122] "RemoveContainer" containerID="7d6407f3a09e7889871e501625f155ce4ccdfbd5bf75f845f25ed94d35a2e75f"
Nov 03 20:33:08.801539 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:33:08.801504    5570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d6407f3a09e7889871e501625f155ce4ccdfbd5bf75f845f25ed94d35a2e75f\": container with ID starting with 7d6407f3a09e7889871e501625f155ce4ccdfbd5bf75f845f25ed94d35a2e75f not found: ID does not exist" containerID="7d6407f3a09e7889871e501625f155ce4ccdfbd5bf75f845f25ed94d35a2e75f"
Nov 03 20:33:08.801656 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:08.801550    5570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d6407f3a09e7889871e501625f155ce4ccdfbd5bf75f845f25ed94d35a2e75f"} err="failed to get container status \"7d6407f3a09e7889871e501625f155ce4ccdfbd5bf75f845f25ed94d35a2e75f\": rpc error: code = NotFound desc = could not find container \"7d6407f3a09e7889871e501625f155ce4ccdfbd5bf75f845f25ed94d35a2e75f\": container with ID starting with 7d6407f3a09e7889871e501625f155ce4ccdfbd5bf75f845f25ed94d35a2e75f not found: ID does not exist"
Nov 03 20:33:08.801656 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:08.801578    5570 scope.go:122] "RemoveContainer" containerID="8aaebcb8af28523aa8fb9da47a8dd3c1e0d18b4d1ea123656dcb34aa2d36d88c"
Nov 03 20:33:08.801968 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:33:08.801943    5570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8aaebcb8af28523aa8fb9da47a8dd3c1e0d18b4d1ea123656dcb34aa2d36d88c\": container with ID starting with 8aaebcb8af28523aa8fb9da47a8dd3c1e0d18b4d1ea123656dcb34aa2d36d88c not found: ID does not exist" containerID="8aaebcb8af28523aa8fb9da47a8dd3c1e0d18b4d1ea123656dcb34aa2d36d88c"
Nov 03 20:33:08.802069 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:08.801976    5570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8aaebcb8af28523aa8fb9da47a8dd3c1e0d18b4d1ea123656dcb34aa2d36d88c"} err="failed to get container status \"8aaebcb8af28523aa8fb9da47a8dd3c1e0d18b4d1ea123656dcb34aa2d36d88c\": rpc error: code = NotFound desc = could not find container \"8aaebcb8af28523aa8fb9da47a8dd3c1e0d18b4d1ea123656dcb34aa2d36d88c\": container with ID starting with 8aaebcb8af28523aa8fb9da47a8dd3c1e0d18b4d1ea123656dcb34aa2d36d88c not found: ID does not exist"
Nov 03 20:33:08.845408 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:08.845369    5570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a701695-c67d-4cda-a46d-7834b22750ed-istio-podinfo\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:33:08.845408 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:08.845400    5570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a701695-c67d-4cda-a46d-7834b22750ed-istio-envoy\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:33:08.845408 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:08.845409    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bbcw7\" (UniqueName: \"kubernetes.io/projected/5a701695-c67d-4cda-a46d-7834b22750ed-kube-api-access-bbcw7\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:33:08.845408 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:08.845420    5570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a701695-c67d-4cda-a46d-7834b22750ed-istio-token\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:33:09.114810 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:09.114734    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-d1aa66-openshift-default-8457c96657-lt2kw"]
Nov 03 20:33:09.118184 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:09.118155    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-d1aa66-openshift-default-8457c96657-lt2kw"]
Nov 03 20:33:09.969199 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:09.969166    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a701695-c67d-4cda-a46d-7834b22750ed" path="/var/lib/kubelet/pods/5a701695-c67d-4cda-a46d-7834b22750ed/volumes"
Nov 03 20:33:10.203747 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:10.203713    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a9ba83-67d4dd56f9-22nhs"]
Nov 03 20:33:10.204438 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:10.204421    5570 cpu_manager.go:414] "containerMap: removing container" podUID="f9114ccd-117b-4fb8-9314-434a65aeabe6" containerName="payload-pre-processing"
Nov 03 20:33:10.204486 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:10.204439    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9114ccd-117b-4fb8-9314-434a65aeabe6" containerName="payload-pre-processing"
Nov 03 20:33:10.204486 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:10.204474    5570 cpu_manager.go:414] "containerMap: removing container" podUID="5a701695-c67d-4cda-a46d-7834b22750ed" containerName="istio-proxy"
Nov 03 20:33:10.204486 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:10.204479    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a701695-c67d-4cda-a46d-7834b22750ed" containerName="istio-proxy"
Nov 03 20:33:10.204583 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:10.204496    5570 cpu_manager.go:414] "containerMap: removing container" podUID="6216548e-093f-4c34-bb8a-01fe1ad91aa8" containerName="payload-processing"
Nov 03 20:33:10.204583 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:10.204502    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6216548e-093f-4c34-bb8a-01fe1ad91aa8" containerName="payload-processing"
Nov 03 20:33:10.204583 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:10.204512    5570 cpu_manager.go:414] "containerMap: removing container" podUID="5a701695-c67d-4cda-a46d-7834b22750ed" containerName="istio-proxy"
Nov 03 20:33:10.204583 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:10.204517    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a701695-c67d-4cda-a46d-7834b22750ed" containerName="istio-proxy"
Nov 03 20:33:10.204583 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:10.204528    5570 cpu_manager.go:414] "containerMap: removing container" podUID="46af9c86-b058-4323-97a2-af8d89d3dc8f" containerName="istio-proxy"
Nov 03 20:33:10.204583 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:10.204533    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46af9c86-b058-4323-97a2-af8d89d3dc8f" containerName="istio-proxy"
Nov 03 20:33:10.204763 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:10.204632    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a701695-c67d-4cda-a46d-7834b22750ed" containerName="istio-proxy"
Nov 03 20:33:10.204763 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:10.204641    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="6216548e-093f-4c34-bb8a-01fe1ad91aa8" containerName="payload-processing"
Nov 03 20:33:10.204763 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:10.204648    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a701695-c67d-4cda-a46d-7834b22750ed" containerName="istio-proxy"
Nov 03 20:33:10.204763 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:10.204653    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="46af9c86-b058-4323-97a2-af8d89d3dc8f" containerName="istio-proxy"
Nov 03 20:33:10.204763 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:10.204664    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9114ccd-117b-4fb8-9314-434a65aeabe6" containerName="payload-pre-processing"
Nov 03 20:33:10.209577 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:10.209552    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a9ba83-67d4dd56f9-22nhs"
Nov 03 20:33:10.229995 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:10.229922    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a9ba83-67d4dd56f9-22nhs"]
Nov 03 20:33:10.311798 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:10.300174    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-a9ba83-dockercfg-726p6\""
Nov 03 20:33:10.396799 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:10.340060    5570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-a9ba83\""
Nov 03 20:33:10.396799 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:10.374840    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7ldl\" (UniqueName: \"kubernetes.io/projected/58fc81a1-5df4-4b6c-92bc-6dcf0836f400-kube-api-access-c7ldl\") pod \"payload-pre-processing-e2e-inf-a-a9ba83-67d4dd56f9-22nhs\" (UID: \"58fc81a1-5df4-4b6c-92bc-6dcf0836f400\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a9ba83-67d4dd56f9-22nhs"
Nov 03 20:33:10.396799 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:10.374953    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58fc81a1-5df4-4b6c-92bc-6dcf0836f400-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-a9ba83-67d4dd56f9-22nhs\" (UID: \"58fc81a1-5df4-4b6c-92bc-6dcf0836f400\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a9ba83-67d4dd56f9-22nhs"
Nov 03 20:33:10.442948 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:10.442876    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a9ba83-67b98df76d-24dtv"]
Nov 03 20:33:10.465076 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:10.465038    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a9ba83-67b98df76d-24dtv"]
Nov 03 20:33:10.465465 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:10.465439    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a9ba83-67b98df76d-24dtv"
Nov 03 20:33:10.487720 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:10.484700    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58fc81a1-5df4-4b6c-92bc-6dcf0836f400-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-a9ba83-67d4dd56f9-22nhs\" (UID: \"58fc81a1-5df4-4b6c-92bc-6dcf0836f400\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a9ba83-67d4dd56f9-22nhs"
Nov 03 20:33:10.487720 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:10.484762    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gt8w\" (UniqueName: \"kubernetes.io/projected/6deae9d9-81da-4c77-a15a-d805ff547c51-kube-api-access-9gt8w\") pod \"payload-processing-e2e-inf-a-a9ba83-67b98df76d-24dtv\" (UID: \"6deae9d9-81da-4c77-a15a-d805ff547c51\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a9ba83-67b98df76d-24dtv"
Nov 03 20:33:10.487720 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:10.485164    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6deae9d9-81da-4c77-a15a-d805ff547c51-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-a9ba83-67b98df76d-24dtv\" (UID: \"6deae9d9-81da-4c77-a15a-d805ff547c51\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a9ba83-67b98df76d-24dtv"
Nov 03 20:33:10.487720 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:10.485214    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7ldl\" (UniqueName: \"kubernetes.io/projected/58fc81a1-5df4-4b6c-92bc-6dcf0836f400-kube-api-access-c7ldl\") pod \"payload-pre-processing-e2e-inf-a-a9ba83-67d4dd56f9-22nhs\" (UID: \"58fc81a1-5df4-4b6c-92bc-6dcf0836f400\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a9ba83-67d4dd56f9-22nhs"
Nov 03 20:33:10.487720 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:10.486593    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58fc81a1-5df4-4b6c-92bc-6dcf0836f400-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-a9ba83-67d4dd56f9-22nhs\" (UID: \"58fc81a1-5df4-4b6c-92bc-6dcf0836f400\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a9ba83-67d4dd56f9-22nhs"
Nov 03 20:33:10.527413 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:10.527374    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7ldl\" (UniqueName: \"kubernetes.io/projected/58fc81a1-5df4-4b6c-92bc-6dcf0836f400-kube-api-access-c7ldl\") pod \"payload-pre-processing-e2e-inf-a-a9ba83-67d4dd56f9-22nhs\" (UID: \"58fc81a1-5df4-4b6c-92bc-6dcf0836f400\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a9ba83-67d4dd56f9-22nhs"
Nov 03 20:33:10.590298 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:10.590229    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6deae9d9-81da-4c77-a15a-d805ff547c51-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-a9ba83-67b98df76d-24dtv\" (UID: \"6deae9d9-81da-4c77-a15a-d805ff547c51\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a9ba83-67b98df76d-24dtv"
Nov 03 20:33:10.590513 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:10.590367    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gt8w\" (UniqueName: \"kubernetes.io/projected/6deae9d9-81da-4c77-a15a-d805ff547c51-kube-api-access-9gt8w\") pod \"payload-processing-e2e-inf-a-a9ba83-67b98df76d-24dtv\" (UID: \"6deae9d9-81da-4c77-a15a-d805ff547c51\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a9ba83-67b98df76d-24dtv"
Nov 03 20:33:10.592669 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:10.592631    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6deae9d9-81da-4c77-a15a-d805ff547c51-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-a9ba83-67b98df76d-24dtv\" (UID: \"6deae9d9-81da-4c77-a15a-d805ff547c51\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a9ba83-67b98df76d-24dtv"
Nov 03 20:33:10.615456 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:10.615417    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gt8w\" (UniqueName: \"kubernetes.io/projected/6deae9d9-81da-4c77-a15a-d805ff547c51-kube-api-access-9gt8w\") pod \"payload-processing-e2e-inf-a-a9ba83-67b98df76d-24dtv\" (UID: \"6deae9d9-81da-4c77-a15a-d805ff547c51\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a9ba83-67b98df76d-24dtv"
Nov 03 20:33:10.826867 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:10.826744    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a9ba83-67d4dd56f9-22nhs"
Nov 03 20:33:10.828924 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:10.828893    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a9ba83-67b98df76d-24dtv"
Nov 03 20:33:11.170860 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:11.146587    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g69f8"]
Nov 03 20:33:11.238128 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:11.222703    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a9ba83-67b98df76d-24dtv"]
Nov 03 20:33:11.297210 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:11.284713    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a9ba83-67d4dd56f9-22nhs"]
Nov 03 20:33:11.308106 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:33:11.301911    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6deae9d9_81da_4c77_a15a_d805ff547c51.slice/crio-5c91b8e9b7e57c1615d4f2f87bda622e1e2a016b0ddbd9ebb2bd02e11f5ac1e6 WatchSource:0}: Error finding container 5c91b8e9b7e57c1615d4f2f87bda622e1e2a016b0ddbd9ebb2bd02e11f5ac1e6: Status 404 returned error can't find the container with id 5c91b8e9b7e57c1615d4f2f87bda622e1e2a016b0ddbd9ebb2bd02e11f5ac1e6
Nov 03 20:33:11.325352 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:33:11.325281    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58fc81a1_5df4_4b6c_92bc_6dcf0836f400.slice/crio-c410110c2c215ea40e1c1a39095a6cc6def8d7da33eb32acff325a545d3ea3b3 WatchSource:0}: Error finding container c410110c2c215ea40e1c1a39095a6cc6def8d7da33eb32acff325a545d3ea3b3: Status 404 returned error can't find the container with id c410110c2c215ea40e1c1a39095a6cc6def8d7da33eb32acff325a545d3ea3b3
Nov 03 20:33:11.880195 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:11.880145    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a9ba83-67b98df76d-24dtv" event={"ID":"6deae9d9-81da-4c77-a15a-d805ff547c51","Type":"ContainerStarted","Data":"bb882318962c16fb856a808f831b224d05ff43f5de4c53413f46980a3b2c636c"}
Nov 03 20:33:11.904275 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:11.900683    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a9ba83-67b98df76d-24dtv" event={"ID":"6deae9d9-81da-4c77-a15a-d805ff547c51","Type":"ContainerStarted","Data":"5c91b8e9b7e57c1615d4f2f87bda622e1e2a016b0ddbd9ebb2bd02e11f5ac1e6"}
Nov 03 20:33:11.908929 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:11.908895    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-a9ba83-67b98df76d-24dtv"
Nov 03 20:33:11.922514 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:11.922446    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a9ba83-67d4dd56f9-22nhs" event={"ID":"58fc81a1-5df4-4b6c-92bc-6dcf0836f400","Type":"ContainerStarted","Data":"922ea1a73f9216f6a93ff2cd116a56b6f399719095090b582b4199a796ce54f5"}
Nov 03 20:33:11.922791 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:11.922747    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a9ba83-67d4dd56f9-22nhs" event={"ID":"58fc81a1-5df4-4b6c-92bc-6dcf0836f400","Type":"ContainerStarted","Data":"c410110c2c215ea40e1c1a39095a6cc6def8d7da33eb32acff325a545d3ea3b3"}
Nov 03 20:33:11.924936 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:11.924906    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a9ba83-67d4dd56f9-22nhs"
Nov 03 20:33:11.931212 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:11.931153    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-a9ba83-67b98df76d-24dtv" podStartSLOduration=1.9311348050000001 podStartE2EDuration="1.931134805s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:33:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:33:11.927817907 +0000 UTC m=+1726.080479794" watchObservedRunningTime="2026-11-03 20:33:11.931134805 +0000 UTC m=+1726.083796691"
Nov 03 20:33:11.980067 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:11.979997    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a9ba83-67d4dd56f9-22nhs" podStartSLOduration=1.979977944 podStartE2EDuration="1.979977944s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:33:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:33:11.978168125 +0000 UTC m=+1726.130830010" watchObservedRunningTime="2026-11-03 20:33:11.979977944 +0000 UTC m=+1726.132639832"
Nov 03 20:33:19.660345 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:19.660304    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-7e4389-openshift-default-5b5bd7f55-kpk7n"]
Nov 03 20:33:19.660766 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:19.660540    5570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-7e4389-openshift-default-5b5bd7f55-kpk7n" podUID="1f5aeb82-01f6-483f-8273-10bd0714e03c" containerName="istio-proxy" containerID="cri-o://b24a6ddb0a915a87947f09e2e5141c213ffbeec08769fa4c0a030faf8f68be69" gracePeriod=30
Nov 03 20:33:22.439805 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:22.425645    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-g262x"]
Nov 03 20:33:22.439805 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:22.430067    5570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-g262x" podUID="8aa4c48e-92c6-49c8-acd3-4bc441ac8ca7" containerName="istio-proxy" containerID="cri-o://83026a35e395a00762993ad189f351188a419bba8a38d57433b5dae409cf0502" gracePeriod=30
Nov 03 20:33:22.453315 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:22.453270    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-rbvqw"]
Nov 03 20:33:22.453872 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:22.453818    5570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-rbvqw" podUID="ae5a764e-25f7-492e-b4aa-8f2696e22f37" containerName="istio-proxy" containerID="cri-o://d1909a45904a0d779c9eee7a33e1d1b228fb1289b0a14179aaa2f840cae28632" gracePeriod=30
Nov 03 20:33:22.454588 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:22.454560    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-jshb8"]
Nov 03 20:33:22.454840 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:22.454797    5570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-jshb8" podUID="2137488e-8e6c-46f7-a0ed-b4721d4f2645" containerName="istio-proxy" containerID="cri-o://e5c7ec1d14a857d58b25629abd68ececf4e69dc30db9346800e1c54320f24671" gracePeriod=30
Nov 03 20:33:22.465677 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:22.465637    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-j7dz6"]
Nov 03 20:33:22.465951 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:22.465920    5570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-j7dz6" podUID="22ea3ab0-6acc-4558-9158-eaa289924f35" containerName="istio-proxy" containerID="cri-o://c35d6c2c4780ada7348150b2cfc4cd1948d3f225714d8b58219a774af050416b" gracePeriod=30
Nov 03 20:33:23.144300 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:23.143560    5570 patch_prober.go:29] interesting pod/e2e-inf-b-1d159a-openshift-default-799c8f9695-b8h67 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": read tcp 10.134.0.2:35148->10.134.0.161:15021: read: connection reset by peer" start-of-body=
Nov 03 20:33:23.144300 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:23.143637    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-1d159a-openshift-default-799c8f9695-b8h67" podUID="40e4fc9c-c3c1-4aa0-98ad-eda347be89c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": read tcp 10.134.0.2:35148->10.134.0.161:15021: read: connection reset by peer"
Nov 03 20:33:23.357987 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:23.357933    5570 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-2-w6-9e1bcb-kserve-7d96d9f6f6-4f7fl" podUID="56353b0f-08ee-4ada-9dc0-ac5afd5a59c7" containerName="main" containerID="cri-o://29dc9938d203f96fc96a132d6cbe946dd3dce03c4807b8721a8ff8f7baea05af" gracePeriod=45
Nov 03 20:33:23.368360 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:23.368289    5570 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-9e1bcb/premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh" podUID="2d3647f2-89e9-4908-a8ee-d84e527abeb0" containerName="main" containerID="cri-o://e5774781017bffdb309abab3f3038fc990f221a8bb660c664b236d0529e6dce2" gracePeriod=45
Nov 03 20:33:23.683281 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:23.683250    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-2-w6-9e1bcb-kserve-7d96d9f6f6-4f7fl"
Nov 03 20:33:23.702056 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:23.702019    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-9e1bcb/premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh"
Nov 03 20:33:23.860685 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:23.858432    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56353b0f-08ee-4ada-9dc0-ac5afd5a59c7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56353b0f-08ee-4ada-9dc0-ac5afd5a59c7-tmp-dir\") pod \"56353b0f-08ee-4ada-9dc0-ac5afd5a59c7\" (UID: \"56353b0f-08ee-4ada-9dc0-ac5afd5a59c7\") "
Nov 03 20:33:23.860685 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:23.858675    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d3647f2-89e9-4908-a8ee-d84e527abeb0-tls-certs\" (UniqueName: \"kubernetes.io/secret/2d3647f2-89e9-4908-a8ee-d84e527abeb0-tls-certs\") pod \"2d3647f2-89e9-4908-a8ee-d84e527abeb0\" (UID: \"2d3647f2-89e9-4908-a8ee-d84e527abeb0\") "
Nov 03 20:33:23.860685 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:23.858735    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d3647f2-89e9-4908-a8ee-d84e527abeb0-kube-api-access-8r4mh\" (UniqueName: \"kubernetes.io/projected/2d3647f2-89e9-4908-a8ee-d84e527abeb0-kube-api-access-8r4mh\") pod \"2d3647f2-89e9-4908-a8ee-d84e527abeb0\" (UID: \"2d3647f2-89e9-4908-a8ee-d84e527abeb0\") "
Nov 03 20:33:23.860685 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:23.858795    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56353b0f-08ee-4ada-9dc0-ac5afd5a59c7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/56353b0f-08ee-4ada-9dc0-ac5afd5a59c7-dshm\") pod \"56353b0f-08ee-4ada-9dc0-ac5afd5a59c7\" (UID: \"56353b0f-08ee-4ada-9dc0-ac5afd5a59c7\") "
Nov 03 20:33:23.860685 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:23.858833    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d3647f2-89e9-4908-a8ee-d84e527abeb0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d3647f2-89e9-4908-a8ee-d84e527abeb0-dshm\") pod \"2d3647f2-89e9-4908-a8ee-d84e527abeb0\" (UID: \"2d3647f2-89e9-4908-a8ee-d84e527abeb0\") "
Nov 03 20:33:23.860685 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:23.858907    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56353b0f-08ee-4ada-9dc0-ac5afd5a59c7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56353b0f-08ee-4ada-9dc0-ac5afd5a59c7-model-cache\") pod \"56353b0f-08ee-4ada-9dc0-ac5afd5a59c7\" (UID: \"56353b0f-08ee-4ada-9dc0-ac5afd5a59c7\") "
Nov 03 20:33:23.860685 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:23.858953    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d3647f2-89e9-4908-a8ee-d84e527abeb0-home\" (UniqueName: \"kubernetes.io/empty-dir/2d3647f2-89e9-4908-a8ee-d84e527abeb0-home\") pod \"2d3647f2-89e9-4908-a8ee-d84e527abeb0\" (UID: \"2d3647f2-89e9-4908-a8ee-d84e527abeb0\") "
Nov 03 20:33:23.860685 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:23.858984    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56353b0f-08ee-4ada-9dc0-ac5afd5a59c7-home\" (UniqueName: \"kubernetes.io/empty-dir/56353b0f-08ee-4ada-9dc0-ac5afd5a59c7-home\") pod \"56353b0f-08ee-4ada-9dc0-ac5afd5a59c7\" (UID: \"56353b0f-08ee-4ada-9dc0-ac5afd5a59c7\") "
Nov 03 20:33:23.860685 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:23.859129    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/56353b0f-08ee-4ada-9dc0-ac5afd5a59c7-tls-certs\" (UniqueName: \"kubernetes.io/secret/56353b0f-08ee-4ada-9dc0-ac5afd5a59c7-tls-certs\") pod \"56353b0f-08ee-4ada-9dc0-ac5afd5a59c7\" (UID: \"56353b0f-08ee-4ada-9dc0-ac5afd5a59c7\") "
Nov 03 20:33:23.860685 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:23.859191    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56353b0f-08ee-4ada-9dc0-ac5afd5a59c7-kube-api-access-pg4w4\" (UniqueName: \"kubernetes.io/projected/56353b0f-08ee-4ada-9dc0-ac5afd5a59c7-kube-api-access-pg4w4\") pod \"56353b0f-08ee-4ada-9dc0-ac5afd5a59c7\" (UID: \"56353b0f-08ee-4ada-9dc0-ac5afd5a59c7\") "
Nov 03 20:33:23.860685 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:23.859231    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d3647f2-89e9-4908-a8ee-d84e527abeb0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d3647f2-89e9-4908-a8ee-d84e527abeb0-tmp-dir\") pod \"2d3647f2-89e9-4908-a8ee-d84e527abeb0\" (UID: \"2d3647f2-89e9-4908-a8ee-d84e527abeb0\") "
Nov 03 20:33:23.860685 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:23.859293    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d3647f2-89e9-4908-a8ee-d84e527abeb0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d3647f2-89e9-4908-a8ee-d84e527abeb0-model-cache\") pod \"2d3647f2-89e9-4908-a8ee-d84e527abeb0\" (UID: \"2d3647f2-89e9-4908-a8ee-d84e527abeb0\") "
Nov 03 20:33:23.860685 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:23.860188    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56353b0f-08ee-4ada-9dc0-ac5afd5a59c7-tmp-dir" pod "56353b0f-08ee-4ada-9dc0-ac5afd5a59c7" (UID: "56353b0f-08ee-4ada-9dc0-ac5afd5a59c7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:33:23.860685 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:23.860505    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d3647f2-89e9-4908-a8ee-d84e527abeb0-home" pod "2d3647f2-89e9-4908-a8ee-d84e527abeb0" (UID: "2d3647f2-89e9-4908-a8ee-d84e527abeb0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:33:23.862024 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:23.861210    5570 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56353b0f-08ee-4ada-9dc0-ac5afd5a59c7-tmp-dir\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:33:23.862024 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:23.861231    5570 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d3647f2-89e9-4908-a8ee-d84e527abeb0-home\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:33:23.862447 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:23.862426    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56353b0f-08ee-4ada-9dc0-ac5afd5a59c7-home" pod "56353b0f-08ee-4ada-9dc0-ac5afd5a59c7" (UID: "56353b0f-08ee-4ada-9dc0-ac5afd5a59c7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:33:23.862581 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:23.862509    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d3647f2-89e9-4908-a8ee-d84e527abeb0-tls-certs" pod "2d3647f2-89e9-4908-a8ee-d84e527abeb0" (UID: "2d3647f2-89e9-4908-a8ee-d84e527abeb0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:33:23.862845 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:23.862791    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d3647f2-89e9-4908-a8ee-d84e527abeb0-tmp-dir" pod "2d3647f2-89e9-4908-a8ee-d84e527abeb0" (UID: "2d3647f2-89e9-4908-a8ee-d84e527abeb0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:33:23.862969 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:23.862941    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-60efdf/test-model-60efdf-kserve-7fb9cb65bf-9774z"]
Nov 03 20:33:23.863233 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:23.863209    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d3647f2-89e9-4908-a8ee-d84e527abeb0-model-cache" pod "2d3647f2-89e9-4908-a8ee-d84e527abeb0" (UID: "2d3647f2-89e9-4908-a8ee-d84e527abeb0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:33:23.863315 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:23.863213    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56353b0f-08ee-4ada-9dc0-ac5afd5a59c7-model-cache" pod "56353b0f-08ee-4ada-9dc0-ac5afd5a59c7" (UID: "56353b0f-08ee-4ada-9dc0-ac5afd5a59c7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:33:23.863960 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:23.863831    5570 cpu_manager.go:414] "containerMap: removing container" podUID="2d3647f2-89e9-4908-a8ee-d84e527abeb0" containerName="main"
Nov 03 20:33:23.863960 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:23.863860    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d3647f2-89e9-4908-a8ee-d84e527abeb0" containerName="main"
Nov 03 20:33:23.863960 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:23.863959    5570 cpu_manager.go:414] "containerMap: removing container" podUID="56353b0f-08ee-4ada-9dc0-ac5afd5a59c7" containerName="main"
Nov 03 20:33:23.864525 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:23.863968    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56353b0f-08ee-4ada-9dc0-ac5afd5a59c7" containerName="main"
Nov 03 20:33:23.864525 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:23.864309    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d3647f2-89e9-4908-a8ee-d84e527abeb0" containerName="main"
Nov 03 20:33:23.864525 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:23.864367    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="56353b0f-08ee-4ada-9dc0-ac5afd5a59c7" containerName="main"
Nov 03 20:33:23.867639 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:23.867611    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d3647f2-89e9-4908-a8ee-d84e527abeb0-dshm" pod "2d3647f2-89e9-4908-a8ee-d84e527abeb0" (UID: "2d3647f2-89e9-4908-a8ee-d84e527abeb0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:33:23.867639 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:23.867621    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56353b0f-08ee-4ada-9dc0-ac5afd5a59c7-kube-api-access-pg4w4" pod "56353b0f-08ee-4ada-9dc0-ac5afd5a59c7" (UID: "56353b0f-08ee-4ada-9dc0-ac5afd5a59c7"). InnerVolumeSpecName "kube-api-access-pg4w4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:33:23.868001 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:23.867972    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d3647f2-89e9-4908-a8ee-d84e527abeb0-kube-api-access-8r4mh" pod "2d3647f2-89e9-4908-a8ee-d84e527abeb0" (UID: "2d3647f2-89e9-4908-a8ee-d84e527abeb0"). InnerVolumeSpecName "kube-api-access-8r4mh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:33:23.868086 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:23.868068    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56353b0f-08ee-4ada-9dc0-ac5afd5a59c7-tls-certs" pod "56353b0f-08ee-4ada-9dc0-ac5afd5a59c7" (UID: "56353b0f-08ee-4ada-9dc0-ac5afd5a59c7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:33:23.869707 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:23.869688    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-60efdf/test-model-60efdf-kserve-7fb9cb65bf-9774z"
Nov 03 20:33:23.869840 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:23.869815    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56353b0f-08ee-4ada-9dc0-ac5afd5a59c7-dshm" pod "56353b0f-08ee-4ada-9dc0-ac5afd5a59c7" (UID: "56353b0f-08ee-4ada-9dc0-ac5afd5a59c7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:33:23.881476 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:23.881442    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-60efdf/test-model-60efdf-kserve-7fb9cb65bf-9774z"]
Nov 03 20:33:23.889122 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:23.889075    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-60efdf\"/\"test-model-60efdf-kserve-self-signed-certs\""
Nov 03 20:33:23.889283 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:23.889245    5570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-60efdf\"/\"openshift-service-ca.crt\""
Nov 03 20:33:23.889357 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:23.889347    5570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-60efdf\"/\"kube-root-ca.crt\""
Nov 03 20:33:23.889442 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:23.889426    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-60efdf\"/\"default-dockercfg-pkxw7\""
Nov 03 20:33:23.965357 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:23.963282    5570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d3647f2-89e9-4908-a8ee-d84e527abeb0-tls-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:33:23.965357 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:23.963336    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8r4mh\" (UniqueName: \"kubernetes.io/projected/2d3647f2-89e9-4908-a8ee-d84e527abeb0-kube-api-access-8r4mh\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:33:23.965357 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:23.963352    5570 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56353b0f-08ee-4ada-9dc0-ac5afd5a59c7-dshm\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:33:23.965357 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:23.963365    5570 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d3647f2-89e9-4908-a8ee-d84e527abeb0-dshm\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:33:23.965357 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:23.963377    5570 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56353b0f-08ee-4ada-9dc0-ac5afd5a59c7-model-cache\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:33:23.965357 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:23.963389    5570 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56353b0f-08ee-4ada-9dc0-ac5afd5a59c7-home\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:33:23.965357 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:23.963417    5570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56353b0f-08ee-4ada-9dc0-ac5afd5a59c7-tls-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:33:23.965357 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:23.963428    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pg4w4\" (UniqueName: \"kubernetes.io/projected/56353b0f-08ee-4ada-9dc0-ac5afd5a59c7-kube-api-access-pg4w4\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:33:23.965357 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:23.963439    5570 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d3647f2-89e9-4908-a8ee-d84e527abeb0-tmp-dir\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:33:23.965357 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:23.963451    5570 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d3647f2-89e9-4908-a8ee-d84e527abeb0-model-cache\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:33:23.977599 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:23.977571    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-a9ba83-67b98df76d-24dtv"
Nov 03 20:33:23.977802 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:23.977651    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a9ba83-67d4dd56f9-22nhs"
Nov 03 20:33:24.065195 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:24.065163    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/543a8a92-e2b1-4d88-9b3f-4473806a9a3d-home\") pod \"test-model-60efdf-kserve-7fb9cb65bf-9774z\" (UID: \"543a8a92-e2b1-4d88-9b3f-4473806a9a3d\") " pod="ai-tenant-e2e-inf-a-60efdf/test-model-60efdf-kserve-7fb9cb65bf-9774z"
Nov 03 20:33:24.065350 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:24.065216    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/543a8a92-e2b1-4d88-9b3f-4473806a9a3d-dshm\") pod \"test-model-60efdf-kserve-7fb9cb65bf-9774z\" (UID: \"543a8a92-e2b1-4d88-9b3f-4473806a9a3d\") " pod="ai-tenant-e2e-inf-a-60efdf/test-model-60efdf-kserve-7fb9cb65bf-9774z"
Nov 03 20:33:24.065418 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:24.065343    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/543a8a92-e2b1-4d88-9b3f-4473806a9a3d-tls-certs\") pod \"test-model-60efdf-kserve-7fb9cb65bf-9774z\" (UID: \"543a8a92-e2b1-4d88-9b3f-4473806a9a3d\") " pod="ai-tenant-e2e-inf-a-60efdf/test-model-60efdf-kserve-7fb9cb65bf-9774z"
Nov 03 20:33:24.065552 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:24.065531    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5h78z\" (UniqueName: \"kubernetes.io/projected/543a8a92-e2b1-4d88-9b3f-4473806a9a3d-kube-api-access-5h78z\") pod \"test-model-60efdf-kserve-7fb9cb65bf-9774z\" (UID: \"543a8a92-e2b1-4d88-9b3f-4473806a9a3d\") " pod="ai-tenant-e2e-inf-a-60efdf/test-model-60efdf-kserve-7fb9cb65bf-9774z"
Nov 03 20:33:24.065810 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:24.065588    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/543a8a92-e2b1-4d88-9b3f-4473806a9a3d-tmp-dir\") pod \"test-model-60efdf-kserve-7fb9cb65bf-9774z\" (UID: \"543a8a92-e2b1-4d88-9b3f-4473806a9a3d\") " pod="ai-tenant-e2e-inf-a-60efdf/test-model-60efdf-kserve-7fb9cb65bf-9774z"
Nov 03 20:33:24.065810 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:24.065691    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/543a8a92-e2b1-4d88-9b3f-4473806a9a3d-model-cache\") pod \"test-model-60efdf-kserve-7fb9cb65bf-9774z\" (UID: \"543a8a92-e2b1-4d88-9b3f-4473806a9a3d\") " pod="ai-tenant-e2e-inf-a-60efdf/test-model-60efdf-kserve-7fb9cb65bf-9774z"
Nov 03 20:33:24.081466 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:24.081403    5570 generic.go:356] "Generic (PLEG): container finished" podID="2d3647f2-89e9-4908-a8ee-d84e527abeb0" containerID="e5774781017bffdb309abab3f3038fc990f221a8bb660c664b236d0529e6dce2" exitCode=0
Nov 03 20:33:24.081466 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:24.081460    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-9e1bcb/premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh"
Nov 03 20:33:24.081746 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:24.081505    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-9e1bcb/premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh" event={"ID":"2d3647f2-89e9-4908-a8ee-d84e527abeb0","Type":"ContainerDied","Data":"e5774781017bffdb309abab3f3038fc990f221a8bb660c664b236d0529e6dce2"}
Nov 03 20:33:24.081746 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:24.081575    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-9e1bcb/premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh" event={"ID":"2d3647f2-89e9-4908-a8ee-d84e527abeb0","Type":"ContainerDied","Data":"e89a95af30ee334c8c6ef4ce4007d0ca1a4b4a9d9aa23c8971cc51b2edaa742b"}
Nov 03 20:33:24.081746 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:24.081601    5570 scope.go:122] "RemoveContainer" containerID="e5774781017bffdb309abab3f3038fc990f221a8bb660c664b236d0529e6dce2"
Nov 03 20:33:24.085736 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:24.085702    5570 generic.go:356] "Generic (PLEG): container finished" podID="56353b0f-08ee-4ada-9dc0-ac5afd5a59c7" containerID="29dc9938d203f96fc96a132d6cbe946dd3dce03c4807b8721a8ff8f7baea05af" exitCode=0
Nov 03 20:33:24.085954 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:24.085928    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-2-w6-9e1bcb-kserve-7d96d9f6f6-4f7fl" event={"ID":"56353b0f-08ee-4ada-9dc0-ac5afd5a59c7","Type":"ContainerDied","Data":"29dc9938d203f96fc96a132d6cbe946dd3dce03c4807b8721a8ff8f7baea05af"}
Nov 03 20:33:24.086039 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:24.085974    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-2-w6-9e1bcb-kserve-7d96d9f6f6-4f7fl" event={"ID":"56353b0f-08ee-4ada-9dc0-ac5afd5a59c7","Type":"ContainerDied","Data":"58acf036aaca0c18a60e119c07c8c1f96cc7ee096678314226baf6da4026534d"}
Nov 03 20:33:24.086284 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:24.086260    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-9e1bcb/distinct-2-w6-9e1bcb-kserve-7d96d9f6f6-4f7fl"
Nov 03 20:33:24.091853 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:24.091816    5570 generic.go:356] "Generic (PLEG): container finished" podID="40e4fc9c-c3c1-4aa0-98ad-eda347be89c5" containerID="e91fa2efdc7c6fd5b80b7258d1bd1d371bd0e89b53a388019b215aa898711527" exitCode=0
Nov 03 20:33:24.091995 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:24.091974    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-1d159a-openshift-default-799c8f9695-b8h67" event={"ID":"40e4fc9c-c3c1-4aa0-98ad-eda347be89c5","Type":"ContainerDied","Data":"e91fa2efdc7c6fd5b80b7258d1bd1d371bd0e89b53a388019b215aa898711527"}
Nov 03 20:33:24.092553 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:24.092523    5570 scope.go:122] "RemoveContainer" containerID="e91fa2efdc7c6fd5b80b7258d1bd1d371bd0e89b53a388019b215aa898711527"
Nov 03 20:33:24.099831 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:24.099806    5570 scope.go:122] "RemoveContainer" containerID="e5774781017bffdb309abab3f3038fc990f221a8bb660c664b236d0529e6dce2"
Nov 03 20:33:24.100233 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:24.100196    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:33:24.100382 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:24.100297    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:33:24.100382 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:24.100343    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:33:24.100569 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:33:24.100406    5570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5774781017bffdb309abab3f3038fc990f221a8bb660c664b236d0529e6dce2\": container with ID starting with e5774781017bffdb309abab3f3038fc990f221a8bb660c664b236d0529e6dce2 not found: ID does not exist" containerID="e5774781017bffdb309abab3f3038fc990f221a8bb660c664b236d0529e6dce2"
Nov 03 20:33:24.100569 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:24.100440    5570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5774781017bffdb309abab3f3038fc990f221a8bb660c664b236d0529e6dce2"} err="failed to get container status \"e5774781017bffdb309abab3f3038fc990f221a8bb660c664b236d0529e6dce2\": rpc error: code = NotFound desc = could not find container \"e5774781017bffdb309abab3f3038fc990f221a8bb660c664b236d0529e6dce2\": container with ID starting with e5774781017bffdb309abab3f3038fc990f221a8bb660c664b236d0529e6dce2 not found: ID does not exist"
Nov 03 20:33:24.100569 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:24.100464    5570 scope.go:122] "RemoveContainer" containerID="29dc9938d203f96fc96a132d6cbe946dd3dce03c4807b8721a8ff8f7baea05af"
Nov 03 20:33:24.117879 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:24.117853    5570 scope.go:122] "RemoveContainer" containerID="29dc9938d203f96fc96a132d6cbe946dd3dce03c4807b8721a8ff8f7baea05af"
Nov 03 20:33:24.118266 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:33:24.118237    5570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29dc9938d203f96fc96a132d6cbe946dd3dce03c4807b8721a8ff8f7baea05af\": container with ID starting with 29dc9938d203f96fc96a132d6cbe946dd3dce03c4807b8721a8ff8f7baea05af not found: ID does not exist" containerID="29dc9938d203f96fc96a132d6cbe946dd3dce03c4807b8721a8ff8f7baea05af"
Nov 03 20:33:24.118350 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:24.118276    5570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29dc9938d203f96fc96a132d6cbe946dd3dce03c4807b8721a8ff8f7baea05af"} err="failed to get container status \"29dc9938d203f96fc96a132d6cbe946dd3dce03c4807b8721a8ff8f7baea05af\": rpc error: code = NotFound desc = could not find container \"29dc9938d203f96fc96a132d6cbe946dd3dce03c4807b8721a8ff8f7baea05af\": container with ID starting with 29dc9938d203f96fc96a132d6cbe946dd3dce03c4807b8721a8ff8f7baea05af not found: ID does not exist"
Nov 03 20:33:24.138010 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:24.137976    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-9e1bcb/distinct-2-w6-9e1bcb-kserve-7d96d9f6f6-4f7fl"]
Nov 03 20:33:24.141239 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:24.141216    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-9e1bcb/distinct-2-w6-9e1bcb-kserve-7d96d9f6f6-4f7fl"]
Nov 03 20:33:24.166920 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:24.166890    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5h78z\" (UniqueName: \"kubernetes.io/projected/543a8a92-e2b1-4d88-9b3f-4473806a9a3d-kube-api-access-5h78z\") pod \"test-model-60efdf-kserve-7fb9cb65bf-9774z\" (UID: \"543a8a92-e2b1-4d88-9b3f-4473806a9a3d\") " pod="ai-tenant-e2e-inf-a-60efdf/test-model-60efdf-kserve-7fb9cb65bf-9774z"
Nov 03 20:33:24.167106 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:24.166942    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/543a8a92-e2b1-4d88-9b3f-4473806a9a3d-tmp-dir\") pod \"test-model-60efdf-kserve-7fb9cb65bf-9774z\" (UID: \"543a8a92-e2b1-4d88-9b3f-4473806a9a3d\") " pod="ai-tenant-e2e-inf-a-60efdf/test-model-60efdf-kserve-7fb9cb65bf-9774z"
Nov 03 20:33:24.167106 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:24.166983    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/543a8a92-e2b1-4d88-9b3f-4473806a9a3d-model-cache\") pod \"test-model-60efdf-kserve-7fb9cb65bf-9774z\" (UID: \"543a8a92-e2b1-4d88-9b3f-4473806a9a3d\") " pod="ai-tenant-e2e-inf-a-60efdf/test-model-60efdf-kserve-7fb9cb65bf-9774z"
Nov 03 20:33:24.167229 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:24.167172    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/543a8a92-e2b1-4d88-9b3f-4473806a9a3d-home\") pod \"test-model-60efdf-kserve-7fb9cb65bf-9774z\" (UID: \"543a8a92-e2b1-4d88-9b3f-4473806a9a3d\") " pod="ai-tenant-e2e-inf-a-60efdf/test-model-60efdf-kserve-7fb9cb65bf-9774z"
Nov 03 20:33:24.167283 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:24.167244    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/543a8a92-e2b1-4d88-9b3f-4473806a9a3d-dshm\") pod \"test-model-60efdf-kserve-7fb9cb65bf-9774z\" (UID: \"543a8a92-e2b1-4d88-9b3f-4473806a9a3d\") " pod="ai-tenant-e2e-inf-a-60efdf/test-model-60efdf-kserve-7fb9cb65bf-9774z"
Nov 03 20:33:24.167334 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:24.167295    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/543a8a92-e2b1-4d88-9b3f-4473806a9a3d-tls-certs\") pod \"test-model-60efdf-kserve-7fb9cb65bf-9774z\" (UID: \"543a8a92-e2b1-4d88-9b3f-4473806a9a3d\") " pod="ai-tenant-e2e-inf-a-60efdf/test-model-60efdf-kserve-7fb9cb65bf-9774z"
Nov 03 20:33:24.167422 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:24.167397    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/543a8a92-e2b1-4d88-9b3f-4473806a9a3d-tmp-dir\") pod \"test-model-60efdf-kserve-7fb9cb65bf-9774z\" (UID: \"543a8a92-e2b1-4d88-9b3f-4473806a9a3d\") " pod="ai-tenant-e2e-inf-a-60efdf/test-model-60efdf-kserve-7fb9cb65bf-9774z"
Nov 03 20:33:24.167520 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:24.167486    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/543a8a92-e2b1-4d88-9b3f-4473806a9a3d-model-cache\") pod \"test-model-60efdf-kserve-7fb9cb65bf-9774z\" (UID: \"543a8a92-e2b1-4d88-9b3f-4473806a9a3d\") " pod="ai-tenant-e2e-inf-a-60efdf/test-model-60efdf-kserve-7fb9cb65bf-9774z"
Nov 03 20:33:24.167644 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:24.167565    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/543a8a92-e2b1-4d88-9b3f-4473806a9a3d-home\") pod \"test-model-60efdf-kserve-7fb9cb65bf-9774z\" (UID: \"543a8a92-e2b1-4d88-9b3f-4473806a9a3d\") " pod="ai-tenant-e2e-inf-a-60efdf/test-model-60efdf-kserve-7fb9cb65bf-9774z"
Nov 03 20:33:24.169753 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:24.169726    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/543a8a92-e2b1-4d88-9b3f-4473806a9a3d-dshm\") pod \"test-model-60efdf-kserve-7fb9cb65bf-9774z\" (UID: \"543a8a92-e2b1-4d88-9b3f-4473806a9a3d\") " pod="ai-tenant-e2e-inf-a-60efdf/test-model-60efdf-kserve-7fb9cb65bf-9774z"
Nov 03 20:33:24.170048 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:24.170024    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/543a8a92-e2b1-4d88-9b3f-4473806a9a3d-tls-certs\") pod \"test-model-60efdf-kserve-7fb9cb65bf-9774z\" (UID: \"543a8a92-e2b1-4d88-9b3f-4473806a9a3d\") " pod="ai-tenant-e2e-inf-a-60efdf/test-model-60efdf-kserve-7fb9cb65bf-9774z"
Nov 03 20:33:24.260244 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:24.258919    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5h78z\" (UniqueName: \"kubernetes.io/projected/543a8a92-e2b1-4d88-9b3f-4473806a9a3d-kube-api-access-5h78z\") pod \"test-model-60efdf-kserve-7fb9cb65bf-9774z\" (UID: \"543a8a92-e2b1-4d88-9b3f-4473806a9a3d\") " pod="ai-tenant-e2e-inf-a-60efdf/test-model-60efdf-kserve-7fb9cb65bf-9774z"
Nov 03 20:33:24.290027 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:24.289985    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-9e1bcb/premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh"]
Nov 03 20:33:24.293341 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:24.293292    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-9e1bcb/premium-w6-9e1bcb-kserve-958d5ffd8-v9tlh"]
Nov 03 20:33:24.496885 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:24.496846    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-60efdf/test-model-60efdf-kserve-7fb9cb65bf-9774z"
Nov 03 20:33:24.806413 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:24.806387    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-60efdf/test-model-60efdf-kserve-7fb9cb65bf-9774z"]
Nov 03 20:33:24.807894 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:33:24.807842    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod543a8a92_e2b1_4d88_9b3f_4473806a9a3d.slice/crio-3ca1594bd9a0812176dd8b2dbfb67883c5b624103286ac4cd3bd73dd3f208db0 WatchSource:0}: Error finding container 3ca1594bd9a0812176dd8b2dbfb67883c5b624103286ac4cd3bd73dd3f208db0: Status 404 returned error can't find the container with id 3ca1594bd9a0812176dd8b2dbfb67883c5b624103286ac4cd3bd73dd3f208db0
Nov 03 20:33:24.940687 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:24.940659    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-7e4389-openshift-default-5b5bd7f55-kpk7n"
Nov 03 20:33:24.989768 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:24.989730    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f5aeb82-01f6-483f-8273-10bd0714e03c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f5aeb82-01f6-483f-8273-10bd0714e03c-workload-socket\") pod \"1f5aeb82-01f6-483f-8273-10bd0714e03c\" (UID: \"1f5aeb82-01f6-483f-8273-10bd0714e03c\") "
Nov 03 20:33:24.990001 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:24.989937    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f5aeb82-01f6-483f-8273-10bd0714e03c-kube-api-access-t9sw9\" (UniqueName: \"kubernetes.io/projected/1f5aeb82-01f6-483f-8273-10bd0714e03c-kube-api-access-t9sw9\") pod \"1f5aeb82-01f6-483f-8273-10bd0714e03c\" (UID: \"1f5aeb82-01f6-483f-8273-10bd0714e03c\") "
Nov 03 20:33:24.990083 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:24.990024    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f5aeb82-01f6-483f-8273-10bd0714e03c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f5aeb82-01f6-483f-8273-10bd0714e03c-workload-certs\") pod \"1f5aeb82-01f6-483f-8273-10bd0714e03c\" (UID: \"1f5aeb82-01f6-483f-8273-10bd0714e03c\") "
Nov 03 20:33:24.990083 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:24.990059    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f5aeb82-01f6-483f-8273-10bd0714e03c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f5aeb82-01f6-483f-8273-10bd0714e03c-istio-data\") pod \"1f5aeb82-01f6-483f-8273-10bd0714e03c\" (UID: \"1f5aeb82-01f6-483f-8273-10bd0714e03c\") "
Nov 03 20:33:24.990083 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:24.990056    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f5aeb82-01f6-483f-8273-10bd0714e03c-workload-socket" pod "1f5aeb82-01f6-483f-8273-10bd0714e03c" (UID: "1f5aeb82-01f6-483f-8273-10bd0714e03c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:33:24.990240 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:24.990189    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f5aeb82-01f6-483f-8273-10bd0714e03c-istio-token\" (UniqueName: \"kubernetes.io/projected/1f5aeb82-01f6-483f-8273-10bd0714e03c-istio-token\") pod \"1f5aeb82-01f6-483f-8273-10bd0714e03c\" (UID: \"1f5aeb82-01f6-483f-8273-10bd0714e03c\") "
Nov 03 20:33:24.990240 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:24.990220    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1f5aeb82-01f6-483f-8273-10bd0714e03c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f5aeb82-01f6-483f-8273-10bd0714e03c-istio-podinfo\") pod \"1f5aeb82-01f6-483f-8273-10bd0714e03c\" (UID: \"1f5aeb82-01f6-483f-8273-10bd0714e03c\") "
Nov 03 20:33:24.990348 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:24.990259    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f5aeb82-01f6-483f-8273-10bd0714e03c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f5aeb82-01f6-483f-8273-10bd0714e03c-credential-socket\") pod \"1f5aeb82-01f6-483f-8273-10bd0714e03c\" (UID: \"1f5aeb82-01f6-483f-8273-10bd0714e03c\") "
Nov 03 20:33:24.990471 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:24.990345    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f5aeb82-01f6-483f-8273-10bd0714e03c-istio-data" pod "1f5aeb82-01f6-483f-8273-10bd0714e03c" (UID: "1f5aeb82-01f6-483f-8273-10bd0714e03c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:33:24.990471 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:24.990409    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f5aeb82-01f6-483f-8273-10bd0714e03c-workload-certs" pod "1f5aeb82-01f6-483f-8273-10bd0714e03c" (UID: "1f5aeb82-01f6-483f-8273-10bd0714e03c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:33:24.990987 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:24.990962    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f5aeb82-01f6-483f-8273-10bd0714e03c-credential-socket" pod "1f5aeb82-01f6-483f-8273-10bd0714e03c" (UID: "1f5aeb82-01f6-483f-8273-10bd0714e03c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:33:24.991475 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:24.991452    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f5aeb82-01f6-483f-8273-10bd0714e03c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f5aeb82-01f6-483f-8273-10bd0714e03c-istio-envoy\") pod \"1f5aeb82-01f6-483f-8273-10bd0714e03c\" (UID: \"1f5aeb82-01f6-483f-8273-10bd0714e03c\") "
Nov 03 20:33:24.991669 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:24.991650    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f5aeb82-01f6-483f-8273-10bd0714e03c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f5aeb82-01f6-483f-8273-10bd0714e03c-istiod-ca-cert\") pod \"1f5aeb82-01f6-483f-8273-10bd0714e03c\" (UID: \"1f5aeb82-01f6-483f-8273-10bd0714e03c\") "
Nov 03 20:33:24.992368 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:24.992340    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f5aeb82-01f6-483f-8273-10bd0714e03c-istiod-ca-cert" pod "1f5aeb82-01f6-483f-8273-10bd0714e03c" (UID: "1f5aeb82-01f6-483f-8273-10bd0714e03c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:33:24.992849 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:24.992828    5570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f5aeb82-01f6-483f-8273-10bd0714e03c-credential-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:33:24.992963 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:24.992950    5570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f5aeb82-01f6-483f-8273-10bd0714e03c-istiod-ca-cert\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:33:24.993066 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:24.993054    5570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f5aeb82-01f6-483f-8273-10bd0714e03c-workload-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:33:24.993172 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:24.993160    5570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f5aeb82-01f6-483f-8273-10bd0714e03c-workload-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:33:24.993254 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:24.993244    5570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f5aeb82-01f6-483f-8273-10bd0714e03c-istio-data\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:33:24.994011 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:24.993966    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1f5aeb82-01f6-483f-8273-10bd0714e03c-istio-podinfo" pod "1f5aeb82-01f6-483f-8273-10bd0714e03c" (UID: "1f5aeb82-01f6-483f-8273-10bd0714e03c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:33:24.994011 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:24.993974    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f5aeb82-01f6-483f-8273-10bd0714e03c-istio-token" pod "1f5aeb82-01f6-483f-8273-10bd0714e03c" (UID: "1f5aeb82-01f6-483f-8273-10bd0714e03c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:33:24.994781 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:24.994746    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f5aeb82-01f6-483f-8273-10bd0714e03c-kube-api-access-t9sw9" pod "1f5aeb82-01f6-483f-8273-10bd0714e03c" (UID: "1f5aeb82-01f6-483f-8273-10bd0714e03c"). InnerVolumeSpecName "kube-api-access-t9sw9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:33:24.995202 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:24.995171    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f5aeb82-01f6-483f-8273-10bd0714e03c-istio-envoy" pod "1f5aeb82-01f6-483f-8273-10bd0714e03c" (UID: "1f5aeb82-01f6-483f-8273-10bd0714e03c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:33:25.036877 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:25.036745    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-9973d9/premium-w2-9973d9-kserve-5c6dd4669c-5gv8x"]
Nov 03 20:33:25.095525 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:25.095485    5570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f5aeb82-01f6-483f-8273-10bd0714e03c-istio-token\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:33:25.095525 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:25.095526    5570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f5aeb82-01f6-483f-8273-10bd0714e03c-istio-podinfo\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:33:25.095754 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:25.095540    5570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f5aeb82-01f6-483f-8273-10bd0714e03c-istio-envoy\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:33:25.095754 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:25.095554    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9sw9\" (UniqueName: \"kubernetes.io/projected/1f5aeb82-01f6-483f-8273-10bd0714e03c-kube-api-access-t9sw9\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:33:25.105198 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:25.105157    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-1d159a-openshift-default-799c8f9695-b8h67" event={"ID":"40e4fc9c-c3c1-4aa0-98ad-eda347be89c5","Type":"ContainerStarted","Data":"d200d60ddf9d712ccd5b388bb3ab8a745c07816a59a52cc300d0254e93a41a89"}
Nov 03 20:33:25.106828 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:25.106760    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-60efdf/test-model-60efdf-kserve-7fb9cb65bf-9774z" event={"ID":"543a8a92-e2b1-4d88-9b3f-4473806a9a3d","Type":"ContainerStarted","Data":"c3dad758f122394a07e2faf548e93e71df95738b6e01e1bfcf3c442d86914f6d"}
Nov 03 20:33:25.106976 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:25.106838    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-60efdf/test-model-60efdf-kserve-7fb9cb65bf-9774z" event={"ID":"543a8a92-e2b1-4d88-9b3f-4473806a9a3d","Type":"ContainerStarted","Data":"3ca1594bd9a0812176dd8b2dbfb67883c5b624103286ac4cd3bd73dd3f208db0"}
Nov 03 20:33:25.108623 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:25.108602    5570 generic.go:356] "Generic (PLEG): container finished" podID="1f5aeb82-01f6-483f-8273-10bd0714e03c" containerID="b24a6ddb0a915a87947f09e2e5141c213ffbeec08769fa4c0a030faf8f68be69" exitCode=0
Nov 03 20:33:25.108735 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:25.108645    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-7e4389-openshift-default-5b5bd7f55-kpk7n"
Nov 03 20:33:25.108735 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:25.108664    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-7e4389-openshift-default-5b5bd7f55-kpk7n" event={"ID":"1f5aeb82-01f6-483f-8273-10bd0714e03c","Type":"ContainerDied","Data":"b24a6ddb0a915a87947f09e2e5141c213ffbeec08769fa4c0a030faf8f68be69"}
Nov 03 20:33:25.108735 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:25.108698    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-7e4389-openshift-default-5b5bd7f55-kpk7n" event={"ID":"1f5aeb82-01f6-483f-8273-10bd0714e03c","Type":"ContainerDied","Data":"b8aff8106a2f472d381d6827c2ccaf4419eb482ade529313331ca5e8096c7b19"}
Nov 03 20:33:25.108735 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:25.108718    5570 scope.go:122] "RemoveContainer" containerID="b24a6ddb0a915a87947f09e2e5141c213ffbeec08769fa4c0a030faf8f68be69"
Nov 03 20:33:25.120735 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:25.120707    5570 scope.go:122] "RemoveContainer" containerID="17e044fcfdd8d20d3cc3fa7ec5897d1ad22ba6fcf7a0f47f100303c23ff48798"
Nov 03 20:33:25.131209 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:25.131182    5570 scope.go:122] "RemoveContainer" containerID="b24a6ddb0a915a87947f09e2e5141c213ffbeec08769fa4c0a030faf8f68be69"
Nov 03 20:33:25.131511 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:33:25.131488    5570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b24a6ddb0a915a87947f09e2e5141c213ffbeec08769fa4c0a030faf8f68be69\": container with ID starting with b24a6ddb0a915a87947f09e2e5141c213ffbeec08769fa4c0a030faf8f68be69 not found: ID does not exist" containerID="b24a6ddb0a915a87947f09e2e5141c213ffbeec08769fa4c0a030faf8f68be69"
Nov 03 20:33:25.131608 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:25.131528    5570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b24a6ddb0a915a87947f09e2e5141c213ffbeec08769fa4c0a030faf8f68be69"} err="failed to get container status \"b24a6ddb0a915a87947f09e2e5141c213ffbeec08769fa4c0a030faf8f68be69\": rpc error: code = NotFound desc = could not find container \"b24a6ddb0a915a87947f09e2e5141c213ffbeec08769fa4c0a030faf8f68be69\": container with ID starting with b24a6ddb0a915a87947f09e2e5141c213ffbeec08769fa4c0a030faf8f68be69 not found: ID does not exist"
Nov 03 20:33:25.131608 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:25.131553    5570 scope.go:122] "RemoveContainer" containerID="17e044fcfdd8d20d3cc3fa7ec5897d1ad22ba6fcf7a0f47f100303c23ff48798"
Nov 03 20:33:25.131842 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:33:25.131815    5570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17e044fcfdd8d20d3cc3fa7ec5897d1ad22ba6fcf7a0f47f100303c23ff48798\": container with ID starting with 17e044fcfdd8d20d3cc3fa7ec5897d1ad22ba6fcf7a0f47f100303c23ff48798 not found: ID does not exist" containerID="17e044fcfdd8d20d3cc3fa7ec5897d1ad22ba6fcf7a0f47f100303c23ff48798"
Nov 03 20:33:25.131911 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:25.131848    5570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17e044fcfdd8d20d3cc3fa7ec5897d1ad22ba6fcf7a0f47f100303c23ff48798"} err="failed to get container status \"17e044fcfdd8d20d3cc3fa7ec5897d1ad22ba6fcf7a0f47f100303c23ff48798\": rpc error: code = NotFound desc = could not find container \"17e044fcfdd8d20d3cc3fa7ec5897d1ad22ba6fcf7a0f47f100303c23ff48798\": container with ID starting with 17e044fcfdd8d20d3cc3fa7ec5897d1ad22ba6fcf7a0f47f100303c23ff48798 not found: ID does not exist"
Nov 03 20:33:25.199070 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:25.199037    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-7e4389-openshift-default-5b5bd7f55-kpk7n"]
Nov 03 20:33:25.207575 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:25.207545    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-7e4389-openshift-default-5b5bd7f55-kpk7n"]
Nov 03 20:33:25.245093 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:25.245021    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-60efdf/test-model-60efdf-kserve-7fb9cb65bf-9774z" podStartSLOduration=2.245007797 podStartE2EDuration="2.245007797s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:33:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:33:25.243754314 +0000 UTC m=+1739.396416199" watchObservedRunningTime="2026-11-03 20:33:25.245007797 +0000 UTC m=+1739.397669684"
Nov 03 20:33:25.497175 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:25.497134    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-60efdf/test-model-60efdf-kserve-7fb9cb65bf-9774z"
Nov 03 20:33:25.909463 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:25.909365    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-1d159a-openshift-default-799c8f9695-b8h67"
Nov 03 20:33:25.909910 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:25.909595    5570 patch_prober.go:29] interesting pod/e2e-inf-b-1d159a-openshift-default-799c8f9695-b8h67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body=
Nov 03 20:33:25.909910 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:25.909644    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-1d159a-openshift-default-799c8f9695-b8h67" podUID="40e4fc9c-c3c1-4aa0-98ad-eda347be89c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused"
Nov 03 20:33:26.005827 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:25.993116    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f5aeb82-01f6-483f-8273-10bd0714e03c" path="/var/lib/kubelet/pods/1f5aeb82-01f6-483f-8273-10bd0714e03c/volumes"
Nov 03 20:33:26.005827 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:25.994749    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d3647f2-89e9-4908-a8ee-d84e527abeb0" path="/var/lib/kubelet/pods/2d3647f2-89e9-4908-a8ee-d84e527abeb0/volumes"
Nov 03 20:33:26.005827 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:25.995457    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56353b0f-08ee-4ada-9dc0-ac5afd5a59c7" path="/var/lib/kubelet/pods/56353b0f-08ee-4ada-9dc0-ac5afd5a59c7/volumes"
Nov 03 20:33:26.411325 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:26.411286    5570 patch_prober.go:29] interesting pod/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-jshb8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 20:33:26.411495 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:26.411358    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-jshb8" podUID="2137488e-8e6c-46f7-a0ed-b4721d4f2645" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 20:33:26.909039 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:26.909004    5570 patch_prober.go:29] interesting pod/e2e-inf-b-1d159a-openshift-default-799c8f9695-b8h67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body=
Nov 03 20:33:26.909263 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:26.909105    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-1d159a-openshift-default-799c8f9695-b8h67" podUID="40e4fc9c-c3c1-4aa0-98ad-eda347be89c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused"
Nov 03 20:33:27.230239 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:33:27.230197    5570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8bwhz" containerName="extractor"
Nov 03 20:33:27.909341 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:27.909302    5570 patch_prober.go:29] interesting pod/e2e-inf-b-1d159a-openshift-default-799c8f9695-b8h67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body=
Nov 03 20:33:27.909533 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:27.909386    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-1d159a-openshift-default-799c8f9695-b8h67" podUID="40e4fc9c-c3c1-4aa0-98ad-eda347be89c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused"
Nov 03 20:33:28.080152 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:28.080122    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-jshb8"
Nov 03 20:33:28.083093 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:28.083063    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-rbvqw"
Nov 03 20:33:28.088596 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:28.088570    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-j7dz6"
Nov 03 20:33:28.095742 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:28.095717    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-g262x"
Nov 03 20:33:28.134700 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:28.134668    5570 generic.go:356] "Generic (PLEG): container finished" podID="ae5a764e-25f7-492e-b4aa-8f2696e22f37" containerID="d1909a45904a0d779c9eee7a33e1d1b228fb1289b0a14179aaa2f840cae28632" exitCode=0
Nov 03 20:33:28.134700 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:28.134712    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-rbvqw"
Nov 03 20:33:28.134971 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:28.134748    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-rbvqw" event={"ID":"ae5a764e-25f7-492e-b4aa-8f2696e22f37","Type":"ContainerDied","Data":"d1909a45904a0d779c9eee7a33e1d1b228fb1289b0a14179aaa2f840cae28632"}
Nov 03 20:33:28.134971 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:28.134825    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-rbvqw" event={"ID":"ae5a764e-25f7-492e-b4aa-8f2696e22f37","Type":"ContainerDied","Data":"1da53265a6710cf9dd25041f57e3eb4b0f2da2c40cb98f13a8473405a95ec4e5"}
Nov 03 20:33:28.134971 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:28.134857    5570 scope.go:122] "RemoveContainer" containerID="d1909a45904a0d779c9eee7a33e1d1b228fb1289b0a14179aaa2f840cae28632"
Nov 03 20:33:28.136250 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:28.136223    5570 generic.go:356] "Generic (PLEG): container finished" podID="8aa4c48e-92c6-49c8-acd3-4bc441ac8ca7" containerID="83026a35e395a00762993ad189f351188a419bba8a38d57433b5dae409cf0502" exitCode=0
Nov 03 20:33:28.136381 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:28.136271    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-g262x"
Nov 03 20:33:28.136381 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:28.136302    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-g262x" event={"ID":"8aa4c48e-92c6-49c8-acd3-4bc441ac8ca7","Type":"ContainerDied","Data":"83026a35e395a00762993ad189f351188a419bba8a38d57433b5dae409cf0502"}
Nov 03 20:33:28.136381 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:28.136350    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-g262x" event={"ID":"8aa4c48e-92c6-49c8-acd3-4bc441ac8ca7","Type":"ContainerDied","Data":"894672646f01bfbb4093e78ccb59e3eab82f7fe17ad848b02b9f19036678c792"}
Nov 03 20:33:28.169131 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:28.169043    5570 generic.go:356] "Generic (PLEG): container finished" podID="2137488e-8e6c-46f7-a0ed-b4721d4f2645" containerID="e5c7ec1d14a857d58b25629abd68ececf4e69dc30db9346800e1c54320f24671" exitCode=0
Nov 03 20:33:28.169131 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:28.169120    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-jshb8"
Nov 03 20:33:28.169371 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:28.169185    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-jshb8" event={"ID":"2137488e-8e6c-46f7-a0ed-b4721d4f2645","Type":"ContainerDied","Data":"e5c7ec1d14a857d58b25629abd68ececf4e69dc30db9346800e1c54320f24671"}
Nov 03 20:33:28.169371 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:28.169224    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-jshb8" event={"ID":"2137488e-8e6c-46f7-a0ed-b4721d4f2645","Type":"ContainerDied","Data":"3e8680d5d08f95692b79cef9b5ca20ce0df9b6c1a4f3ae92411d7dc55892cdff"}
Nov 03 20:33:28.170909 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:28.170885    5570 generic.go:356] "Generic (PLEG): container finished" podID="22ea3ab0-6acc-4558-9158-eaa289924f35" containerID="c35d6c2c4780ada7348150b2cfc4cd1948d3f225714d8b58219a774af050416b" exitCode=0
Nov 03 20:33:28.171047 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:28.170950    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-j7dz6" event={"ID":"22ea3ab0-6acc-4558-9158-eaa289924f35","Type":"ContainerDied","Data":"c35d6c2c4780ada7348150b2cfc4cd1948d3f225714d8b58219a774af050416b"}
Nov 03 20:33:28.171047 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:28.170955    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-j7dz6"
Nov 03 20:33:28.171047 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:28.170991    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-j7dz6" event={"ID":"22ea3ab0-6acc-4558-9158-eaa289924f35","Type":"ContainerDied","Data":"14b3d9881b512a53a263be0260dab7169a79def626efceacb6eb0073305b2136"}
Nov 03 20:33:28.175348 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:28.175320    5570 scope.go:122] "RemoveContainer" containerID="d1909a45904a0d779c9eee7a33e1d1b228fb1289b0a14179aaa2f840cae28632"
Nov 03 20:33:28.175672 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:33:28.175648    5570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1909a45904a0d779c9eee7a33e1d1b228fb1289b0a14179aaa2f840cae28632\": container with ID starting with d1909a45904a0d779c9eee7a33e1d1b228fb1289b0a14179aaa2f840cae28632 not found: ID does not exist" containerID="d1909a45904a0d779c9eee7a33e1d1b228fb1289b0a14179aaa2f840cae28632"
Nov 03 20:33:28.175821 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:28.175681    5570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1909a45904a0d779c9eee7a33e1d1b228fb1289b0a14179aaa2f840cae28632"} err="failed to get container status \"d1909a45904a0d779c9eee7a33e1d1b228fb1289b0a14179aaa2f840cae28632\": rpc error: code = NotFound desc = could not find container \"d1909a45904a0d779c9eee7a33e1d1b228fb1289b0a14179aaa2f840cae28632\": container with ID starting with d1909a45904a0d779c9eee7a33e1d1b228fb1289b0a14179aaa2f840cae28632 not found: ID does not exist"
Nov 03 20:33:28.175821 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:28.175698    5570 scope.go:122] "RemoveContainer" containerID="83026a35e395a00762993ad189f351188a419bba8a38d57433b5dae409cf0502"
Nov 03 20:33:28.178230 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:28.178206    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae5a764e-25f7-492e-b4aa-8f2696e22f37-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae5a764e-25f7-492e-b4aa-8f2696e22f37-istio-data\") pod \"ae5a764e-25f7-492e-b4aa-8f2696e22f37\" (UID: \"ae5a764e-25f7-492e-b4aa-8f2696e22f37\") "
Nov 03 20:33:28.178306 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:28.178256    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22ea3ab0-6acc-4558-9158-eaa289924f35-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22ea3ab0-6acc-4558-9158-eaa289924f35-istio-data\") pod \"22ea3ab0-6acc-4558-9158-eaa289924f35\" (UID: \"22ea3ab0-6acc-4558-9158-eaa289924f35\") "
Nov 03 20:33:28.178306 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:28.178285    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8aa4c48e-92c6-49c8-acd3-4bc441ac8ca7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8aa4c48e-92c6-49c8-acd3-4bc441ac8ca7-workload-certs\") pod \"8aa4c48e-92c6-49c8-acd3-4bc441ac8ca7\" (UID: \"8aa4c48e-92c6-49c8-acd3-4bc441ac8ca7\") "
Nov 03 20:33:28.178405 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:28.178368    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2137488e-8e6c-46f7-a0ed-b4721d4f2645-istio-token\" (UniqueName: \"kubernetes.io/projected/2137488e-8e6c-46f7-a0ed-b4721d4f2645-istio-token\") pod \"2137488e-8e6c-46f7-a0ed-b4721d4f2645\" (UID: \"2137488e-8e6c-46f7-a0ed-b4721d4f2645\") "
Nov 03 20:33:28.178449 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:28.178415    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22ea3ab0-6acc-4558-9158-eaa289924f35-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22ea3ab0-6acc-4558-9158-eaa289924f35-istiod-ca-cert\") pod \"22ea3ab0-6acc-4558-9158-eaa289924f35\" (UID: \"22ea3ab0-6acc-4558-9158-eaa289924f35\") "
Nov 03 20:33:28.178487 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:28.178452    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8aa4c48e-92c6-49c8-acd3-4bc441ac8ca7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8aa4c48e-92c6-49c8-acd3-4bc441ac8ca7-istio-podinfo\") pod \"8aa4c48e-92c6-49c8-acd3-4bc441ac8ca7\" (UID: \"8aa4c48e-92c6-49c8-acd3-4bc441ac8ca7\") "
Nov 03 20:33:28.178524 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:28.178490    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae5a764e-25f7-492e-b4aa-8f2696e22f37-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae5a764e-25f7-492e-b4aa-8f2696e22f37-workload-certs\") pod \"ae5a764e-25f7-492e-b4aa-8f2696e22f37\" (UID: \"ae5a764e-25f7-492e-b4aa-8f2696e22f37\") "
Nov 03 20:33:28.178559 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:28.178526    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae5a764e-25f7-492e-b4aa-8f2696e22f37-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae5a764e-25f7-492e-b4aa-8f2696e22f37-credential-socket\") pod \"ae5a764e-25f7-492e-b4aa-8f2696e22f37\" (UID: \"ae5a764e-25f7-492e-b4aa-8f2696e22f37\") "
Nov 03 20:33:28.178621 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:28.178605    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2137488e-8e6c-46f7-a0ed-b4721d4f2645-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2137488e-8e6c-46f7-a0ed-b4721d4f2645-workload-certs\") pod \"2137488e-8e6c-46f7-a0ed-b4721d4f2645\" (UID: \"2137488e-8e6c-46f7-a0ed-b4721d4f2645\") "
Nov 03 20:33:28.178663 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:28.178620    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8aa4c48e-92c6-49c8-acd3-4bc441ac8ca7-workload-certs" pod "8aa4c48e-92c6-49c8-acd3-4bc441ac8ca7" (UID: "8aa4c48e-92c6-49c8-acd3-4bc441ac8ca7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:33:28.178663 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:28.178648    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae5a764e-25f7-492e-b4aa-8f2696e22f37-kube-api-access-52h2d\" (UniqueName: \"kubernetes.io/projected/ae5a764e-25f7-492e-b4aa-8f2696e22f37-kube-api-access-52h2d\") pod \"ae5a764e-25f7-492e-b4aa-8f2696e22f37\" (UID: \"ae5a764e-25f7-492e-b4aa-8f2696e22f37\") "
Nov 03 20:33:28.178727 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:28.178686    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2137488e-8e6c-46f7-a0ed-b4721d4f2645-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2137488e-8e6c-46f7-a0ed-b4721d4f2645-credential-socket\") pod \"2137488e-8e6c-46f7-a0ed-b4721d4f2645\" (UID: \"2137488e-8e6c-46f7-a0ed-b4721d4f2645\") "
Nov 03 20:33:28.178763 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:28.178732    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22ea3ab0-6acc-4558-9158-eaa289924f35-kube-api-access-m4rvl\" (UniqueName: \"kubernetes.io/projected/22ea3ab0-6acc-4558-9158-eaa289924f35-kube-api-access-m4rvl\") pod \"22ea3ab0-6acc-4558-9158-eaa289924f35\" (UID: \"22ea3ab0-6acc-4558-9158-eaa289924f35\") "
Nov 03 20:33:28.178839 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:28.178793    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22ea3ab0-6acc-4558-9158-eaa289924f35-istio-token\" (UniqueName: \"kubernetes.io/projected/22ea3ab0-6acc-4558-9158-eaa289924f35-istio-token\") pod \"22ea3ab0-6acc-4558-9158-eaa289924f35\" (UID: \"22ea3ab0-6acc-4558-9158-eaa289924f35\") "
Nov 03 20:33:28.178879 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:28.178852    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2137488e-8e6c-46f7-a0ed-b4721d4f2645-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2137488e-8e6c-46f7-a0ed-b4721d4f2645-istiod-ca-cert\") pod \"2137488e-8e6c-46f7-a0ed-b4721d4f2645\" (UID: \"2137488e-8e6c-46f7-a0ed-b4721d4f2645\") "
Nov 03 20:33:28.178926 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:28.178892    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2137488e-8e6c-46f7-a0ed-b4721d4f2645-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2137488e-8e6c-46f7-a0ed-b4721d4f2645-istio-envoy\") pod \"2137488e-8e6c-46f7-a0ed-b4721d4f2645\" (UID: \"2137488e-8e6c-46f7-a0ed-b4721d4f2645\") "
Nov 03 20:33:28.178926 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:28.178921    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae5a764e-25f7-492e-b4aa-8f2696e22f37-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae5a764e-25f7-492e-b4aa-8f2696e22f37-istiod-ca-cert\") pod \"ae5a764e-25f7-492e-b4aa-8f2696e22f37\" (UID: \"ae5a764e-25f7-492e-b4aa-8f2696e22f37\") "
Nov 03 20:33:28.179048 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:28.178969    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22ea3ab0-6acc-4558-9158-eaa289924f35-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22ea3ab0-6acc-4558-9158-eaa289924f35-workload-certs\") pod \"22ea3ab0-6acc-4558-9158-eaa289924f35\" (UID: \"22ea3ab0-6acc-4558-9158-eaa289924f35\") "
Nov 03 20:33:28.179048 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:28.179015    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2137488e-8e6c-46f7-a0ed-b4721d4f2645-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2137488e-8e6c-46f7-a0ed-b4721d4f2645-istio-podinfo\") pod \"2137488e-8e6c-46f7-a0ed-b4721d4f2645\" (UID: \"2137488e-8e6c-46f7-a0ed-b4721d4f2645\") "
Nov 03 20:33:28.179167 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:28.179051    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8aa4c48e-92c6-49c8-acd3-4bc441ac8ca7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8aa4c48e-92c6-49c8-acd3-4bc441ac8ca7-workload-socket\") pod \"8aa4c48e-92c6-49c8-acd3-4bc441ac8ca7\" (UID: \"8aa4c48e-92c6-49c8-acd3-4bc441ac8ca7\") "
Nov 03 20:33:28.179167 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:28.179088    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae5a764e-25f7-492e-b4aa-8f2696e22f37-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae5a764e-25f7-492e-b4aa-8f2696e22f37-istio-envoy\") pod \"ae5a764e-25f7-492e-b4aa-8f2696e22f37\" (UID: \"ae5a764e-25f7-492e-b4aa-8f2696e22f37\") "
Nov 03 20:33:28.179167 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:28.179121    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22ea3ab0-6acc-4558-9158-eaa289924f35-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22ea3ab0-6acc-4558-9158-eaa289924f35-istio-envoy\") pod \"22ea3ab0-6acc-4558-9158-eaa289924f35\" (UID: \"22ea3ab0-6acc-4558-9158-eaa289924f35\") "
Nov 03 20:33:28.179167 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:28.179147    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8aa4c48e-92c6-49c8-acd3-4bc441ac8ca7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8aa4c48e-92c6-49c8-acd3-4bc441ac8ca7-istiod-ca-cert\") pod \"8aa4c48e-92c6-49c8-acd3-4bc441ac8ca7\" (UID: \"8aa4c48e-92c6-49c8-acd3-4bc441ac8ca7\") "
Nov 03 20:33:28.179369 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:28.179196    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae5a764e-25f7-492e-b4aa-8f2696e22f37-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae5a764e-25f7-492e-b4aa-8f2696e22f37-workload-socket\") pod \"ae5a764e-25f7-492e-b4aa-8f2696e22f37\" (UID: \"ae5a764e-25f7-492e-b4aa-8f2696e22f37\") "
Nov 03 20:33:28.179369 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:28.179225    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8aa4c48e-92c6-49c8-acd3-4bc441ac8ca7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8aa4c48e-92c6-49c8-acd3-4bc441ac8ca7-istio-envoy\") pod \"8aa4c48e-92c6-49c8-acd3-4bc441ac8ca7\" (UID: \"8aa4c48e-92c6-49c8-acd3-4bc441ac8ca7\") "
Nov 03 20:33:28.179811 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:28.179526    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22ea3ab0-6acc-4558-9158-eaa289924f35-workload-certs" pod "22ea3ab0-6acc-4558-9158-eaa289924f35" (UID: "22ea3ab0-6acc-4558-9158-eaa289924f35"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:33:28.179811 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:28.179568    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2137488e-8e6c-46f7-a0ed-b4721d4f2645-workload-certs" pod "2137488e-8e6c-46f7-a0ed-b4721d4f2645" (UID: "2137488e-8e6c-46f7-a0ed-b4721d4f2645"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:33:28.179811 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:28.179681    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22ea3ab0-6acc-4558-9158-eaa289924f35-istiod-ca-cert" pod "22ea3ab0-6acc-4558-9158-eaa289924f35" (UID: "22ea3ab0-6acc-4558-9158-eaa289924f35"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:33:28.179811 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:28.179721    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae5a764e-25f7-492e-b4aa-8f2696e22f37-istio-token\" (UniqueName: \"kubernetes.io/projected/ae5a764e-25f7-492e-b4aa-8f2696e22f37-istio-token\") pod \"ae5a764e-25f7-492e-b4aa-8f2696e22f37\" (UID: \"ae5a764e-25f7-492e-b4aa-8f2696e22f37\") "
Nov 03 20:33:28.179811 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:28.179788    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22ea3ab0-6acc-4558-9158-eaa289924f35-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22ea3ab0-6acc-4558-9158-eaa289924f35-credential-socket\") pod \"22ea3ab0-6acc-4558-9158-eaa289924f35\" (UID: \"22ea3ab0-6acc-4558-9158-eaa289924f35\") "
Nov 03 20:33:28.180232 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:28.179816    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8aa4c48e-92c6-49c8-acd3-4bc441ac8ca7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8aa4c48e-92c6-49c8-acd3-4bc441ac8ca7-credential-socket\") pod \"8aa4c48e-92c6-49c8-acd3-4bc441ac8ca7\" (UID: \"8aa4c48e-92c6-49c8-acd3-4bc441ac8ca7\") "
Nov 03 20:33:28.180232 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:28.179908    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2137488e-8e6c-46f7-a0ed-b4721d4f2645-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2137488e-8e6c-46f7-a0ed-b4721d4f2645-istio-data\") pod \"2137488e-8e6c-46f7-a0ed-b4721d4f2645\" (UID: \"2137488e-8e6c-46f7-a0ed-b4721d4f2645\") "
Nov 03 20:33:28.180232 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:28.179936    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/22ea3ab0-6acc-4558-9158-eaa289924f35-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22ea3ab0-6acc-4558-9158-eaa289924f35-istio-podinfo\") pod \"22ea3ab0-6acc-4558-9158-eaa289924f35\" (UID: \"22ea3ab0-6acc-4558-9158-eaa289924f35\") "
Nov 03 20:33:28.180232 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:28.179989    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2137488e-8e6c-46f7-a0ed-b4721d4f2645-kube-api-access-8nf4b\" (UniqueName: \"kubernetes.io/projected/2137488e-8e6c-46f7-a0ed-b4721d4f2645-kube-api-access-8nf4b\") pod \"2137488e-8e6c-46f7-a0ed-b4721d4f2645\" (UID: \"2137488e-8e6c-46f7-a0ed-b4721d4f2645\") "
Nov 03 20:33:28.180232 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:28.179993    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2137488e-8e6c-46f7-a0ed-b4721d4f2645-credential-socket" pod "2137488e-8e6c-46f7-a0ed-b4721d4f2645" (UID: "2137488e-8e6c-46f7-a0ed-b4721d4f2645"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:33:28.180232 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:28.180019    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22ea3ab0-6acc-4558-9158-eaa289924f35-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22ea3ab0-6acc-4558-9158-eaa289924f35-workload-socket\") pod \"22ea3ab0-6acc-4558-9158-eaa289924f35\" (UID: \"22ea3ab0-6acc-4558-9158-eaa289924f35\") "
Nov 03 20:33:28.180232 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:28.180052    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8aa4c48e-92c6-49c8-acd3-4bc441ac8ca7-kube-api-access-xmw5r\" (UniqueName: \"kubernetes.io/projected/8aa4c48e-92c6-49c8-acd3-4bc441ac8ca7-kube-api-access-xmw5r\") pod \"8aa4c48e-92c6-49c8-acd3-4bc441ac8ca7\" (UID: \"8aa4c48e-92c6-49c8-acd3-4bc441ac8ca7\") "
Nov 03 20:33:28.180232 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:28.180103    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8aa4c48e-92c6-49c8-acd3-4bc441ac8ca7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8aa4c48e-92c6-49c8-acd3-4bc441ac8ca7-istio-data\") pod \"8aa4c48e-92c6-49c8-acd3-4bc441ac8ca7\" (UID: \"8aa4c48e-92c6-49c8-acd3-4bc441ac8ca7\") "
Nov 03 20:33:28.180232 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:28.180132    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8aa4c48e-92c6-49c8-acd3-4bc441ac8ca7-istio-token\" (UniqueName: \"kubernetes.io/projected/8aa4c48e-92c6-49c8-acd3-4bc441ac8ca7-istio-token\") pod \"8aa4c48e-92c6-49c8-acd3-4bc441ac8ca7\" (UID: \"8aa4c48e-92c6-49c8-acd3-4bc441ac8ca7\") "
Nov 03 20:33:28.180849 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:28.180360    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2137488e-8e6c-46f7-a0ed-b4721d4f2645-istiod-ca-cert" pod "2137488e-8e6c-46f7-a0ed-b4721d4f2645" (UID: "2137488e-8e6c-46f7-a0ed-b4721d4f2645"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:33:28.183060 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:28.181638    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22ea3ab0-6acc-4558-9158-eaa289924f35-credential-socket" pod "22ea3ab0-6acc-4558-9158-eaa289924f35" (UID: "22ea3ab0-6acc-4558-9158-eaa289924f35"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:33:28.183060 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:28.181981    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8aa4c48e-92c6-49c8-acd3-4bc441ac8ca7-istio-data" pod "8aa4c48e-92c6-49c8-acd3-4bc441ac8ca7" (UID: "8aa4c48e-92c6-49c8-acd3-4bc441ac8ca7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:33:28.183060 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:28.181989    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2137488e-8e6c-46f7-a0ed-b4721d4f2645-istio-data" pod "2137488e-8e6c-46f7-a0ed-b4721d4f2645" (UID: "2137488e-8e6c-46f7-a0ed-b4721d4f2645"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:33:28.183060 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:28.182097    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ae5a764e-25f7-492e-b4aa-8f2696e22f37-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae5a764e-25f7-492e-b4aa-8f2696e22f37-istio-podinfo\") pod \"ae5a764e-25f7-492e-b4aa-8f2696e22f37\" (UID: \"ae5a764e-25f7-492e-b4aa-8f2696e22f37\") "
Nov 03 20:33:28.183060 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:28.182148    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2137488e-8e6c-46f7-a0ed-b4721d4f2645-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2137488e-8e6c-46f7-a0ed-b4721d4f2645-workload-socket\") pod \"2137488e-8e6c-46f7-a0ed-b4721d4f2645\" (UID: \"2137488e-8e6c-46f7-a0ed-b4721d4f2645\") "
Nov 03 20:33:28.183060 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:28.182175    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22ea3ab0-6acc-4558-9158-eaa289924f35-kube-api-access-m4rvl" pod "22ea3ab0-6acc-4558-9158-eaa289924f35" (UID: "22ea3ab0-6acc-4558-9158-eaa289924f35"). InnerVolumeSpecName "kube-api-access-m4rvl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:33:28.183060 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:28.182631    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8aa4c48e-92c6-49c8-acd3-4bc441ac8ca7-credential-socket" pod "8aa4c48e-92c6-49c8-acd3-4bc441ac8ca7" (UID: "8aa4c48e-92c6-49c8-acd3-4bc441ac8ca7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:33:28.183060 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:28.182919    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2137488e-8e6c-46f7-a0ed-b4721d4f2645-workload-socket" pod "2137488e-8e6c-46f7-a0ed-b4721d4f2645" (UID: "2137488e-8e6c-46f7-a0ed-b4721d4f2645"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:33:28.183060 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:28.182974    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22ea3ab0-6acc-4558-9158-eaa289924f35-workload-socket" pod "22ea3ab0-6acc-4558-9158-eaa289924f35" (UID: "22ea3ab0-6acc-4558-9158-eaa289924f35"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:33:28.184537 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:28.183112    5570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8aa4c48e-92c6-49c8-acd3-4bc441ac8ca7-workload-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:33:28.184537 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:28.183123    5570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22ea3ab0-6acc-4558-9158-eaa289924f35-istiod-ca-cert\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:33:28.184537 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:28.183131    5570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2137488e-8e6c-46f7-a0ed-b4721d4f2645-workload-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:33:28.184537 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:28.183139    5570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2137488e-8e6c-46f7-a0ed-b4721d4f2645-credential-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:33:28.184537 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:28.183148    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4rvl\" (UniqueName: \"kubernetes.io/projected/22ea3ab0-6acc-4558-9158-eaa289924f35-kube-api-access-m4rvl\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:33:28.184537 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:28.183156    5570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2137488e-8e6c-46f7-a0ed-b4721d4f2645-istiod-ca-cert\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:33:28.184537 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:28.183164    5570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22ea3ab0-6acc-4558-9158-eaa289924f35-workload-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:33:28.184537 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:28.183172    5570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22ea3ab0-6acc-4558-9158-eaa289924f35-credential-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:33:28.184537 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:28.183180    5570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8aa4c48e-92c6-49c8-acd3-4bc441ac8ca7-credential-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:33:28.184537 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:28.183188    5570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2137488e-8e6c-46f7-a0ed-b4721d4f2645-istio-data\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:33:28.184537 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:28.183198    5570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22ea3ab0-6acc-4558-9158-eaa289924f35-workload-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:33:28.184537 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:28.183206    5570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8aa4c48e-92c6-49c8-acd3-4bc441ac8ca7-istio-data\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:33:28.184537 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:28.183215    5570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2137488e-8e6c-46f7-a0ed-b4721d4f2645-workload-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:33:28.184537 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:28.183745    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8aa4c48e-92c6-49c8-acd3-4bc441ac8ca7-istiod-ca-cert" pod "8aa4c48e-92c6-49c8-acd3-4bc441ac8ca7" (UID: "8aa4c48e-92c6-49c8-acd3-4bc441ac8ca7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:33:28.184537 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:28.184086    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8aa4c48e-92c6-49c8-acd3-4bc441ac8ca7-workload-socket" pod "8aa4c48e-92c6-49c8-acd3-4bc441ac8ca7" (UID: "8aa4c48e-92c6-49c8-acd3-4bc441ac8ca7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:33:28.185255 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:28.184755    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22ea3ab0-6acc-4558-9158-eaa289924f35-istio-data" pod "22ea3ab0-6acc-4558-9158-eaa289924f35" (UID: "22ea3ab0-6acc-4558-9158-eaa289924f35"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:33:28.185291 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:28.185247    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2137488e-8e6c-46f7-a0ed-b4721d4f2645-istio-token" pod "2137488e-8e6c-46f7-a0ed-b4721d4f2645" (UID: "2137488e-8e6c-46f7-a0ed-b4721d4f2645"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:33:28.186486 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:28.185592    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae5a764e-25f7-492e-b4aa-8f2696e22f37-credential-socket" pod "ae5a764e-25f7-492e-b4aa-8f2696e22f37" (UID: "ae5a764e-25f7-492e-b4aa-8f2696e22f37"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:33:28.186486 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:28.186012    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae5a764e-25f7-492e-b4aa-8f2696e22f37-workload-socket" pod "ae5a764e-25f7-492e-b4aa-8f2696e22f37" (UID: "ae5a764e-25f7-492e-b4aa-8f2696e22f37"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:33:28.186486 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:28.186026    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae5a764e-25f7-492e-b4aa-8f2696e22f37-workload-certs" pod "ae5a764e-25f7-492e-b4aa-8f2696e22f37" (UID: "ae5a764e-25f7-492e-b4aa-8f2696e22f37"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:33:28.186486 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:28.186070    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae5a764e-25f7-492e-b4aa-8f2696e22f37-istiod-ca-cert" pod "ae5a764e-25f7-492e-b4aa-8f2696e22f37" (UID: "ae5a764e-25f7-492e-b4aa-8f2696e22f37"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:33:28.187144 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:28.187122    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae5a764e-25f7-492e-b4aa-8f2696e22f37-istio-data" pod "ae5a764e-25f7-492e-b4aa-8f2696e22f37" (UID: "ae5a764e-25f7-492e-b4aa-8f2696e22f37"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:33:28.187259 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:28.187234    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2137488e-8e6c-46f7-a0ed-b4721d4f2645-kube-api-access-8nf4b" pod "2137488e-8e6c-46f7-a0ed-b4721d4f2645" (UID: "2137488e-8e6c-46f7-a0ed-b4721d4f2645"). InnerVolumeSpecName "kube-api-access-8nf4b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:33:28.187465 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:28.187445    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22ea3ab0-6acc-4558-9158-eaa289924f35-istio-envoy" pod "22ea3ab0-6acc-4558-9158-eaa289924f35" (UID: "22ea3ab0-6acc-4558-9158-eaa289924f35"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:33:28.189761 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:28.189734    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae5a764e-25f7-492e-b4aa-8f2696e22f37-istio-token" pod "ae5a764e-25f7-492e-b4aa-8f2696e22f37" (UID: "ae5a764e-25f7-492e-b4aa-8f2696e22f37"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:33:28.190209 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:28.190172    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae5a764e-25f7-492e-b4aa-8f2696e22f37-istio-envoy" pod "ae5a764e-25f7-492e-b4aa-8f2696e22f37" (UID: "ae5a764e-25f7-492e-b4aa-8f2696e22f37"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:33:28.190494 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:28.190462    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae5a764e-25f7-492e-b4aa-8f2696e22f37-kube-api-access-52h2d" pod "ae5a764e-25f7-492e-b4aa-8f2696e22f37" (UID: "ae5a764e-25f7-492e-b4aa-8f2696e22f37"). InnerVolumeSpecName "kube-api-access-52h2d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:33:28.191503 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:28.191452    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ae5a764e-25f7-492e-b4aa-8f2696e22f37-istio-podinfo" pod "ae5a764e-25f7-492e-b4aa-8f2696e22f37" (UID: "ae5a764e-25f7-492e-b4aa-8f2696e22f37"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:33:28.191897 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:28.191675    5570 scope.go:122] "RemoveContainer" containerID="83026a35e395a00762993ad189f351188a419bba8a38d57433b5dae409cf0502"
Nov 03 20:33:28.192318 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:33:28.192203    5570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83026a35e395a00762993ad189f351188a419bba8a38d57433b5dae409cf0502\": container with ID starting with 83026a35e395a00762993ad189f351188a419bba8a38d57433b5dae409cf0502 not found: ID does not exist" containerID="83026a35e395a00762993ad189f351188a419bba8a38d57433b5dae409cf0502"
Nov 03 20:33:28.192318 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:28.192229    5570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83026a35e395a00762993ad189f351188a419bba8a38d57433b5dae409cf0502"} err="failed to get container status \"83026a35e395a00762993ad189f351188a419bba8a38d57433b5dae409cf0502\": rpc error: code = NotFound desc = could not find container \"83026a35e395a00762993ad189f351188a419bba8a38d57433b5dae409cf0502\": container with ID starting with 83026a35e395a00762993ad189f351188a419bba8a38d57433b5dae409cf0502 not found: ID does not exist"
Nov 03 20:33:28.192318 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:28.192249    5570 scope.go:122] "RemoveContainer" containerID="e5c7ec1d14a857d58b25629abd68ececf4e69dc30db9346800e1c54320f24671"
Nov 03 20:33:28.193064 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:28.193045    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8aa4c48e-92c6-49c8-acd3-4bc441ac8ca7-kube-api-access-xmw5r" pod "8aa4c48e-92c6-49c8-acd3-4bc441ac8ca7" (UID: "8aa4c48e-92c6-49c8-acd3-4bc441ac8ca7"). InnerVolumeSpecName "kube-api-access-xmw5r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:33:28.193198 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:28.193167    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2137488e-8e6c-46f7-a0ed-b4721d4f2645-istio-podinfo" pod "2137488e-8e6c-46f7-a0ed-b4721d4f2645" (UID: "2137488e-8e6c-46f7-a0ed-b4721d4f2645"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:33:28.193451 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:28.193348    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/22ea3ab0-6acc-4558-9158-eaa289924f35-istio-podinfo" pod "22ea3ab0-6acc-4558-9158-eaa289924f35" (UID: "22ea3ab0-6acc-4558-9158-eaa289924f35"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:33:28.193451 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:28.193391    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8aa4c48e-92c6-49c8-acd3-4bc441ac8ca7-istio-envoy" pod "8aa4c48e-92c6-49c8-acd3-4bc441ac8ca7" (UID: "8aa4c48e-92c6-49c8-acd3-4bc441ac8ca7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:33:28.193451 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:28.193431    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2137488e-8e6c-46f7-a0ed-b4721d4f2645-istio-envoy" pod "2137488e-8e6c-46f7-a0ed-b4721d4f2645" (UID: "2137488e-8e6c-46f7-a0ed-b4721d4f2645"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:33:28.193631 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:28.193494    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8aa4c48e-92c6-49c8-acd3-4bc441ac8ca7-istio-podinfo" pod "8aa4c48e-92c6-49c8-acd3-4bc441ac8ca7" (UID: "8aa4c48e-92c6-49c8-acd3-4bc441ac8ca7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:33:28.193631 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:28.193566    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22ea3ab0-6acc-4558-9158-eaa289924f35-istio-token" pod "22ea3ab0-6acc-4558-9158-eaa289924f35" (UID: "22ea3ab0-6acc-4558-9158-eaa289924f35"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:33:28.194623 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:28.194601    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8aa4c48e-92c6-49c8-acd3-4bc441ac8ca7-istio-token" pod "8aa4c48e-92c6-49c8-acd3-4bc441ac8ca7" (UID: "8aa4c48e-92c6-49c8-acd3-4bc441ac8ca7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:33:28.208724 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:28.208605    5570 scope.go:122] "RemoveContainer" containerID="117061888c8f2f87481828900cffe3e409615fd7d0abfd26850b40bf839788ee"
Nov 03 20:33:28.222558 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:28.222531    5570 scope.go:122] "RemoveContainer" containerID="e5c7ec1d14a857d58b25629abd68ececf4e69dc30db9346800e1c54320f24671"
Nov 03 20:33:28.223017 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:33:28.222995    5570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5c7ec1d14a857d58b25629abd68ececf4e69dc30db9346800e1c54320f24671\": container with ID starting with e5c7ec1d14a857d58b25629abd68ececf4e69dc30db9346800e1c54320f24671 not found: ID does not exist" containerID="e5c7ec1d14a857d58b25629abd68ececf4e69dc30db9346800e1c54320f24671"
Nov 03 20:33:28.223094 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:28.223029    5570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5c7ec1d14a857d58b25629abd68ececf4e69dc30db9346800e1c54320f24671"} err="failed to get container status \"e5c7ec1d14a857d58b25629abd68ececf4e69dc30db9346800e1c54320f24671\": rpc error: code = NotFound desc = could not find container \"e5c7ec1d14a857d58b25629abd68ececf4e69dc30db9346800e1c54320f24671\": container with ID starting with e5c7ec1d14a857d58b25629abd68ececf4e69dc30db9346800e1c54320f24671 not found: ID does not exist"
Nov 03 20:33:28.223094 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:28.223048    5570 scope.go:122] "RemoveContainer" containerID="117061888c8f2f87481828900cffe3e409615fd7d0abfd26850b40bf839788ee"
Nov 03 20:33:28.223375 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:33:28.223352    5570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"117061888c8f2f87481828900cffe3e409615fd7d0abfd26850b40bf839788ee\": container with ID starting with 117061888c8f2f87481828900cffe3e409615fd7d0abfd26850b40bf839788ee not found: ID does not exist" containerID="117061888c8f2f87481828900cffe3e409615fd7d0abfd26850b40bf839788ee"
Nov 03 20:33:28.223437 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:28.223379    5570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"117061888c8f2f87481828900cffe3e409615fd7d0abfd26850b40bf839788ee"} err="failed to get container status \"117061888c8f2f87481828900cffe3e409615fd7d0abfd26850b40bf839788ee\": rpc error: code = NotFound desc = could not find container \"117061888c8f2f87481828900cffe3e409615fd7d0abfd26850b40bf839788ee\": container with ID starting with 117061888c8f2f87481828900cffe3e409615fd7d0abfd26850b40bf839788ee not found: ID does not exist"
Nov 03 20:33:28.223437 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:28.223394    5570 scope.go:122] "RemoveContainer" containerID="c35d6c2c4780ada7348150b2cfc4cd1948d3f225714d8b58219a774af050416b"
Nov 03 20:33:28.236477 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:28.236454    5570 scope.go:122] "RemoveContainer" containerID="c35d6c2c4780ada7348150b2cfc4cd1948d3f225714d8b58219a774af050416b"
Nov 03 20:33:28.267002 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:33:28.266954    5570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c35d6c2c4780ada7348150b2cfc4cd1948d3f225714d8b58219a774af050416b\": container with ID starting with c35d6c2c4780ada7348150b2cfc4cd1948d3f225714d8b58219a774af050416b not found: ID does not exist" containerID="c35d6c2c4780ada7348150b2cfc4cd1948d3f225714d8b58219a774af050416b"
Nov 03 20:33:28.267194 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:28.267010    5570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c35d6c2c4780ada7348150b2cfc4cd1948d3f225714d8b58219a774af050416b"} err="failed to get container status \"c35d6c2c4780ada7348150b2cfc4cd1948d3f225714d8b58219a774af050416b\": rpc error: code = NotFound desc = could not find container \"c35d6c2c4780ada7348150b2cfc4cd1948d3f225714d8b58219a774af050416b\": container with ID starting with c35d6c2c4780ada7348150b2cfc4cd1948d3f225714d8b58219a774af050416b not found: ID does not exist"
Nov 03 20:33:28.285462 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:28.285370    5570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22ea3ab0-6acc-4558-9158-eaa289924f35-istio-token\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:33:28.285462 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:28.285412    5570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2137488e-8e6c-46f7-a0ed-b4721d4f2645-istio-envoy\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:33:28.285462 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:28.285426    5570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae5a764e-25f7-492e-b4aa-8f2696e22f37-istiod-ca-cert\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:33:28.285462 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:28.285439    5570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2137488e-8e6c-46f7-a0ed-b4721d4f2645-istio-podinfo\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:33:28.285462 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:28.285453    5570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8aa4c48e-92c6-49c8-acd3-4bc441ac8ca7-workload-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:33:28.285462 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:28.285465    5570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae5a764e-25f7-492e-b4aa-8f2696e22f37-istio-envoy\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:33:28.285805 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:28.285476    5570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22ea3ab0-6acc-4558-9158-eaa289924f35-istio-envoy\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:33:28.285805 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:28.285488    5570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8aa4c48e-92c6-49c8-acd3-4bc441ac8ca7-istiod-ca-cert\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:33:28.285805 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:28.285501    5570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae5a764e-25f7-492e-b4aa-8f2696e22f37-workload-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:33:28.285805 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:28.285512    5570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8aa4c48e-92c6-49c8-acd3-4bc441ac8ca7-istio-envoy\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:33:28.285805 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:28.285524    5570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae5a764e-25f7-492e-b4aa-8f2696e22f37-istio-token\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:33:28.285805 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:28.285536    5570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22ea3ab0-6acc-4558-9158-eaa289924f35-istio-podinfo\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:33:28.285805 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:28.285549    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8nf4b\" (UniqueName: \"kubernetes.io/projected/2137488e-8e6c-46f7-a0ed-b4721d4f2645-kube-api-access-8nf4b\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:33:28.285805 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:28.285561    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmw5r\" (UniqueName: \"kubernetes.io/projected/8aa4c48e-92c6-49c8-acd3-4bc441ac8ca7-kube-api-access-xmw5r\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:33:28.285805 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:28.285574    5570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8aa4c48e-92c6-49c8-acd3-4bc441ac8ca7-istio-token\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:33:28.285805 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:28.285588    5570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae5a764e-25f7-492e-b4aa-8f2696e22f37-istio-podinfo\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:33:28.285805 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:28.285600    5570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae5a764e-25f7-492e-b4aa-8f2696e22f37-istio-data\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:33:28.285805 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:28.285611    5570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22ea3ab0-6acc-4558-9158-eaa289924f35-istio-data\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:33:28.285805 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:28.285622    5570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2137488e-8e6c-46f7-a0ed-b4721d4f2645-istio-token\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:33:28.285805 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:28.285635    5570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8aa4c48e-92c6-49c8-acd3-4bc441ac8ca7-istio-podinfo\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:33:28.285805 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:28.285646    5570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae5a764e-25f7-492e-b4aa-8f2696e22f37-workload-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:33:28.285805 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:28.285659    5570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae5a764e-25f7-492e-b4aa-8f2696e22f37-credential-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:33:28.285805 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:28.285673    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52h2d\" (UniqueName: \"kubernetes.io/projected/ae5a764e-25f7-492e-b4aa-8f2696e22f37-kube-api-access-52h2d\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:33:28.513338 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:28.513306    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-rbvqw"]
Nov 03 20:33:28.517246 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:28.517219    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-rbvqw"]
Nov 03 20:33:28.566191 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:28.566050    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-g262x"]
Nov 03 20:33:28.570960 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:28.570923    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-g262x"]
Nov 03 20:33:28.603660 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:28.603620    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-jshb8"]
Nov 03 20:33:28.606653 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:28.606619    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-jshb8"]
Nov 03 20:33:28.663716 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:28.663660    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-j7dz6"]
Nov 03 20:33:28.666657 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:28.666625    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-j7dz6"]
Nov 03 20:33:28.869864 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:28.869727    5570 patch_prober.go:29] interesting pod/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-g262x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:33:28.869864 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:28.869812    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w5-e3ba87-openshift-default-68f684648c-g262x" podUID="8aa4c48e-92c6-49c8-acd3-4bc441ac8ca7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:33:28.909831 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:28.909756    5570 patch_prober.go:29] interesting pod/e2e-inf-b-1d159a-openshift-default-799c8f9695-b8h67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body=
Nov 03 20:33:28.910058 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:28.909849    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-1d159a-openshift-default-799c8f9695-b8h67" podUID="40e4fc9c-c3c1-4aa0-98ad-eda347be89c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused"
Nov 03 20:33:29.909355 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:29.909319    5570 patch_prober.go:29] interesting pod/e2e-inf-b-1d159a-openshift-default-799c8f9695-b8h67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body=
Nov 03 20:33:29.909911 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:29.909383    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-1d159a-openshift-default-799c8f9695-b8h67" podUID="40e4fc9c-c3c1-4aa0-98ad-eda347be89c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused"
Nov 03 20:33:29.971180 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:29.971146    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2137488e-8e6c-46f7-a0ed-b4721d4f2645" path="/var/lib/kubelet/pods/2137488e-8e6c-46f7-a0ed-b4721d4f2645/volumes"
Nov 03 20:33:29.971694 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:29.971681    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22ea3ab0-6acc-4558-9158-eaa289924f35" path="/var/lib/kubelet/pods/22ea3ab0-6acc-4558-9158-eaa289924f35/volumes"
Nov 03 20:33:29.972213 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:29.972198    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8aa4c48e-92c6-49c8-acd3-4bc441ac8ca7" path="/var/lib/kubelet/pods/8aa4c48e-92c6-49c8-acd3-4bc441ac8ca7/volumes"
Nov 03 20:33:29.973049 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:29.973032    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae5a764e-25f7-492e-b4aa-8f2696e22f37" path="/var/lib/kubelet/pods/ae5a764e-25f7-492e-b4aa-8f2696e22f37/volumes"
Nov 03 20:33:30.869684 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:30.869648    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a9ba83-754d766d5d-nncw7"]
Nov 03 20:33:30.870315 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:30.870298    5570 cpu_manager.go:414] "containerMap: removing container" podUID="1f5aeb82-01f6-483f-8273-10bd0714e03c" containerName="istio-proxy"
Nov 03 20:33:30.870378 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:30.870317    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f5aeb82-01f6-483f-8273-10bd0714e03c" containerName="istio-proxy"
Nov 03 20:33:30.870378 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:30.870327    5570 cpu_manager.go:414] "containerMap: removing container" podUID="8aa4c48e-92c6-49c8-acd3-4bc441ac8ca7" containerName="istio-proxy"
Nov 03 20:33:30.870378 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:30.870334    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8aa4c48e-92c6-49c8-acd3-4bc441ac8ca7" containerName="istio-proxy"
Nov 03 20:33:30.870378 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:30.870347    5570 cpu_manager.go:414] "containerMap: removing container" podUID="2137488e-8e6c-46f7-a0ed-b4721d4f2645" containerName="istio-proxy"
Nov 03 20:33:30.870378 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:30.870352    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2137488e-8e6c-46f7-a0ed-b4721d4f2645" containerName="istio-proxy"
Nov 03 20:33:30.870378 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:30.870369    5570 cpu_manager.go:414] "containerMap: removing container" podUID="1f5aeb82-01f6-483f-8273-10bd0714e03c" containerName="istio-proxy"
Nov 03 20:33:30.870378 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:30.870374    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f5aeb82-01f6-483f-8273-10bd0714e03c" containerName="istio-proxy"
Nov 03 20:33:30.870601 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:30.870383    5570 cpu_manager.go:414] "containerMap: removing container" podUID="2137488e-8e6c-46f7-a0ed-b4721d4f2645" containerName="istio-proxy"
Nov 03 20:33:30.870601 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:30.870387    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2137488e-8e6c-46f7-a0ed-b4721d4f2645" containerName="istio-proxy"
Nov 03 20:33:30.870601 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:30.870398    5570 cpu_manager.go:414] "containerMap: removing container" podUID="22ea3ab0-6acc-4558-9158-eaa289924f35" containerName="istio-proxy"
Nov 03 20:33:30.870601 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:30.870402    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22ea3ab0-6acc-4558-9158-eaa289924f35" containerName="istio-proxy"
Nov 03 20:33:30.870601 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:30.870420    5570 cpu_manager.go:414] "containerMap: removing container" podUID="ae5a764e-25f7-492e-b4aa-8f2696e22f37" containerName="istio-proxy"
Nov 03 20:33:30.870601 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:30.870425    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae5a764e-25f7-492e-b4aa-8f2696e22f37" containerName="istio-proxy"
Nov 03 20:33:30.870601 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:30.870515    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f5aeb82-01f6-483f-8273-10bd0714e03c" containerName="istio-proxy"
Nov 03 20:33:30.870601 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:30.870525    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="2137488e-8e6c-46f7-a0ed-b4721d4f2645" containerName="istio-proxy"
Nov 03 20:33:30.870601 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:30.870533    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="22ea3ab0-6acc-4558-9158-eaa289924f35" containerName="istio-proxy"
Nov 03 20:33:30.870601 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:30.870541    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae5a764e-25f7-492e-b4aa-8f2696e22f37" containerName="istio-proxy"
Nov 03 20:33:30.870601 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:30.870550    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="8aa4c48e-92c6-49c8-acd3-4bc441ac8ca7" containerName="istio-proxy"
Nov 03 20:33:30.876027 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:30.876003    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a9ba83-754d766d5d-nncw7"
Nov 03 20:33:30.895902 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:30.895849    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a9ba83-754d766d5d-nncw7"]
Nov 03 20:33:30.907066 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:30.907036    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-a9ba83-serving-cert\""
Nov 03 20:33:30.913505 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:30.913467    5570 patch_prober.go:29] interesting pod/e2e-inf-b-1d159a-openshift-default-799c8f9695-b8h67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body=
Nov 03 20:33:30.914025 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:30.913535    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-1d159a-openshift-default-799c8f9695-b8h67" podUID="40e4fc9c-c3c1-4aa0-98ad-eda347be89c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused"
Nov 03 20:33:31.034259 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:31.034211    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf50af56-3e8b-4a3b-b871-5c882235c813-openshift-service-ca\") pod \"maas-api-e2e-inf-a-a9ba83-754d766d5d-nncw7\" (UID: \"bf50af56-3e8b-4a3b-b871-5c882235c813\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a9ba83-754d766d5d-nncw7"
Nov 03 20:33:31.034451 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:31.034282    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf50af56-3e8b-4a3b-b871-5c882235c813-metrics-certs\") pod \"maas-api-e2e-inf-a-a9ba83-754d766d5d-nncw7\" (UID: \"bf50af56-3e8b-4a3b-b871-5c882235c813\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a9ba83-754d766d5d-nncw7"
Nov 03 20:33:31.034451 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:31.034314    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khfjz\" (UniqueName: \"kubernetes.io/projected/bf50af56-3e8b-4a3b-b871-5c882235c813-kube-api-access-khfjz\") pod \"maas-api-e2e-inf-a-a9ba83-754d766d5d-nncw7\" (UID: \"bf50af56-3e8b-4a3b-b871-5c882235c813\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a9ba83-754d766d5d-nncw7"
Nov 03 20:33:31.034639 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:31.034618    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf50af56-3e8b-4a3b-b871-5c882235c813-maas-api-tls\") pod \"maas-api-e2e-inf-a-a9ba83-754d766d5d-nncw7\" (UID: \"bf50af56-3e8b-4a3b-b871-5c882235c813\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a9ba83-754d766d5d-nncw7"
Nov 03 20:33:31.146850 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:31.141758    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf50af56-3e8b-4a3b-b871-5c882235c813-maas-api-tls\") pod \"maas-api-e2e-inf-a-a9ba83-754d766d5d-nncw7\" (UID: \"bf50af56-3e8b-4a3b-b871-5c882235c813\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a9ba83-754d766d5d-nncw7"
Nov 03 20:33:31.146850 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:31.141989    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf50af56-3e8b-4a3b-b871-5c882235c813-openshift-service-ca\") pod \"maas-api-e2e-inf-a-a9ba83-754d766d5d-nncw7\" (UID: \"bf50af56-3e8b-4a3b-b871-5c882235c813\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a9ba83-754d766d5d-nncw7"
Nov 03 20:33:31.146850 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:31.142027    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf50af56-3e8b-4a3b-b871-5c882235c813-metrics-certs\") pod \"maas-api-e2e-inf-a-a9ba83-754d766d5d-nncw7\" (UID: \"bf50af56-3e8b-4a3b-b871-5c882235c813\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a9ba83-754d766d5d-nncw7"
Nov 03 20:33:31.146850 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:31.142061    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khfjz\" (UniqueName: \"kubernetes.io/projected/bf50af56-3e8b-4a3b-b871-5c882235c813-kube-api-access-khfjz\") pod \"maas-api-e2e-inf-a-a9ba83-754d766d5d-nncw7\" (UID: \"bf50af56-3e8b-4a3b-b871-5c882235c813\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a9ba83-754d766d5d-nncw7"
Nov 03 20:33:31.146850 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:31.143385    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf50af56-3e8b-4a3b-b871-5c882235c813-openshift-service-ca\") pod \"maas-api-e2e-inf-a-a9ba83-754d766d5d-nncw7\" (UID: \"bf50af56-3e8b-4a3b-b871-5c882235c813\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a9ba83-754d766d5d-nncw7"
Nov 03 20:33:31.168466 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:31.168427    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf50af56-3e8b-4a3b-b871-5c882235c813-maas-api-tls\") pod \"maas-api-e2e-inf-a-a9ba83-754d766d5d-nncw7\" (UID: \"bf50af56-3e8b-4a3b-b871-5c882235c813\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a9ba83-754d766d5d-nncw7"
Nov 03 20:33:31.169491 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:31.169464    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf50af56-3e8b-4a3b-b871-5c882235c813-metrics-certs\") pod \"maas-api-e2e-inf-a-a9ba83-754d766d5d-nncw7\" (UID: \"bf50af56-3e8b-4a3b-b871-5c882235c813\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a9ba83-754d766d5d-nncw7"
Nov 03 20:33:31.173419 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:31.173380    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khfjz\" (UniqueName: \"kubernetes.io/projected/bf50af56-3e8b-4a3b-b871-5c882235c813-kube-api-access-khfjz\") pod \"maas-api-e2e-inf-a-a9ba83-754d766d5d-nncw7\" (UID: \"bf50af56-3e8b-4a3b-b871-5c882235c813\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a9ba83-754d766d5d-nncw7"
Nov 03 20:33:31.192690 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:31.192660    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a9ba83-754d766d5d-nncw7"
Nov 03 20:33:31.537080 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:31.537039    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a9ba83-754d766d5d-nncw7"]
Nov 03 20:33:31.539509 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:33:31.539471    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf50af56_3e8b_4a3b_b871_5c882235c813.slice/crio-a54e45ef93c7166b3de04d9eacd8b4d15234ea159306c56e0b394acc9d1abcc4 WatchSource:0}: Error finding container a54e45ef93c7166b3de04d9eacd8b4d15234ea159306c56e0b394acc9d1abcc4: Status 404 returned error can't find the container with id a54e45ef93c7166b3de04d9eacd8b4d15234ea159306c56e0b394acc9d1abcc4
Nov 03 20:33:31.931421 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:31.931387    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-1d159a-openshift-default-799c8f9695-b8h67"
Nov 03 20:33:31.933287 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:31.933182    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-1d159a-openshift-default-799c8f9695-b8h67"
Nov 03 20:33:31.946747 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:31.946586    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-1d159a-openshift-default-799c8f9695-b8h67"
Nov 03 20:33:32.209381 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:32.209332    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a9ba83-754d766d5d-nncw7" event={"ID":"bf50af56-3e8b-4a3b-b871-5c882235c813","Type":"ContainerStarted","Data":"9ea8e68afdde1dbc16ab135a6e73e506bce67315790170fcf2326f8d640c9269"}
Nov 03 20:33:32.209607 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:32.209411    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a9ba83-754d766d5d-nncw7"
Nov 03 20:33:32.209607 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:32.209436    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a9ba83-754d766d5d-nncw7" event={"ID":"bf50af56-3e8b-4a3b-b871-5c882235c813","Type":"ContainerStarted","Data":"a54e45ef93c7166b3de04d9eacd8b4d15234ea159306c56e0b394acc9d1abcc4"}
Nov 03 20:33:32.254629 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:32.254571    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a9ba83-754d766d5d-nncw7" podStartSLOduration=1.955340853 podStartE2EDuration="2.254555592s" totalImagesPullingTime="299.214739ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:33:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:33:32.252870244 +0000 UTC m=+1746.405532128" watchObservedRunningTime="2026-11-03 20:33:32.254555592 +0000 UTC m=+1746.407217478"
Nov 03 20:33:34.511893 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:34.501688    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-60efdf/test-model-60efdf-kserve-7fb9cb65bf-9774z"
Nov 03 20:33:34.545348 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:34.540406    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-60efdf/test-model-60efdf-kserve-7fb9cb65bf-9774z"
Nov 03 20:33:35.245279 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:35.245247    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-60efdf/test-model-60efdf-kserve-7fb9cb65bf-9774z"
Nov 03 20:33:38.224376 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:38.224344    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a9ba83-754d766d5d-nncw7"
Nov 03 20:33:39.122135 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:39.122103    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d05604-5c69f8bc7-8cb6g"]
Nov 03 20:33:39.122334 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:39.122313    5570 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d05604-5c69f8bc7-8cb6g" podUID="55419b15-d4a1-4344-a5d2-e995ccbd2487" containerName="maas-api" containerID="cri-o://a1f50f02c65c3ea829bf7d26c4dc911c5d7b10280db8901df377c0c630197b22" gracePeriod=30
Nov 03 20:33:39.294416 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:39.294382    5570 generic.go:356] "Generic (PLEG): container finished" podID="55419b15-d4a1-4344-a5d2-e995ccbd2487" containerID="a1f50f02c65c3ea829bf7d26c4dc911c5d7b10280db8901df377c0c630197b22" exitCode=0
Nov 03 20:33:39.294852 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:39.294459    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d05604-5c69f8bc7-8cb6g" event={"ID":"55419b15-d4a1-4344-a5d2-e995ccbd2487","Type":"ContainerDied","Data":"a1f50f02c65c3ea829bf7d26c4dc911c5d7b10280db8901df377c0c630197b22"}
Nov 03 20:33:39.655542 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:39.655213    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d05604-5c69f8bc7-8cb6g"
Nov 03 20:33:39.733797 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:39.732761    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55419b15-d4a1-4344-a5d2-e995ccbd2487-kube-api-access-nxnbc\" (UniqueName: \"kubernetes.io/projected/55419b15-d4a1-4344-a5d2-e995ccbd2487-kube-api-access-nxnbc\") pod \"55419b15-d4a1-4344-a5d2-e995ccbd2487\" (UID: \"55419b15-d4a1-4344-a5d2-e995ccbd2487\") "
Nov 03 20:33:39.733797 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:39.733520    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55419b15-d4a1-4344-a5d2-e995ccbd2487-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55419b15-d4a1-4344-a5d2-e995ccbd2487-openshift-service-ca\") pod \"55419b15-d4a1-4344-a5d2-e995ccbd2487\" (UID: \"55419b15-d4a1-4344-a5d2-e995ccbd2487\") "
Nov 03 20:33:39.733797 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:39.733669    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/55419b15-d4a1-4344-a5d2-e995ccbd2487-metrics-certs\" (UniqueName: \"kubernetes.io/secret/55419b15-d4a1-4344-a5d2-e995ccbd2487-metrics-certs\") pod \"55419b15-d4a1-4344-a5d2-e995ccbd2487\" (UID: \"55419b15-d4a1-4344-a5d2-e995ccbd2487\") "
Nov 03 20:33:39.734133 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:39.733840    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/55419b15-d4a1-4344-a5d2-e995ccbd2487-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55419b15-d4a1-4344-a5d2-e995ccbd2487-maas-api-tls\") pod \"55419b15-d4a1-4344-a5d2-e995ccbd2487\" (UID: \"55419b15-d4a1-4344-a5d2-e995ccbd2487\") "
Nov 03 20:33:39.735956 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:39.735932    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55419b15-d4a1-4344-a5d2-e995ccbd2487-openshift-service-ca" pod "55419b15-d4a1-4344-a5d2-e995ccbd2487" (UID: "55419b15-d4a1-4344-a5d2-e995ccbd2487"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:33:39.738072 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:39.738028    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55419b15-d4a1-4344-a5d2-e995ccbd2487-maas-api-tls" pod "55419b15-d4a1-4344-a5d2-e995ccbd2487" (UID: "55419b15-d4a1-4344-a5d2-e995ccbd2487"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:33:39.738363 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:39.738337    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55419b15-d4a1-4344-a5d2-e995ccbd2487-metrics-certs" pod "55419b15-d4a1-4344-a5d2-e995ccbd2487" (UID: "55419b15-d4a1-4344-a5d2-e995ccbd2487"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:33:39.738560 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:39.738524    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55419b15-d4a1-4344-a5d2-e995ccbd2487-kube-api-access-nxnbc" pod "55419b15-d4a1-4344-a5d2-e995ccbd2487" (UID: "55419b15-d4a1-4344-a5d2-e995ccbd2487"). InnerVolumeSpecName "kube-api-access-nxnbc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:33:39.837231 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:39.837140    5570 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55419b15-d4a1-4344-a5d2-e995ccbd2487-maas-api-tls\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:33:39.837231 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:39.837171    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxnbc\" (UniqueName: \"kubernetes.io/projected/55419b15-d4a1-4344-a5d2-e995ccbd2487-kube-api-access-nxnbc\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:33:39.837231 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:39.837182    5570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55419b15-d4a1-4344-a5d2-e995ccbd2487-openshift-service-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:33:39.837231 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:39.837192    5570 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55419b15-d4a1-4344-a5d2-e995ccbd2487-metrics-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:33:40.060080 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:40.060035    5570 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-9973d9/premium-w2-9973d9-kserve-5c6dd4669c-5gv8x" podUID="cefc89ad-7ae1-433b-979d-6601678a6d98" containerName="main" containerID="cri-o://5621b0943b02c827ef26cd3d11392f6b882f3c7f2ffbbdb118669c90ff795faa" gracePeriod=45
Nov 03 20:33:40.099130 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:40.099033    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9973d9/premium-w2-9973d9-kserve-5c6dd4669c-5gv8x" podUID="cefc89ad-7ae1-433b-979d-6601678a6d98" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": dial tcp 10.134.0.54:8000: connect: connection refused"
Nov 03 20:33:40.304243 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:40.304206    5570 generic.go:356] "Generic (PLEG): container finished" podID="cefc89ad-7ae1-433b-979d-6601678a6d98" containerID="5621b0943b02c827ef26cd3d11392f6b882f3c7f2ffbbdb118669c90ff795faa" exitCode=0
Nov 03 20:33:40.304720 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:40.304270    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-9973d9/premium-w2-9973d9-kserve-5c6dd4669c-5gv8x" event={"ID":"cefc89ad-7ae1-433b-979d-6601678a6d98","Type":"ContainerDied","Data":"5621b0943b02c827ef26cd3d11392f6b882f3c7f2ffbbdb118669c90ff795faa"}
Nov 03 20:33:40.306063 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:40.306041    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d05604-5c69f8bc7-8cb6g" event={"ID":"55419b15-d4a1-4344-a5d2-e995ccbd2487","Type":"ContainerDied","Data":"9886a427233b83be189be394640239456b03a30538a88a984b3d5f5aeb25413d"}
Nov 03 20:33:40.306188 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:40.306061    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d05604-5c69f8bc7-8cb6g"
Nov 03 20:33:40.306188 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:40.306099    5570 scope.go:122] "RemoveContainer" containerID="a1f50f02c65c3ea829bf7d26c4dc911c5d7b10280db8901df377c0c630197b22"
Nov 03 20:33:40.361649 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:40.361568    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-9973d9/premium-w2-9973d9-kserve-5c6dd4669c-5gv8x"
Nov 03 20:33:40.364622 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:40.364590    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d05604-5c69f8bc7-8cb6g"]
Nov 03 20:33:40.378074 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:40.378039    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d05604-5c69f8bc7-8cb6g"]
Nov 03 20:33:40.445872 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:40.445834    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cefc89ad-7ae1-433b-979d-6601678a6d98-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cefc89ad-7ae1-433b-979d-6601678a6d98-tmp-dir\") pod \"cefc89ad-7ae1-433b-979d-6601678a6d98\" (UID: \"cefc89ad-7ae1-433b-979d-6601678a6d98\") "
Nov 03 20:33:40.446068 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:40.445996    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cefc89ad-7ae1-433b-979d-6601678a6d98-tls-certs\" (UniqueName: \"kubernetes.io/secret/cefc89ad-7ae1-433b-979d-6601678a6d98-tls-certs\") pod \"cefc89ad-7ae1-433b-979d-6601678a6d98\" (UID: \"cefc89ad-7ae1-433b-979d-6601678a6d98\") "
Nov 03 20:33:40.446068 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:40.446041    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cefc89ad-7ae1-433b-979d-6601678a6d98-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cefc89ad-7ae1-433b-979d-6601678a6d98-model-cache\") pod \"cefc89ad-7ae1-433b-979d-6601678a6d98\" (UID: \"cefc89ad-7ae1-433b-979d-6601678a6d98\") "
Nov 03 20:33:40.446211 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:40.446190    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cefc89ad-7ae1-433b-979d-6601678a6d98-home\" (UniqueName: \"kubernetes.io/empty-dir/cefc89ad-7ae1-433b-979d-6601678a6d98-home\") pod \"cefc89ad-7ae1-433b-979d-6601678a6d98\" (UID: \"cefc89ad-7ae1-433b-979d-6601678a6d98\") "
Nov 03 20:33:40.446283 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:40.446263    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cefc89ad-7ae1-433b-979d-6601678a6d98-kube-api-access-xqth8\" (UniqueName: \"kubernetes.io/projected/cefc89ad-7ae1-433b-979d-6601678a6d98-kube-api-access-xqth8\") pod \"cefc89ad-7ae1-433b-979d-6601678a6d98\" (UID: \"cefc89ad-7ae1-433b-979d-6601678a6d98\") "
Nov 03 20:33:40.446342 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:40.446288    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cefc89ad-7ae1-433b-979d-6601678a6d98-dshm\" (UniqueName: \"kubernetes.io/empty-dir/cefc89ad-7ae1-433b-979d-6601678a6d98-dshm\") pod \"cefc89ad-7ae1-433b-979d-6601678a6d98\" (UID: \"cefc89ad-7ae1-433b-979d-6601678a6d98\") "
Nov 03 20:33:40.446342 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:40.446308    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cefc89ad-7ae1-433b-979d-6601678a6d98-tmp-dir" pod "cefc89ad-7ae1-433b-979d-6601678a6d98" (UID: "cefc89ad-7ae1-433b-979d-6601678a6d98"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:33:40.446588 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:40.446564    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cefc89ad-7ae1-433b-979d-6601678a6d98-home" pod "cefc89ad-7ae1-433b-979d-6601678a6d98" (UID: "cefc89ad-7ae1-433b-979d-6601678a6d98"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:33:40.446862 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:40.446798    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cefc89ad-7ae1-433b-979d-6601678a6d98-model-cache" pod "cefc89ad-7ae1-433b-979d-6601678a6d98" (UID: "cefc89ad-7ae1-433b-979d-6601678a6d98"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:33:40.447118 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:40.447094    5570 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cefc89ad-7ae1-433b-979d-6601678a6d98-tmp-dir\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:33:40.447203 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:40.447123    5570 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cefc89ad-7ae1-433b-979d-6601678a6d98-model-cache\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:33:40.447203 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:40.447137    5570 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cefc89ad-7ae1-433b-979d-6601678a6d98-home\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:33:40.450107 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:40.449990    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cefc89ad-7ae1-433b-979d-6601678a6d98-kube-api-access-xqth8" pod "cefc89ad-7ae1-433b-979d-6601678a6d98" (UID: "cefc89ad-7ae1-433b-979d-6601678a6d98"). InnerVolumeSpecName "kube-api-access-xqth8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:33:40.453940 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:40.453912    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cefc89ad-7ae1-433b-979d-6601678a6d98-dshm" pod "cefc89ad-7ae1-433b-979d-6601678a6d98" (UID: "cefc89ad-7ae1-433b-979d-6601678a6d98"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:33:40.454065 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:40.453919    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cefc89ad-7ae1-433b-979d-6601678a6d98-tls-certs" pod "cefc89ad-7ae1-433b-979d-6601678a6d98" (UID: "cefc89ad-7ae1-433b-979d-6601678a6d98"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:33:40.549207 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:40.549171    5570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cefc89ad-7ae1-433b-979d-6601678a6d98-tls-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:33:40.549207 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:40.549209    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xqth8\" (UniqueName: \"kubernetes.io/projected/cefc89ad-7ae1-433b-979d-6601678a6d98-kube-api-access-xqth8\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:33:40.549496 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:40.549226    5570 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cefc89ad-7ae1-433b-979d-6601678a6d98-dshm\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:33:40.785305 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:40.785268    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-1d159a/test-model-1d159a-kserve-5766d9c789-lqs6f"]
Nov 03 20:33:40.785996 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:40.785975    5570 cpu_manager.go:414] "containerMap: removing container" podUID="cefc89ad-7ae1-433b-979d-6601678a6d98" containerName="main"
Nov 03 20:33:40.785996 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:40.785996    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cefc89ad-7ae1-433b-979d-6601678a6d98" containerName="main"
Nov 03 20:33:40.786192 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:40.786019    5570 cpu_manager.go:414] "containerMap: removing container" podUID="55419b15-d4a1-4344-a5d2-e995ccbd2487" containerName="maas-api"
Nov 03 20:33:40.786192 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:40.786026    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55419b15-d4a1-4344-a5d2-e995ccbd2487" containerName="maas-api"
Nov 03 20:33:40.786292 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:40.786211    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="cefc89ad-7ae1-433b-979d-6601678a6d98" containerName="main"
Nov 03 20:33:40.786292 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:40.786227    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f5aeb82-01f6-483f-8273-10bd0714e03c" containerName="istio-proxy"
Nov 03 20:33:40.786292 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:40.786242    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="2137488e-8e6c-46f7-a0ed-b4721d4f2645" containerName="istio-proxy"
Nov 03 20:33:40.786292 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:40.786253    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="55419b15-d4a1-4344-a5d2-e995ccbd2487" containerName="maas-api"
Nov 03 20:33:40.790051 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:40.790021    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-1d159a/test-model-1d159a-kserve-5766d9c789-lqs6f"
Nov 03 20:33:40.808682 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:40.808190    5570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-1d159a\"/\"kube-root-ca.crt\""
Nov 03 20:33:40.811210 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:40.811005    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-1d159a/test-model-1d159a-kserve-5766d9c789-lqs6f"]
Nov 03 20:33:40.817555 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:40.817530    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-1d159a\"/\"default-dockercfg-vxq5t\""
Nov 03 20:33:40.817804 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:40.817767    5570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-1d159a\"/\"openshift-service-ca.crt\""
Nov 03 20:33:40.817902 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:40.817825    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-1d159a\"/\"test-model-1d159a-kserve-self-signed-certs\""
Nov 03 20:33:40.852925 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:40.852888    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgrpl\" (UniqueName: \"kubernetes.io/projected/6a03d044-d7ba-4c94-b408-a7cdeaea9273-kube-api-access-fgrpl\") pod \"test-model-1d159a-kserve-5766d9c789-lqs6f\" (UID: \"6a03d044-d7ba-4c94-b408-a7cdeaea9273\") " pod="ai-tenant-e2e-inf-b-1d159a/test-model-1d159a-kserve-5766d9c789-lqs6f"
Nov 03 20:33:40.853140 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:40.853044    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a03d044-d7ba-4c94-b408-a7cdeaea9273-model-cache\") pod \"test-model-1d159a-kserve-5766d9c789-lqs6f\" (UID: \"6a03d044-d7ba-4c94-b408-a7cdeaea9273\") " pod="ai-tenant-e2e-inf-b-1d159a/test-model-1d159a-kserve-5766d9c789-lqs6f"
Nov 03 20:33:40.853140 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:40.853090    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a03d044-d7ba-4c94-b408-a7cdeaea9273-home\") pod \"test-model-1d159a-kserve-5766d9c789-lqs6f\" (UID: \"6a03d044-d7ba-4c94-b408-a7cdeaea9273\") " pod="ai-tenant-e2e-inf-b-1d159a/test-model-1d159a-kserve-5766d9c789-lqs6f"
Nov 03 20:33:40.853216 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:40.853173    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a03d044-d7ba-4c94-b408-a7cdeaea9273-tmp-dir\") pod \"test-model-1d159a-kserve-5766d9c789-lqs6f\" (UID: \"6a03d044-d7ba-4c94-b408-a7cdeaea9273\") " pod="ai-tenant-e2e-inf-b-1d159a/test-model-1d159a-kserve-5766d9c789-lqs6f"
Nov 03 20:33:40.853791 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:40.853223    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a03d044-d7ba-4c94-b408-a7cdeaea9273-dshm\") pod \"test-model-1d159a-kserve-5766d9c789-lqs6f\" (UID: \"6a03d044-d7ba-4c94-b408-a7cdeaea9273\") " pod="ai-tenant-e2e-inf-b-1d159a/test-model-1d159a-kserve-5766d9c789-lqs6f"
Nov 03 20:33:40.854033 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:40.853997    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a03d044-d7ba-4c94-b408-a7cdeaea9273-tls-certs\") pod \"test-model-1d159a-kserve-5766d9c789-lqs6f\" (UID: \"6a03d044-d7ba-4c94-b408-a7cdeaea9273\") " pod="ai-tenant-e2e-inf-b-1d159a/test-model-1d159a-kserve-5766d9c789-lqs6f"
Nov 03 20:33:40.955766 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:40.955726    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a03d044-d7ba-4c94-b408-a7cdeaea9273-model-cache\") pod \"test-model-1d159a-kserve-5766d9c789-lqs6f\" (UID: \"6a03d044-d7ba-4c94-b408-a7cdeaea9273\") " pod="ai-tenant-e2e-inf-b-1d159a/test-model-1d159a-kserve-5766d9c789-lqs6f"
Nov 03 20:33:40.956013 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:40.955770    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a03d044-d7ba-4c94-b408-a7cdeaea9273-home\") pod \"test-model-1d159a-kserve-5766d9c789-lqs6f\" (UID: \"6a03d044-d7ba-4c94-b408-a7cdeaea9273\") " pod="ai-tenant-e2e-inf-b-1d159a/test-model-1d159a-kserve-5766d9c789-lqs6f"
Nov 03 20:33:40.956013 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:40.955852    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a03d044-d7ba-4c94-b408-a7cdeaea9273-tmp-dir\") pod \"test-model-1d159a-kserve-5766d9c789-lqs6f\" (UID: \"6a03d044-d7ba-4c94-b408-a7cdeaea9273\") " pod="ai-tenant-e2e-inf-b-1d159a/test-model-1d159a-kserve-5766d9c789-lqs6f"
Nov 03 20:33:40.956013 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:40.955875    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a03d044-d7ba-4c94-b408-a7cdeaea9273-dshm\") pod \"test-model-1d159a-kserve-5766d9c789-lqs6f\" (UID: \"6a03d044-d7ba-4c94-b408-a7cdeaea9273\") " pod="ai-tenant-e2e-inf-b-1d159a/test-model-1d159a-kserve-5766d9c789-lqs6f"
Nov 03 20:33:40.956142 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:40.956056    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a03d044-d7ba-4c94-b408-a7cdeaea9273-tls-certs\") pod \"test-model-1d159a-kserve-5766d9c789-lqs6f\" (UID: \"6a03d044-d7ba-4c94-b408-a7cdeaea9273\") " pod="ai-tenant-e2e-inf-b-1d159a/test-model-1d159a-kserve-5766d9c789-lqs6f"
Nov 03 20:33:40.956227 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:40.956209    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgrpl\" (UniqueName: \"kubernetes.io/projected/6a03d044-d7ba-4c94-b408-a7cdeaea9273-kube-api-access-fgrpl\") pod \"test-model-1d159a-kserve-5766d9c789-lqs6f\" (UID: \"6a03d044-d7ba-4c94-b408-a7cdeaea9273\") " pod="ai-tenant-e2e-inf-b-1d159a/test-model-1d159a-kserve-5766d9c789-lqs6f"
Nov 03 20:33:40.956276 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:40.956252    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a03d044-d7ba-4c94-b408-a7cdeaea9273-model-cache\") pod \"test-model-1d159a-kserve-5766d9c789-lqs6f\" (UID: \"6a03d044-d7ba-4c94-b408-a7cdeaea9273\") " pod="ai-tenant-e2e-inf-b-1d159a/test-model-1d159a-kserve-5766d9c789-lqs6f"
Nov 03 20:33:40.956337 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:40.956269    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a03d044-d7ba-4c94-b408-a7cdeaea9273-tmp-dir\") pod \"test-model-1d159a-kserve-5766d9c789-lqs6f\" (UID: \"6a03d044-d7ba-4c94-b408-a7cdeaea9273\") " pod="ai-tenant-e2e-inf-b-1d159a/test-model-1d159a-kserve-5766d9c789-lqs6f"
Nov 03 20:33:40.956400 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:40.956333    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a03d044-d7ba-4c94-b408-a7cdeaea9273-home\") pod \"test-model-1d159a-kserve-5766d9c789-lqs6f\" (UID: \"6a03d044-d7ba-4c94-b408-a7cdeaea9273\") " pod="ai-tenant-e2e-inf-b-1d159a/test-model-1d159a-kserve-5766d9c789-lqs6f"
Nov 03 20:33:40.958284 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:40.958261    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a03d044-d7ba-4c94-b408-a7cdeaea9273-dshm\") pod \"test-model-1d159a-kserve-5766d9c789-lqs6f\" (UID: \"6a03d044-d7ba-4c94-b408-a7cdeaea9273\") " pod="ai-tenant-e2e-inf-b-1d159a/test-model-1d159a-kserve-5766d9c789-lqs6f"
Nov 03 20:33:40.958655 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:40.958637    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a03d044-d7ba-4c94-b408-a7cdeaea9273-tls-certs\") pod \"test-model-1d159a-kserve-5766d9c789-lqs6f\" (UID: \"6a03d044-d7ba-4c94-b408-a7cdeaea9273\") " pod="ai-tenant-e2e-inf-b-1d159a/test-model-1d159a-kserve-5766d9c789-lqs6f"
Nov 03 20:33:40.976322 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:40.976293    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgrpl\" (UniqueName: \"kubernetes.io/projected/6a03d044-d7ba-4c94-b408-a7cdeaea9273-kube-api-access-fgrpl\") pod \"test-model-1d159a-kserve-5766d9c789-lqs6f\" (UID: \"6a03d044-d7ba-4c94-b408-a7cdeaea9273\") " pod="ai-tenant-e2e-inf-b-1d159a/test-model-1d159a-kserve-5766d9c789-lqs6f"
Nov 03 20:33:41.110406 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:41.110069    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-1d159a/test-model-1d159a-kserve-5766d9c789-lqs6f"
Nov 03 20:33:41.358506 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:41.358478    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-9973d9/premium-w2-9973d9-kserve-5c6dd4669c-5gv8x"
Nov 03 20:33:41.359096 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:41.359063    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-9973d9/premium-w2-9973d9-kserve-5c6dd4669c-5gv8x" event={"ID":"cefc89ad-7ae1-433b-979d-6601678a6d98","Type":"ContainerDied","Data":"31a655f8b91204ee832414b270ae7631816d077d59bb13d5dc61658121746f10"}
Nov 03 20:33:41.359256 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:41.359242    5570 scope.go:122] "RemoveContainer" containerID="5621b0943b02c827ef26cd3d11392f6b882f3c7f2ffbbdb118669c90ff795faa"
Nov 03 20:33:41.435053 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:41.434998    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-9973d9/premium-w2-9973d9-kserve-5c6dd4669c-5gv8x"]
Nov 03 20:33:41.443723 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:41.443686    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-9973d9/premium-w2-9973d9-kserve-5c6dd4669c-5gv8x"]
Nov 03 20:33:41.607207 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:41.607177    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-1d159a/test-model-1d159a-kserve-5766d9c789-lqs6f"]
Nov 03 20:33:41.609638 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:33:41.609597    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6a03d044_d7ba_4c94_b408_a7cdeaea9273.slice/crio-da909e275ff2c6c652d883b8b3bf910af795054cb560fe5cede3571cf6a89943 WatchSource:0}: Error finding container da909e275ff2c6c652d883b8b3bf910af795054cb560fe5cede3571cf6a89943: Status 404 returned error can't find the container with id da909e275ff2c6c652d883b8b3bf910af795054cb560fe5cede3571cf6a89943
Nov 03 20:33:41.966046 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:41.966015    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55419b15-d4a1-4344-a5d2-e995ccbd2487" path="/var/lib/kubelet/pods/55419b15-d4a1-4344-a5d2-e995ccbd2487/volumes"
Nov 03 20:33:41.966560 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:41.966546    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cefc89ad-7ae1-433b-979d-6601678a6d98" path="/var/lib/kubelet/pods/cefc89ad-7ae1-433b-979d-6601678a6d98/volumes"
Nov 03 20:33:42.377716 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:42.377620    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-1d159a/test-model-1d159a-kserve-5766d9c789-lqs6f" event={"ID":"6a03d044-d7ba-4c94-b408-a7cdeaea9273","Type":"ContainerStarted","Data":"941895b3742e544fedb84438fe2779fa5a8a1143cc792b26b0ad7d80ef6b1915"}
Nov 03 20:33:42.377716 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:42.377665    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-1d159a/test-model-1d159a-kserve-5766d9c789-lqs6f" event={"ID":"6a03d044-d7ba-4c94-b408-a7cdeaea9273","Type":"ContainerStarted","Data":"da909e275ff2c6c652d883b8b3bf910af795054cb560fe5cede3571cf6a89943"}
Nov 03 20:33:42.420852 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:42.420788    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-1d159a/test-model-1d159a-kserve-5766d9c789-lqs6f" podStartSLOduration=2.420753426 podStartE2EDuration="2.420753426s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:33:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:33:42.419313991 +0000 UTC m=+1756.571975878" watchObservedRunningTime="2026-11-03 20:33:42.420753426 +0000 UTC m=+1756.573415316"
Nov 03 20:33:43.111701 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:43.111650    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-1d159a/test-model-1d159a-kserve-5766d9c789-lqs6f"
Nov 03 20:33:48.746537 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:48.746500    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-d05604-openshift-default-6bdffb549c-w62g5"]
Nov 03 20:33:48.747054 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:48.746800    5570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-d05604-openshift-default-6bdffb549c-w62g5" podUID="b9ec153a-ac81-4e70-8524-c3121c31c41f" containerName="istio-proxy" containerID="cri-o://e9716d133038c46e93015bdc1dbb0c3fd5a6381f966a6284449b58660e1465e2" gracePeriod=30
Nov 03 20:33:51.110923 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:51.110885    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-1d159a/test-model-1d159a-kserve-5766d9c789-lqs6f"
Nov 03 20:33:51.123611 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:51.123579    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-1d159a/test-model-1d159a-kserve-5766d9c789-lqs6f"
Nov 03 20:33:51.493829 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:51.493766    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-1d159a/test-model-1d159a-kserve-5766d9c789-lqs6f"
Nov 03 20:33:54.054080 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:54.054047    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d05604-openshift-default-6bdffb549c-w62g5"
Nov 03 20:33:54.188008 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:54.187957    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-e40816/distinct-2-w0-e40816-kserve-79fb46654d-s242x"]
Nov 03 20:33:54.193653 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:54.193619    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-e40816/simulator-w0-e40816-kserve-688f79796f-2tvw5"]
Nov 03 20:33:54.227108 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:54.227053    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9ec153a-ac81-4e70-8524-c3121c31c41f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9ec153a-ac81-4e70-8524-c3121c31c41f-istiod-ca-cert\") pod \"b9ec153a-ac81-4e70-8524-c3121c31c41f\" (UID: \"b9ec153a-ac81-4e70-8524-c3121c31c41f\") "
Nov 03 20:33:54.227305 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:54.227146    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9ec153a-ac81-4e70-8524-c3121c31c41f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9ec153a-ac81-4e70-8524-c3121c31c41f-credential-socket\") pod \"b9ec153a-ac81-4e70-8524-c3121c31c41f\" (UID: \"b9ec153a-ac81-4e70-8524-c3121c31c41f\") "
Nov 03 20:33:54.227305 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:54.227175    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9ec153a-ac81-4e70-8524-c3121c31c41f-kube-api-access-jzvxp\" (UniqueName: \"kubernetes.io/projected/b9ec153a-ac81-4e70-8524-c3121c31c41f-kube-api-access-jzvxp\") pod \"b9ec153a-ac81-4e70-8524-c3121c31c41f\" (UID: \"b9ec153a-ac81-4e70-8524-c3121c31c41f\") "
Nov 03 20:33:54.227305 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:54.227232    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b9ec153a-ac81-4e70-8524-c3121c31c41f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9ec153a-ac81-4e70-8524-c3121c31c41f-istio-podinfo\") pod \"b9ec153a-ac81-4e70-8524-c3121c31c41f\" (UID: \"b9ec153a-ac81-4e70-8524-c3121c31c41f\") "
Nov 03 20:33:54.227492 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:54.227331    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9ec153a-ac81-4e70-8524-c3121c31c41f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9ec153a-ac81-4e70-8524-c3121c31c41f-workload-certs\") pod \"b9ec153a-ac81-4e70-8524-c3121c31c41f\" (UID: \"b9ec153a-ac81-4e70-8524-c3121c31c41f\") "
Nov 03 20:33:54.227492 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:54.227431    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9ec153a-ac81-4e70-8524-c3121c31c41f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9ec153a-ac81-4e70-8524-c3121c31c41f-istio-data\") pod \"b9ec153a-ac81-4e70-8524-c3121c31c41f\" (UID: \"b9ec153a-ac81-4e70-8524-c3121c31c41f\") "
Nov 03 20:33:54.227492 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:54.227464    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9ec153a-ac81-4e70-8524-c3121c31c41f-credential-socket" pod "b9ec153a-ac81-4e70-8524-c3121c31c41f" (UID: "b9ec153a-ac81-4e70-8524-c3121c31c41f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:33:54.227492 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:54.227470    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9ec153a-ac81-4e70-8524-c3121c31c41f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9ec153a-ac81-4e70-8524-c3121c31c41f-workload-socket\") pod \"b9ec153a-ac81-4e70-8524-c3121c31c41f\" (UID: \"b9ec153a-ac81-4e70-8524-c3121c31c41f\") "
Nov 03 20:33:54.227787 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:54.227521    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9ec153a-ac81-4e70-8524-c3121c31c41f-istiod-ca-cert" pod "b9ec153a-ac81-4e70-8524-c3121c31c41f" (UID: "b9ec153a-ac81-4e70-8524-c3121c31c41f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:33:54.227787 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:54.227531    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9ec153a-ac81-4e70-8524-c3121c31c41f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9ec153a-ac81-4e70-8524-c3121c31c41f-istio-envoy\") pod \"b9ec153a-ac81-4e70-8524-c3121c31c41f\" (UID: \"b9ec153a-ac81-4e70-8524-c3121c31c41f\") "
Nov 03 20:33:54.227787 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:54.227672    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9ec153a-ac81-4e70-8524-c3121c31c41f-workload-socket" pod "b9ec153a-ac81-4e70-8524-c3121c31c41f" (UID: "b9ec153a-ac81-4e70-8524-c3121c31c41f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:33:54.227787 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:54.227721    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9ec153a-ac81-4e70-8524-c3121c31c41f-istio-token\" (UniqueName: \"kubernetes.io/projected/b9ec153a-ac81-4e70-8524-c3121c31c41f-istio-token\") pod \"b9ec153a-ac81-4e70-8524-c3121c31c41f\" (UID: \"b9ec153a-ac81-4e70-8524-c3121c31c41f\") "
Nov 03 20:33:54.228023 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:54.227857    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9ec153a-ac81-4e70-8524-c3121c31c41f-workload-certs" pod "b9ec153a-ac81-4e70-8524-c3121c31c41f" (UID: "b9ec153a-ac81-4e70-8524-c3121c31c41f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:33:54.228477 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:54.228387    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9ec153a-ac81-4e70-8524-c3121c31c41f-istio-data" pod "b9ec153a-ac81-4e70-8524-c3121c31c41f" (UID: "b9ec153a-ac81-4e70-8524-c3121c31c41f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:33:54.228608 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:54.228519    5570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9ec153a-ac81-4e70-8524-c3121c31c41f-workload-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:33:54.228608 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:54.228542    5570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9ec153a-ac81-4e70-8524-c3121c31c41f-workload-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:33:54.228608 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:54.228555    5570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9ec153a-ac81-4e70-8524-c3121c31c41f-istiod-ca-cert\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:33:54.228608 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:54.228568    5570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9ec153a-ac81-4e70-8524-c3121c31c41f-credential-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:33:54.230105 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:54.230077    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9ec153a-ac81-4e70-8524-c3121c31c41f-istio-token" pod "b9ec153a-ac81-4e70-8524-c3121c31c41f" (UID: "b9ec153a-ac81-4e70-8524-c3121c31c41f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:33:54.230300 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:54.230274    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9ec153a-ac81-4e70-8524-c3121c31c41f-istio-envoy" pod "b9ec153a-ac81-4e70-8524-c3121c31c41f" (UID: "b9ec153a-ac81-4e70-8524-c3121c31c41f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:33:54.230413 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:54.230299    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9ec153a-ac81-4e70-8524-c3121c31c41f-kube-api-access-jzvxp" pod "b9ec153a-ac81-4e70-8524-c3121c31c41f" (UID: "b9ec153a-ac81-4e70-8524-c3121c31c41f"). InnerVolumeSpecName "kube-api-access-jzvxp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:33:54.230466 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:54.230410    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b9ec153a-ac81-4e70-8524-c3121c31c41f-istio-podinfo" pod "b9ec153a-ac81-4e70-8524-c3121c31c41f" (UID: "b9ec153a-ac81-4e70-8524-c3121c31c41f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:33:54.330379 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:54.329546    5570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9ec153a-ac81-4e70-8524-c3121c31c41f-istio-podinfo\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:33:54.330379 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:54.329573    5570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9ec153a-ac81-4e70-8524-c3121c31c41f-istio-data\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:33:54.330379 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:54.329586    5570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9ec153a-ac81-4e70-8524-c3121c31c41f-istio-envoy\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:33:54.330379 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:54.329597    5570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9ec153a-ac81-4e70-8524-c3121c31c41f-istio-token\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:33:54.330379 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:54.329608    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzvxp\" (UniqueName: \"kubernetes.io/projected/b9ec153a-ac81-4e70-8524-c3121c31c41f-kube-api-access-jzvxp\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:33:54.576817 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:54.567955    5570 generic.go:356] "Generic (PLEG): container finished" podID="b9ec153a-ac81-4e70-8524-c3121c31c41f" containerID="e9716d133038c46e93015bdc1dbb0c3fd5a6381f966a6284449b58660e1465e2" exitCode=0
Nov 03 20:33:54.576817 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:54.568168    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d05604-openshift-default-6bdffb549c-w62g5" event={"ID":"b9ec153a-ac81-4e70-8524-c3121c31c41f","Type":"ContainerDied","Data":"e9716d133038c46e93015bdc1dbb0c3fd5a6381f966a6284449b58660e1465e2"}
Nov 03 20:33:54.576817 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:54.568223    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d05604-openshift-default-6bdffb549c-w62g5" event={"ID":"b9ec153a-ac81-4e70-8524-c3121c31c41f","Type":"ContainerDied","Data":"5d9aec411ccf53ae23211129b4d1f200eb4318a7c86e3c851d379201e1925059"}
Nov 03 20:33:54.576817 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:54.568252    5570 scope.go:122] "RemoveContainer" containerID="e9716d133038c46e93015bdc1dbb0c3fd5a6381f966a6284449b58660e1465e2"
Nov 03 20:33:54.576817 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:54.568537    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d05604-openshift-default-6bdffb549c-w62g5"
Nov 03 20:33:54.814891 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:54.811729    5570 scope.go:122] "RemoveContainer" containerID="ee06ea841d618a90e18d8c5fa9a485cb9edcb998c251762a4960ceceeef12d2b"
Nov 03 20:33:54.844109 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:54.844050    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-d05604-openshift-default-6bdffb549c-w62g5"]
Nov 03 20:33:54.852666 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:54.852622    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-d05604-openshift-default-6bdffb549c-w62g5"]
Nov 03 20:33:54.859481 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:54.859460    5570 scope.go:122] "RemoveContainer" containerID="e9716d133038c46e93015bdc1dbb0c3fd5a6381f966a6284449b58660e1465e2"
Nov 03 20:33:54.859887 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:33:54.859842    5570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9716d133038c46e93015bdc1dbb0c3fd5a6381f966a6284449b58660e1465e2\": container with ID starting with e9716d133038c46e93015bdc1dbb0c3fd5a6381f966a6284449b58660e1465e2 not found: ID does not exist" containerID="e9716d133038c46e93015bdc1dbb0c3fd5a6381f966a6284449b58660e1465e2"
Nov 03 20:33:54.860002 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:54.859893    5570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9716d133038c46e93015bdc1dbb0c3fd5a6381f966a6284449b58660e1465e2"} err="failed to get container status \"e9716d133038c46e93015bdc1dbb0c3fd5a6381f966a6284449b58660e1465e2\": rpc error: code = NotFound desc = could not find container \"e9716d133038c46e93015bdc1dbb0c3fd5a6381f966a6284449b58660e1465e2\": container with ID starting with e9716d133038c46e93015bdc1dbb0c3fd5a6381f966a6284449b58660e1465e2 not found: ID does not exist"
Nov 03 20:33:54.860002 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:54.859915    5570 scope.go:122] "RemoveContainer" containerID="ee06ea841d618a90e18d8c5fa9a485cb9edcb998c251762a4960ceceeef12d2b"
Nov 03 20:33:54.860213 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:33:54.860194    5570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee06ea841d618a90e18d8c5fa9a485cb9edcb998c251762a4960ceceeef12d2b\": container with ID starting with ee06ea841d618a90e18d8c5fa9a485cb9edcb998c251762a4960ceceeef12d2b not found: ID does not exist" containerID="ee06ea841d618a90e18d8c5fa9a485cb9edcb998c251762a4960ceceeef12d2b"
Nov 03 20:33:54.860252 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:54.860220    5570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee06ea841d618a90e18d8c5fa9a485cb9edcb998c251762a4960ceceeef12d2b"} err="failed to get container status \"ee06ea841d618a90e18d8c5fa9a485cb9edcb998c251762a4960ceceeef12d2b\": rpc error: code = NotFound desc = could not find container \"ee06ea841d618a90e18d8c5fa9a485cb9edcb998c251762a4960ceceeef12d2b\": container with ID starting with ee06ea841d618a90e18d8c5fa9a485cb9edcb998c251762a4960ceceeef12d2b not found: ID does not exist"
Nov 03 20:33:55.586687 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:55.586653    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-9e1bcb-c7dccc559-tbldl"]
Nov 03 20:33:55.587223 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:55.586953    5570 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-9e1bcb-c7dccc559-tbldl" podUID="cd3d374d-4721-4ccc-8283-dccefe85be90" containerName="maas-api" containerID="cri-o://1d8effea7f0c0f28c9dc1aa5c2f7c7b5d37e0207f9f40937e2246c1697181af4" gracePeriod=30
Nov 03 20:33:55.735410 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:55.735368    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-9e1bcb-c7dccc559-tbldl" podUID="cd3d374d-4721-4ccc-8283-dccefe85be90" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.46:8443/health\": dial tcp 10.134.0.46:8443: connect: connection refused"
Nov 03 20:33:56.009674 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:56.009634    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9ec153a-ac81-4e70-8524-c3121c31c41f" path="/var/lib/kubelet/pods/b9ec153a-ac81-4e70-8524-c3121c31c41f/volumes"
Nov 03 20:33:56.039494 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:56.039448    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-9e1bcb-c7dccc559-tbldl"
Nov 03 20:33:56.197614 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:56.197571    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd3d374d-4721-4ccc-8283-dccefe85be90-kube-api-access-jvm78\" (UniqueName: \"kubernetes.io/projected/cd3d374d-4721-4ccc-8283-dccefe85be90-kube-api-access-jvm78\") pod \"cd3d374d-4721-4ccc-8283-dccefe85be90\" (UID: \"cd3d374d-4721-4ccc-8283-dccefe85be90\") "
Nov 03 20:33:56.200797 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:56.198340    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd3d374d-4721-4ccc-8283-dccefe85be90-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd3d374d-4721-4ccc-8283-dccefe85be90-openshift-service-ca\") pod \"cd3d374d-4721-4ccc-8283-dccefe85be90\" (UID: \"cd3d374d-4721-4ccc-8283-dccefe85be90\") "
Nov 03 20:33:56.200797 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:56.198446    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd3d374d-4721-4ccc-8283-dccefe85be90-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd3d374d-4721-4ccc-8283-dccefe85be90-maas-api-tls\") pod \"cd3d374d-4721-4ccc-8283-dccefe85be90\" (UID: \"cd3d374d-4721-4ccc-8283-dccefe85be90\") "
Nov 03 20:33:56.200797 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:56.198489    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd3d374d-4721-4ccc-8283-dccefe85be90-metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd3d374d-4721-4ccc-8283-dccefe85be90-metrics-certs\") pod \"cd3d374d-4721-4ccc-8283-dccefe85be90\" (UID: \"cd3d374d-4721-4ccc-8283-dccefe85be90\") "
Nov 03 20:33:56.202061 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:56.201207    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd3d374d-4721-4ccc-8283-dccefe85be90-openshift-service-ca" pod "cd3d374d-4721-4ccc-8283-dccefe85be90" (UID: "cd3d374d-4721-4ccc-8283-dccefe85be90"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:33:56.206968 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:56.206908    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd3d374d-4721-4ccc-8283-dccefe85be90-metrics-certs" pod "cd3d374d-4721-4ccc-8283-dccefe85be90" (UID: "cd3d374d-4721-4ccc-8283-dccefe85be90"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:33:56.209009 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:56.208846    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd3d374d-4721-4ccc-8283-dccefe85be90-kube-api-access-jvm78" pod "cd3d374d-4721-4ccc-8283-dccefe85be90" (UID: "cd3d374d-4721-4ccc-8283-dccefe85be90"). InnerVolumeSpecName "kube-api-access-jvm78". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:33:56.210848 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:56.210805    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd3d374d-4721-4ccc-8283-dccefe85be90-maas-api-tls" pod "cd3d374d-4721-4ccc-8283-dccefe85be90" (UID: "cd3d374d-4721-4ccc-8283-dccefe85be90"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:33:56.300859 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:56.300816    5570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd3d374d-4721-4ccc-8283-dccefe85be90-openshift-service-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:33:56.300859 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:56.300853    5570 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd3d374d-4721-4ccc-8283-dccefe85be90-maas-api-tls\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:33:56.300859 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:56.300863    5570 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd3d374d-4721-4ccc-8283-dccefe85be90-metrics-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:33:56.301101 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:56.300871    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvm78\" (UniqueName: \"kubernetes.io/projected/cd3d374d-4721-4ccc-8283-dccefe85be90-kube-api-access-jvm78\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:33:56.595912 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:56.595409    5570 generic.go:356] "Generic (PLEG): container finished" podID="cd3d374d-4721-4ccc-8283-dccefe85be90" containerID="1d8effea7f0c0f28c9dc1aa5c2f7c7b5d37e0207f9f40937e2246c1697181af4" exitCode=0
Nov 03 20:33:56.595912 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:56.595623    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-9e1bcb-c7dccc559-tbldl" event={"ID":"cd3d374d-4721-4ccc-8283-dccefe85be90","Type":"ContainerDied","Data":"1d8effea7f0c0f28c9dc1aa5c2f7c7b5d37e0207f9f40937e2246c1697181af4"}
Nov 03 20:33:56.595912 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:56.595683    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-9e1bcb-c7dccc559-tbldl" event={"ID":"cd3d374d-4721-4ccc-8283-dccefe85be90","Type":"ContainerDied","Data":"2c08e6b58cedb3b90322891e3bb17564d3c85a561c90615151bed415a8d6ed31"}
Nov 03 20:33:56.595912 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:56.595717    5570 scope.go:122] "RemoveContainer" containerID="1d8effea7f0c0f28c9dc1aa5c2f7c7b5d37e0207f9f40937e2246c1697181af4"
Nov 03 20:33:56.596741 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:56.596684    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-9e1bcb-c7dccc559-tbldl"
Nov 03 20:33:56.651692 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:56.651650    5570 scope.go:122] "RemoveContainer" containerID="1d8effea7f0c0f28c9dc1aa5c2f7c7b5d37e0207f9f40937e2246c1697181af4"
Nov 03 20:33:56.652101 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:33:56.652072    5570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d8effea7f0c0f28c9dc1aa5c2f7c7b5d37e0207f9f40937e2246c1697181af4\": container with ID starting with 1d8effea7f0c0f28c9dc1aa5c2f7c7b5d37e0207f9f40937e2246c1697181af4 not found: ID does not exist" containerID="1d8effea7f0c0f28c9dc1aa5c2f7c7b5d37e0207f9f40937e2246c1697181af4"
Nov 03 20:33:56.652208 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:56.652110    5570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d8effea7f0c0f28c9dc1aa5c2f7c7b5d37e0207f9f40937e2246c1697181af4"} err="failed to get container status \"1d8effea7f0c0f28c9dc1aa5c2f7c7b5d37e0207f9f40937e2246c1697181af4\": rpc error: code = NotFound desc = could not find container \"1d8effea7f0c0f28c9dc1aa5c2f7c7b5d37e0207f9f40937e2246c1697181af4\": container with ID starting with 1d8effea7f0c0f28c9dc1aa5c2f7c7b5d37e0207f9f40937e2246c1697181af4 not found: ID does not exist"
Nov 03 20:33:56.695470 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:56.695428    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-9e1bcb-c7dccc559-tbldl"]
Nov 03 20:33:56.714763 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:56.714724    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-9e1bcb-c7dccc559-tbldl"]
Nov 03 20:33:57.229109 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:33:57.229059    5570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8bwhz" containerName="extractor"
Nov 03 20:33:57.966533 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:57.966499    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd3d374d-4721-4ccc-8283-dccefe85be90" path="/var/lib/kubelet/pods/cd3d374d-4721-4ccc-8283-dccefe85be90/volumes"
Nov 03 20:33:59.033799 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:59.026217    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g69f8"]
Nov 03 20:33:59.996852 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:33:59.996814    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-e40816-548465ddf5-mzdj7"]
Nov 03 20:34:01.713088 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:01.713039    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-4b95f3-7544494d7-9j6kp"]
Nov 03 20:34:01.713696 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:01.713678    5570 cpu_manager.go:414] "containerMap: removing container" podUID="b9ec153a-ac81-4e70-8524-c3121c31c41f" containerName="istio-proxy"
Nov 03 20:34:01.713696 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:01.713695    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9ec153a-ac81-4e70-8524-c3121c31c41f" containerName="istio-proxy"
Nov 03 20:34:01.713855 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:01.713716    5570 cpu_manager.go:414] "containerMap: removing container" podUID="cd3d374d-4721-4ccc-8283-dccefe85be90" containerName="maas-api"
Nov 03 20:34:01.713855 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:01.713721    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd3d374d-4721-4ccc-8283-dccefe85be90" containerName="maas-api"
Nov 03 20:34:01.713855 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:01.713733    5570 cpu_manager.go:414] "containerMap: removing container" podUID="b9ec153a-ac81-4e70-8524-c3121c31c41f" containerName="istio-proxy"
Nov 03 20:34:01.713855 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:01.713738    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9ec153a-ac81-4e70-8524-c3121c31c41f" containerName="istio-proxy"
Nov 03 20:34:01.713855 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:01.713857    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9ec153a-ac81-4e70-8524-c3121c31c41f" containerName="istio-proxy"
Nov 03 20:34:01.714133 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:01.713870    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd3d374d-4721-4ccc-8283-dccefe85be90" containerName="maas-api"
Nov 03 20:34:01.714133 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:01.713878    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9ec153a-ac81-4e70-8524-c3121c31c41f" containerName="istio-proxy"
Nov 03 20:34:01.718589 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:01.718562    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4b95f3-7544494d7-9j6kp"
Nov 03 20:34:01.736033 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:01.736002    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-4b95f3-7544494d7-9j6kp"]
Nov 03 20:34:01.764799 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:01.752685    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-4b95f3-dockercfg-brzsq\""
Nov 03 20:34:01.873798 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:01.871093    5570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-4b95f3\""
Nov 03 20:34:01.873798 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:01.872276    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4s7r7\" (UniqueName: \"kubernetes.io/projected/c32d9ae0-0cf8-4f64-ba57-c2197625231d-kube-api-access-4s7r7\") pod \"payload-pre-processing-e2e-inf-b-4b95f3-7544494d7-9j6kp\" (UID: \"c32d9ae0-0cf8-4f64-ba57-c2197625231d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4b95f3-7544494d7-9j6kp"
Nov 03 20:34:01.873798 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:01.872449    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c32d9ae0-0cf8-4f64-ba57-c2197625231d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-4b95f3-7544494d7-9j6kp\" (UID: \"c32d9ae0-0cf8-4f64-ba57-c2197625231d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4b95f3-7544494d7-9j6kp"
Nov 03 20:34:01.978867 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:01.977283    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c32d9ae0-0cf8-4f64-ba57-c2197625231d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-4b95f3-7544494d7-9j6kp\" (UID: \"c32d9ae0-0cf8-4f64-ba57-c2197625231d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4b95f3-7544494d7-9j6kp"
Nov 03 20:34:01.978867 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:01.977524    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4s7r7\" (UniqueName: \"kubernetes.io/projected/c32d9ae0-0cf8-4f64-ba57-c2197625231d-kube-api-access-4s7r7\") pod \"payload-pre-processing-e2e-inf-b-4b95f3-7544494d7-9j6kp\" (UID: \"c32d9ae0-0cf8-4f64-ba57-c2197625231d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4b95f3-7544494d7-9j6kp"
Nov 03 20:34:02.022819 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:02.022337    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c32d9ae0-0cf8-4f64-ba57-c2197625231d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-4b95f3-7544494d7-9j6kp\" (UID: \"c32d9ae0-0cf8-4f64-ba57-c2197625231d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4b95f3-7544494d7-9j6kp"
Nov 03 20:34:02.094698 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:02.094646    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-4b95f3-9dc7f8fb8-glb8b"]
Nov 03 20:34:02.099325 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:02.099136    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4s7r7\" (UniqueName: \"kubernetes.io/projected/c32d9ae0-0cf8-4f64-ba57-c2197625231d-kube-api-access-4s7r7\") pod \"payload-pre-processing-e2e-inf-b-4b95f3-7544494d7-9j6kp\" (UID: \"c32d9ae0-0cf8-4f64-ba57-c2197625231d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4b95f3-7544494d7-9j6kp"
Nov 03 20:34:02.108514 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:02.102619    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-4b95f3-9dc7f8fb8-glb8b"]
Nov 03 20:34:02.108514 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:02.102742    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-4b95f3-9dc7f8fb8-glb8b"
Nov 03 20:34:02.193171 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:02.193129    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n68g5\" (UniqueName: \"kubernetes.io/projected/7a27b292-b8fa-4c2c-baab-8a82994152bf-kube-api-access-n68g5\") pod \"payload-processing-e2e-inf-b-4b95f3-9dc7f8fb8-glb8b\" (UID: \"7a27b292-b8fa-4c2c-baab-8a82994152bf\") " pod="openshift-ingress/payload-processing-e2e-inf-b-4b95f3-9dc7f8fb8-glb8b"
Nov 03 20:34:02.193171 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:02.193172    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a27b292-b8fa-4c2c-baab-8a82994152bf-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-4b95f3-9dc7f8fb8-glb8b\" (UID: \"7a27b292-b8fa-4c2c-baab-8a82994152bf\") " pod="openshift-ingress/payload-processing-e2e-inf-b-4b95f3-9dc7f8fb8-glb8b"
Nov 03 20:34:02.294618 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:02.294526    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n68g5\" (UniqueName: \"kubernetes.io/projected/7a27b292-b8fa-4c2c-baab-8a82994152bf-kube-api-access-n68g5\") pod \"payload-processing-e2e-inf-b-4b95f3-9dc7f8fb8-glb8b\" (UID: \"7a27b292-b8fa-4c2c-baab-8a82994152bf\") " pod="openshift-ingress/payload-processing-e2e-inf-b-4b95f3-9dc7f8fb8-glb8b"
Nov 03 20:34:02.294618 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:02.294566    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a27b292-b8fa-4c2c-baab-8a82994152bf-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-4b95f3-9dc7f8fb8-glb8b\" (UID: \"7a27b292-b8fa-4c2c-baab-8a82994152bf\") " pod="openshift-ingress/payload-processing-e2e-inf-b-4b95f3-9dc7f8fb8-glb8b"
Nov 03 20:34:02.295597 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:02.295569    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a27b292-b8fa-4c2c-baab-8a82994152bf-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-4b95f3-9dc7f8fb8-glb8b\" (UID: \"7a27b292-b8fa-4c2c-baab-8a82994152bf\") " pod="openshift-ingress/payload-processing-e2e-inf-b-4b95f3-9dc7f8fb8-glb8b"
Nov 03 20:34:02.316169 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:02.316131    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n68g5\" (UniqueName: \"kubernetes.io/projected/7a27b292-b8fa-4c2c-baab-8a82994152bf-kube-api-access-n68g5\") pod \"payload-processing-e2e-inf-b-4b95f3-9dc7f8fb8-glb8b\" (UID: \"7a27b292-b8fa-4c2c-baab-8a82994152bf\") " pod="openshift-ingress/payload-processing-e2e-inf-b-4b95f3-9dc7f8fb8-glb8b"
Nov 03 20:34:02.334094 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:02.334057    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4b95f3-7544494d7-9j6kp"
Nov 03 20:34:02.482358 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:02.474162    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-4b95f3-9dc7f8fb8-glb8b"
Nov 03 20:34:02.675977 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:02.675931    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-4b95f3-7544494d7-9j6kp"]
Nov 03 20:34:02.871157 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:02.871066    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-4b95f3-9dc7f8fb8-glb8b"]
Nov 03 20:34:02.876399 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:34:02.876351    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a27b292_b8fa_4c2c_baab_8a82994152bf.slice/crio-ec79e53715382869bf5a9918f662c5077bad7bc9e07f5528a5e2914c892c06c3 WatchSource:0}: Error finding container ec79e53715382869bf5a9918f662c5077bad7bc9e07f5528a5e2914c892c06c3: Status 404 returned error can't find the container with id ec79e53715382869bf5a9918f662c5077bad7bc9e07f5528a5e2914c892c06c3
Nov 03 20:34:03.772084 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:03.772045    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-4b95f3-9dc7f8fb8-glb8b" event={"ID":"7a27b292-b8fa-4c2c-baab-8a82994152bf","Type":"ContainerStarted","Data":"c7cfdeab7d60a641bc464aecac52d90e1709f0d8cf8876a563825c911061129f"}
Nov 03 20:34:03.772323 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:03.772110    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-4b95f3-9dc7f8fb8-glb8b" event={"ID":"7a27b292-b8fa-4c2c-baab-8a82994152bf","Type":"ContainerStarted","Data":"ec79e53715382869bf5a9918f662c5077bad7bc9e07f5528a5e2914c892c06c3"}
Nov 03 20:34:03.772323 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:03.772157    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-4b95f3-9dc7f8fb8-glb8b"
Nov 03 20:34:03.774175 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:03.774147    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4b95f3-7544494d7-9j6kp" event={"ID":"c32d9ae0-0cf8-4f64-ba57-c2197625231d","Type":"ContainerStarted","Data":"02517b38a9d6adf9df62e9aa6ef1ae2b50b5ad31da8c677c3a4899186ee0d32e"}
Nov 03 20:34:03.774330 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:03.774195    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4b95f3-7544494d7-9j6kp" event={"ID":"c32d9ae0-0cf8-4f64-ba57-c2197625231d","Type":"ContainerStarted","Data":"ca9963b9ae5b8001e7f1f32a7ca965107b250c1bd7bdead358b20fcabc0d5658"}
Nov 03 20:34:03.774330 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:03.774240    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4b95f3-7544494d7-9j6kp"
Nov 03 20:34:03.822085 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:03.822011    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-4b95f3-9dc7f8fb8-glb8b" podStartSLOduration=2.821992251 podStartE2EDuration="2.821992251s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:34:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:34:03.818972473 +0000 UTC m=+1777.971634359" watchObservedRunningTime="2026-11-03 20:34:03.821992251 +0000 UTC m=+1777.974654137"
Nov 03 20:34:03.865511 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:03.865447    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4b95f3-7544494d7-9j6kp" podStartSLOduration=2.865426941 podStartE2EDuration="2.865426941s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:34:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:34:03.863928128 +0000 UTC m=+1778.016590018" watchObservedRunningTime="2026-11-03 20:34:03.865426941 +0000 UTC m=+1778.018088830"
Nov 03 20:34:04.997981 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:04.997899    5570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e40816-548465ddf5-mzdj7" podUID="94a03b52-b293-4475-8b23-fbbbae06fb26" containerName="payload-pre-processing" containerID="cri-o://e8cc79d0834cf7d7b1ed1f4463ba238eecea9b687962905be6718222cd5de000" gracePeriod=25
Nov 03 20:34:05.774389 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:05.774343    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-z4kzq"]
Nov 03 20:34:05.774951 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:05.774911    5570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-z4kzq" podUID="e185c296-cd84-4b0c-9316-ef10085cda0a" containerName="istio-proxy" containerID="cri-o://3a3fe6ad7dbd552ebf8f796694c6eddbbde0140bca4dd3da4fe06ba8b279f1bc" gracePeriod=30
Nov 03 20:34:06.181463 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:06.181359    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1cc51cc3-k5g2m"]
Nov 03 20:34:06.186149 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:06.186122    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cc51cc3-k5g2m"
Nov 03 20:34:06.207436 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:06.207398    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1cc51cc3-k5g2m"]
Nov 03 20:34:06.228213 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:06.228118    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:34:06.228536 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:06.228498    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l7s5w\""
Nov 03 20:34:06.368668 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:06.368626    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89266b89-4697-45b3-af99-c2dddf7034bc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1cc51cc3-k5g2m\" (UID: \"89266b89-4697-45b3-af99-c2dddf7034bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cc51cc3-k5g2m"
Nov 03 20:34:06.368919 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:06.368894    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89266b89-4697-45b3-af99-c2dddf7034bc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1cc51cc3-k5g2m\" (UID: \"89266b89-4697-45b3-af99-c2dddf7034bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cc51cc3-k5g2m"
Nov 03 20:34:06.368994 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:06.368951    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26zbb\" (UniqueName: \"kubernetes.io/projected/89266b89-4697-45b3-af99-c2dddf7034bc-kube-api-access-26zbb\") pod \"maas-api-revoke-sub-1cc51cc3-k5g2m\" (UID: \"89266b89-4697-45b3-af99-c2dddf7034bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cc51cc3-k5g2m"
Nov 03 20:34:06.503401 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:06.479377    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89266b89-4697-45b3-af99-c2dddf7034bc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1cc51cc3-k5g2m\" (UID: \"89266b89-4697-45b3-af99-c2dddf7034bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cc51cc3-k5g2m"
Nov 03 20:34:06.503401 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:06.479505    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26zbb\" (UniqueName: \"kubernetes.io/projected/89266b89-4697-45b3-af99-c2dddf7034bc-kube-api-access-26zbb\") pod \"maas-api-revoke-sub-1cc51cc3-k5g2m\" (UID: \"89266b89-4697-45b3-af99-c2dddf7034bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cc51cc3-k5g2m"
Nov 03 20:34:06.503401 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:06.479585    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89266b89-4697-45b3-af99-c2dddf7034bc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1cc51cc3-k5g2m\" (UID: \"89266b89-4697-45b3-af99-c2dddf7034bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cc51cc3-k5g2m"
Nov 03 20:34:06.513823 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:06.507930    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89266b89-4697-45b3-af99-c2dddf7034bc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1cc51cc3-k5g2m\" (UID: \"89266b89-4697-45b3-af99-c2dddf7034bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cc51cc3-k5g2m"
Nov 03 20:34:06.612736 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:06.612231    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26zbb\" (UniqueName: \"kubernetes.io/projected/89266b89-4697-45b3-af99-c2dddf7034bc-kube-api-access-26zbb\") pod \"maas-api-revoke-sub-1cc51cc3-k5g2m\" (UID: \"89266b89-4697-45b3-af99-c2dddf7034bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cc51cc3-k5g2m"
Nov 03 20:34:06.630796 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:06.628643    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89266b89-4697-45b3-af99-c2dddf7034bc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1cc51cc3-k5g2m\" (UID: \"89266b89-4697-45b3-af99-c2dddf7034bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cc51cc3-k5g2m"
Nov 03 20:34:06.812086 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:06.809379    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cc51cc3-k5g2m"
Nov 03 20:34:07.419613 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:07.419579    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1cc51cc3-k5g2m"]
Nov 03 20:34:07.420536 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:34:07.420506    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89266b89_4697_45b3_af99_c2dddf7034bc.slice/crio-c029071fa14fe6303f268a8ee5d60d5e3bd95ed3c790e6da7af3d29dbde71be3 WatchSource:0}: Error finding container c029071fa14fe6303f268a8ee5d60d5e3bd95ed3c790e6da7af3d29dbde71be3: Status 404 returned error can't find the container with id c029071fa14fe6303f268a8ee5d60d5e3bd95ed3c790e6da7af3d29dbde71be3
Nov 03 20:34:08.020396 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:08.020343    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cc51cc3-k5g2m" event={"ID":"89266b89-4697-45b3-af99-c2dddf7034bc","Type":"ContainerStarted","Data":"bd865b0cd67a42e1a3cc7f30194b6c6b2aca22f3624bc77b1841477d22ba873b"}
Nov 03 20:34:08.020582 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:08.020445    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cc51cc3-k5g2m" event={"ID":"89266b89-4697-45b3-af99-c2dddf7034bc","Type":"ContainerStarted","Data":"c029071fa14fe6303f268a8ee5d60d5e3bd95ed3c790e6da7af3d29dbde71be3"}
Nov 03 20:34:08.065936 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:08.065876    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cc51cc3-k5g2m" podStartSLOduration=2.065862253 podStartE2EDuration="2.065862253s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:34:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:34:08.065279215 +0000 UTC m=+1782.217941102" watchObservedRunningTime="2026-11-03 20:34:08.065862253 +0000 UTC m=+1782.218524138"
Nov 03 20:34:08.765031 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:08.764985    5570 patch_prober.go:29] interesting pod/e2e-inf-a-a9ba83-openshift-default-8c675fb59-486wc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:34:08.765525 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:08.765096    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-a9ba83-openshift-default-8c675fb59-486wc" podUID="f913fe36-7256-4171-ad6f-db25850da8ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:34:09.031944 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:09.031902    5570 generic.go:356] "Generic (PLEG): container finished" podID="89266b89-4697-45b3-af99-c2dddf7034bc" containerID="bd865b0cd67a42e1a3cc7f30194b6c6b2aca22f3624bc77b1841477d22ba873b" exitCode=0
Nov 03 20:34:09.032175 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:09.032156    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cc51cc3-k5g2m" event={"ID":"89266b89-4697-45b3-af99-c2dddf7034bc","Type":"ContainerDied","Data":"bd865b0cd67a42e1a3cc7f30194b6c6b2aca22f3624bc77b1841477d22ba873b"}
Nov 03 20:34:09.251799 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:09.230051    5570 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-e40816/distinct-2-w0-e40816-kserve-79fb46654d-s242x" podUID="4ad9b048-925c-4d01-a839-a5f25d4dd02d" containerName="main" containerID="cri-o://3b63087f2e199ac26d89e7d01f1f7e0ffe300e43c3dd7858cf7f07376f678516" gracePeriod=45
Nov 03 20:34:09.280205 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:09.277839    5570 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-e40816/simulator-w0-e40816-kserve-688f79796f-2tvw5" podUID="c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62" containerName="main" containerID="cri-o://a72316ce80e586f5136087ae6bd2eddc4062a714495080e83050487e5e330a81" gracePeriod=45
Nov 03 20:34:09.594279 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:09.594235    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e40816/distinct-2-w0-e40816-kserve-79fb46654d-s242x"
Nov 03 20:34:09.628390 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:09.628360    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e40816/simulator-w0-e40816-kserve-688f79796f-2tvw5"
Nov 03 20:34:09.719471 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:09.719434    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62-home\" (UniqueName: \"kubernetes.io/empty-dir/c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62-home\") pod \"c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62\" (UID: \"c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62\") "
Nov 03 20:34:09.719683 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:09.719560    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ad9b048-925c-4d01-a839-a5f25d4dd02d-home\" (UniqueName: \"kubernetes.io/empty-dir/4ad9b048-925c-4d01-a839-a5f25d4dd02d-home\") pod \"4ad9b048-925c-4d01-a839-a5f25d4dd02d\" (UID: \"4ad9b048-925c-4d01-a839-a5f25d4dd02d\") "
Nov 03 20:34:09.719683 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:09.719598    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ad9b048-925c-4d01-a839-a5f25d4dd02d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ad9b048-925c-4d01-a839-a5f25d4dd02d-dshm\") pod \"4ad9b048-925c-4d01-a839-a5f25d4dd02d\" (UID: \"4ad9b048-925c-4d01-a839-a5f25d4dd02d\") "
Nov 03 20:34:09.719683 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:09.719627    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62-dshm\") pod \"c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62\" (UID: \"c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62\") "
Nov 03 20:34:09.719683 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:09.719659    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62-model-cache\") pod \"c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62\" (UID: \"c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62\") "
Nov 03 20:34:09.720061 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:09.719716    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62-tmp-dir\") pod \"c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62\" (UID: \"c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62\") "
Nov 03 20:34:09.720061 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:09.719799    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62-home" pod "c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62" (UID: "c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:34:09.720061 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:09.719925    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62-kube-api-access-q7q7j\" (UniqueName: \"kubernetes.io/projected/c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62-kube-api-access-q7q7j\") pod \"c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62\" (UID: \"c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62\") "
Nov 03 20:34:09.720061 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:09.719971    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62-tls-certs\" (UniqueName: \"kubernetes.io/secret/c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62-tls-certs\") pod \"c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62\" (UID: \"c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62\") "
Nov 03 20:34:09.720326 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:09.720101    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ad9b048-925c-4d01-a839-a5f25d4dd02d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ad9b048-925c-4d01-a839-a5f25d4dd02d-tmp-dir\") pod \"4ad9b048-925c-4d01-a839-a5f25d4dd02d\" (UID: \"4ad9b048-925c-4d01-a839-a5f25d4dd02d\") "
Nov 03 20:34:09.720326 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:09.720157    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ad9b048-925c-4d01-a839-a5f25d4dd02d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ad9b048-925c-4d01-a839-a5f25d4dd02d-model-cache\") pod \"4ad9b048-925c-4d01-a839-a5f25d4dd02d\" (UID: \"4ad9b048-925c-4d01-a839-a5f25d4dd02d\") "
Nov 03 20:34:09.720326 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:09.720194    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ad9b048-925c-4d01-a839-a5f25d4dd02d-kube-api-access-924nf\" (UniqueName: \"kubernetes.io/projected/4ad9b048-925c-4d01-a839-a5f25d4dd02d-kube-api-access-924nf\") pod \"4ad9b048-925c-4d01-a839-a5f25d4dd02d\" (UID: \"4ad9b048-925c-4d01-a839-a5f25d4dd02d\") "
Nov 03 20:34:09.720326 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:09.720267    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4ad9b048-925c-4d01-a839-a5f25d4dd02d-tls-certs\" (UniqueName: \"kubernetes.io/secret/4ad9b048-925c-4d01-a839-a5f25d4dd02d-tls-certs\") pod \"4ad9b048-925c-4d01-a839-a5f25d4dd02d\" (UID: \"4ad9b048-925c-4d01-a839-a5f25d4dd02d\") "
Nov 03 20:34:09.720842 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:09.719926    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ad9b048-925c-4d01-a839-a5f25d4dd02d-home" pod "4ad9b048-925c-4d01-a839-a5f25d4dd02d" (UID: "4ad9b048-925c-4d01-a839-a5f25d4dd02d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:34:09.721155 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:09.721116    5570 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62-home\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:34:09.721269 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:09.721159    5570 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ad9b048-925c-4d01-a839-a5f25d4dd02d-home\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:34:09.721269 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:09.720048    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62-tmp-dir" pod "c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62" (UID: "c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:34:09.721269 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:09.720233    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62-model-cache" pod "c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62" (UID: "c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:34:09.722022 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:09.721964    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ad9b048-925c-4d01-a839-a5f25d4dd02d-tmp-dir" pod "4ad9b048-925c-4d01-a839-a5f25d4dd02d" (UID: "4ad9b048-925c-4d01-a839-a5f25d4dd02d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:34:09.722225 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:09.722191    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ad9b048-925c-4d01-a839-a5f25d4dd02d-model-cache" pod "4ad9b048-925c-4d01-a839-a5f25d4dd02d" (UID: "4ad9b048-925c-4d01-a839-a5f25d4dd02d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:34:09.723572 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:09.723546    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ad9b048-925c-4d01-a839-a5f25d4dd02d-tls-certs" pod "4ad9b048-925c-4d01-a839-a5f25d4dd02d" (UID: "4ad9b048-925c-4d01-a839-a5f25d4dd02d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:34:09.724076 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:09.724051    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ad9b048-925c-4d01-a839-a5f25d4dd02d-dshm" pod "4ad9b048-925c-4d01-a839-a5f25d4dd02d" (UID: "4ad9b048-925c-4d01-a839-a5f25d4dd02d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:34:09.724735 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:09.724706    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62-dshm" pod "c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62" (UID: "c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:34:09.724849 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:09.724758    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62-tls-certs" pod "c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62" (UID: "c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:34:09.725367 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:09.725335    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62-kube-api-access-q7q7j" pod "c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62" (UID: "c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62"). InnerVolumeSpecName "kube-api-access-q7q7j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:34:09.725538 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:09.725516    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ad9b048-925c-4d01-a839-a5f25d4dd02d-kube-api-access-924nf" pod "4ad9b048-925c-4d01-a839-a5f25d4dd02d" (UID: "4ad9b048-925c-4d01-a839-a5f25d4dd02d"). InnerVolumeSpecName "kube-api-access-924nf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:34:09.829877 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:09.826157    5570 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ad9b048-925c-4d01-a839-a5f25d4dd02d-dshm\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:34:09.829877 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:09.826199    5570 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62-dshm\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:34:09.829877 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:09.826213    5570 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62-model-cache\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:34:09.829877 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:09.826228    5570 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62-tmp-dir\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:34:09.829877 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:09.826241    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7q7j\" (UniqueName: \"kubernetes.io/projected/c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62-kube-api-access-q7q7j\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:34:09.829877 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:09.826254    5570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62-tls-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:34:09.829877 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:09.826266    5570 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ad9b048-925c-4d01-a839-a5f25d4dd02d-tmp-dir\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:34:09.829877 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:09.826276    5570 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ad9b048-925c-4d01-a839-a5f25d4dd02d-model-cache\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:34:09.829877 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:09.826289    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-924nf\" (UniqueName: \"kubernetes.io/projected/4ad9b048-925c-4d01-a839-a5f25d4dd02d-kube-api-access-924nf\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:34:09.829877 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:09.826301    5570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ad9b048-925c-4d01-a839-a5f25d4dd02d-tls-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:34:10.042403 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:10.042362    5570 generic.go:356] "Generic (PLEG): container finished" podID="c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62" containerID="a72316ce80e586f5136087ae6bd2eddc4062a714495080e83050487e5e330a81" exitCode=0
Nov 03 20:34:10.042603 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:10.042406    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e40816/simulator-w0-e40816-kserve-688f79796f-2tvw5"
Nov 03 20:34:10.042603 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:10.042441    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e40816/simulator-w0-e40816-kserve-688f79796f-2tvw5" event={"ID":"c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62","Type":"ContainerDied","Data":"a72316ce80e586f5136087ae6bd2eddc4062a714495080e83050487e5e330a81"}
Nov 03 20:34:10.042603 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:10.042488    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e40816/simulator-w0-e40816-kserve-688f79796f-2tvw5" event={"ID":"c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62","Type":"ContainerDied","Data":"8106f236cc15616f87882da99a0451d4d904a6dab96fedbadd78a1c6bff657dc"}
Nov 03 20:34:10.042603 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:10.042511    5570 scope.go:122] "RemoveContainer" containerID="a72316ce80e586f5136087ae6bd2eddc4062a714495080e83050487e5e330a81"
Nov 03 20:34:10.044357 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:10.044333    5570 generic.go:356] "Generic (PLEG): container finished" podID="4ad9b048-925c-4d01-a839-a5f25d4dd02d" containerID="3b63087f2e199ac26d89e7d01f1f7e0ffe300e43c3dd7858cf7f07376f678516" exitCode=0
Nov 03 20:34:10.044497 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:10.044400    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e40816/distinct-2-w0-e40816-kserve-79fb46654d-s242x"
Nov 03 20:34:10.044497 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:10.044440    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e40816/distinct-2-w0-e40816-kserve-79fb46654d-s242x" event={"ID":"4ad9b048-925c-4d01-a839-a5f25d4dd02d","Type":"ContainerDied","Data":"3b63087f2e199ac26d89e7d01f1f7e0ffe300e43c3dd7858cf7f07376f678516"}
Nov 03 20:34:10.044497 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:10.044473    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e40816/distinct-2-w0-e40816-kserve-79fb46654d-s242x" event={"ID":"4ad9b048-925c-4d01-a839-a5f25d4dd02d","Type":"ContainerDied","Data":"c0add56f78dc489b025b78d9f763d2c9a70e77b19c0546cd5e4551a48519dd49"}
Nov 03 20:34:10.054269 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:10.054249    5570 scope.go:122] "RemoveContainer" containerID="a72316ce80e586f5136087ae6bd2eddc4062a714495080e83050487e5e330a81"
Nov 03 20:34:10.054618 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:34:10.054589    5570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a72316ce80e586f5136087ae6bd2eddc4062a714495080e83050487e5e330a81\": container with ID starting with a72316ce80e586f5136087ae6bd2eddc4062a714495080e83050487e5e330a81 not found: ID does not exist" containerID="a72316ce80e586f5136087ae6bd2eddc4062a714495080e83050487e5e330a81"
Nov 03 20:34:10.054684 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:10.054629    5570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a72316ce80e586f5136087ae6bd2eddc4062a714495080e83050487e5e330a81"} err="failed to get container status \"a72316ce80e586f5136087ae6bd2eddc4062a714495080e83050487e5e330a81\": rpc error: code = NotFound desc = could not find container \"a72316ce80e586f5136087ae6bd2eddc4062a714495080e83050487e5e330a81\": container with ID starting with a72316ce80e586f5136087ae6bd2eddc4062a714495080e83050487e5e330a81 not found: ID does not exist"
Nov 03 20:34:10.054684 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:10.054651    5570 scope.go:122] "RemoveContainer" containerID="3b63087f2e199ac26d89e7d01f1f7e0ffe300e43c3dd7858cf7f07376f678516"
Nov 03 20:34:10.065696 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:10.065668    5570 scope.go:122] "RemoveContainer" containerID="3b63087f2e199ac26d89e7d01f1f7e0ffe300e43c3dd7858cf7f07376f678516"
Nov 03 20:34:10.066147 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:34:10.066123    5570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b63087f2e199ac26d89e7d01f1f7e0ffe300e43c3dd7858cf7f07376f678516\": container with ID starting with 3b63087f2e199ac26d89e7d01f1f7e0ffe300e43c3dd7858cf7f07376f678516 not found: ID does not exist" containerID="3b63087f2e199ac26d89e7d01f1f7e0ffe300e43c3dd7858cf7f07376f678516"
Nov 03 20:34:10.066242 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:10.066158    5570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b63087f2e199ac26d89e7d01f1f7e0ffe300e43c3dd7858cf7f07376f678516"} err="failed to get container status \"3b63087f2e199ac26d89e7d01f1f7e0ffe300e43c3dd7858cf7f07376f678516\": rpc error: code = NotFound desc = could not find container \"3b63087f2e199ac26d89e7d01f1f7e0ffe300e43c3dd7858cf7f07376f678516\": container with ID starting with 3b63087f2e199ac26d89e7d01f1f7e0ffe300e43c3dd7858cf7f07376f678516 not found: ID does not exist"
Nov 03 20:34:10.101894 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:10.101799    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-e40816/distinct-2-w0-e40816-kserve-79fb46654d-s242x"]
Nov 03 20:34:10.118397 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:10.118352    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-e40816/distinct-2-w0-e40816-kserve-79fb46654d-s242x"]
Nov 03 20:34:10.158012 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:10.157973    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-e40816/simulator-w0-e40816-kserve-688f79796f-2tvw5"]
Nov 03 20:34:10.165140 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:10.165101    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-e40816/simulator-w0-e40816-kserve-688f79796f-2tvw5"]
Nov 03 20:34:10.203365 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:10.203337    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cc51cc3-k5g2m"
Nov 03 20:34:10.333518 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:10.333471    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89266b89-4697-45b3-af99-c2dddf7034bc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89266b89-4697-45b3-af99-c2dddf7034bc-maas-api-openshift-ca\") pod \"89266b89-4697-45b3-af99-c2dddf7034bc\" (UID: \"89266b89-4697-45b3-af99-c2dddf7034bc\") "
Nov 03 20:34:10.333731 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:10.333597    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/89266b89-4697-45b3-af99-c2dddf7034bc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89266b89-4697-45b3-af99-c2dddf7034bc-maas-api-xks-ca\") pod \"89266b89-4697-45b3-af99-c2dddf7034bc\" (UID: \"89266b89-4697-45b3-af99-c2dddf7034bc\") "
Nov 03 20:34:10.333731 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:10.333668    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89266b89-4697-45b3-af99-c2dddf7034bc-kube-api-access-26zbb\" (UniqueName: \"kubernetes.io/projected/89266b89-4697-45b3-af99-c2dddf7034bc-kube-api-access-26zbb\") pod \"89266b89-4697-45b3-af99-c2dddf7034bc\" (UID: \"89266b89-4697-45b3-af99-c2dddf7034bc\") "
Nov 03 20:34:10.333988 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:10.333953    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89266b89-4697-45b3-af99-c2dddf7034bc-maas-api-openshift-ca" pod "89266b89-4697-45b3-af99-c2dddf7034bc" (UID: "89266b89-4697-45b3-af99-c2dddf7034bc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:34:10.334399 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:10.334374    5570 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89266b89-4697-45b3-af99-c2dddf7034bc-maas-api-openshift-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:34:10.335878 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:10.335836    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89266b89-4697-45b3-af99-c2dddf7034bc-kube-api-access-26zbb" pod "89266b89-4697-45b3-af99-c2dddf7034bc" (UID: "89266b89-4697-45b3-af99-c2dddf7034bc"). InnerVolumeSpecName "kube-api-access-26zbb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:34:10.336219 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:10.336202    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89266b89-4697-45b3-af99-c2dddf7034bc-maas-api-xks-ca" pod "89266b89-4697-45b3-af99-c2dddf7034bc" (UID: "89266b89-4697-45b3-af99-c2dddf7034bc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:34:10.435894 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:10.435767    5570 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89266b89-4697-45b3-af99-c2dddf7034bc-maas-api-xks-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:34:10.435894 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:10.435816    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26zbb\" (UniqueName: \"kubernetes.io/projected/89266b89-4697-45b3-af99-c2dddf7034bc-kube-api-access-26zbb\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:34:10.538073 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:10.538019    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e40816/distinct-2-w0-e40816-kserve-79fb46654d-s242x" podUID="4ad9b048-925c-4d01-a839-a5f25d4dd02d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:34:11.059735 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:11.059703    5570 generic.go:356] "Generic (PLEG): container finished" podID="e185c296-cd84-4b0c-9316-ef10085cda0a" containerID="3a3fe6ad7dbd552ebf8f796694c6eddbbde0140bca4dd3da4fe06ba8b279f1bc" exitCode=0
Nov 03 20:34:11.060301 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:11.059838    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-z4kzq" event={"ID":"e185c296-cd84-4b0c-9316-ef10085cda0a","Type":"ContainerDied","Data":"3a3fe6ad7dbd552ebf8f796694c6eddbbde0140bca4dd3da4fe06ba8b279f1bc"}
Nov 03 20:34:11.061465 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:11.061399    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cc51cc3-k5g2m"
Nov 03 20:34:11.061465 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:11.061427    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cc51cc3-k5g2m" event={"ID":"89266b89-4697-45b3-af99-c2dddf7034bc","Type":"ContainerDied","Data":"c029071fa14fe6303f268a8ee5d60d5e3bd95ed3c790e6da7af3d29dbde71be3"}
Nov 03 20:34:11.061654 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:11.061474    5570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c029071fa14fe6303f268a8ee5d60d5e3bd95ed3c790e6da7af3d29dbde71be3"
Nov 03 20:34:11.079817 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:11.079782    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-z4kzq"
Nov 03 20:34:11.256747 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:11.256711    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e185c296-cd84-4b0c-9316-ef10085cda0a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e185c296-cd84-4b0c-9316-ef10085cda0a-istio-podinfo\") pod \"e185c296-cd84-4b0c-9316-ef10085cda0a\" (UID: \"e185c296-cd84-4b0c-9316-ef10085cda0a\") "
Nov 03 20:34:11.258235 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:11.256798    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e185c296-cd84-4b0c-9316-ef10085cda0a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e185c296-cd84-4b0c-9316-ef10085cda0a-workload-certs\") pod \"e185c296-cd84-4b0c-9316-ef10085cda0a\" (UID: \"e185c296-cd84-4b0c-9316-ef10085cda0a\") "
Nov 03 20:34:11.258235 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:11.256885    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e185c296-cd84-4b0c-9316-ef10085cda0a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e185c296-cd84-4b0c-9316-ef10085cda0a-istiod-ca-cert\") pod \"e185c296-cd84-4b0c-9316-ef10085cda0a\" (UID: \"e185c296-cd84-4b0c-9316-ef10085cda0a\") "
Nov 03 20:34:11.258235 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:11.256915    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e185c296-cd84-4b0c-9316-ef10085cda0a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e185c296-cd84-4b0c-9316-ef10085cda0a-workload-socket\") pod \"e185c296-cd84-4b0c-9316-ef10085cda0a\" (UID: \"e185c296-cd84-4b0c-9316-ef10085cda0a\") "
Nov 03 20:34:11.258235 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:11.256989    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e185c296-cd84-4b0c-9316-ef10085cda0a-kube-api-access-j4m4g\" (UniqueName: \"kubernetes.io/projected/e185c296-cd84-4b0c-9316-ef10085cda0a-kube-api-access-j4m4g\") pod \"e185c296-cd84-4b0c-9316-ef10085cda0a\" (UID: \"e185c296-cd84-4b0c-9316-ef10085cda0a\") "
Nov 03 20:34:11.258235 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:11.257084    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e185c296-cd84-4b0c-9316-ef10085cda0a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e185c296-cd84-4b0c-9316-ef10085cda0a-credential-socket\") pod \"e185c296-cd84-4b0c-9316-ef10085cda0a\" (UID: \"e185c296-cd84-4b0c-9316-ef10085cda0a\") "
Nov 03 20:34:11.258235 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:11.257211    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e185c296-cd84-4b0c-9316-ef10085cda0a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e185c296-cd84-4b0c-9316-ef10085cda0a-istio-data\") pod \"e185c296-cd84-4b0c-9316-ef10085cda0a\" (UID: \"e185c296-cd84-4b0c-9316-ef10085cda0a\") "
Nov 03 20:34:11.258235 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:11.257250    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e185c296-cd84-4b0c-9316-ef10085cda0a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e185c296-cd84-4b0c-9316-ef10085cda0a-istio-envoy\") pod \"e185c296-cd84-4b0c-9316-ef10085cda0a\" (UID: \"e185c296-cd84-4b0c-9316-ef10085cda0a\") "
Nov 03 20:34:11.258235 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:11.257277    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e185c296-cd84-4b0c-9316-ef10085cda0a-istio-token\" (UniqueName: \"kubernetes.io/projected/e185c296-cd84-4b0c-9316-ef10085cda0a-istio-token\") pod \"e185c296-cd84-4b0c-9316-ef10085cda0a\" (UID: \"e185c296-cd84-4b0c-9316-ef10085cda0a\") "
Nov 03 20:34:11.258235 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:11.257429    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e185c296-cd84-4b0c-9316-ef10085cda0a-workload-certs" pod "e185c296-cd84-4b0c-9316-ef10085cda0a" (UID: "e185c296-cd84-4b0c-9316-ef10085cda0a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:34:11.258235 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:11.257432    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e185c296-cd84-4b0c-9316-ef10085cda0a-istiod-ca-cert" pod "e185c296-cd84-4b0c-9316-ef10085cda0a" (UID: "e185c296-cd84-4b0c-9316-ef10085cda0a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:34:11.258235 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:11.257714    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e185c296-cd84-4b0c-9316-ef10085cda0a-istio-data" pod "e185c296-cd84-4b0c-9316-ef10085cda0a" (UID: "e185c296-cd84-4b0c-9316-ef10085cda0a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:34:11.258235 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:11.258025    5570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e185c296-cd84-4b0c-9316-ef10085cda0a-workload-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:34:11.258235 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:11.258062    5570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e185c296-cd84-4b0c-9316-ef10085cda0a-istiod-ca-cert\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:34:11.258235 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:11.258075    5570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e185c296-cd84-4b0c-9316-ef10085cda0a-istio-data\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:34:11.259092 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:11.259051    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e185c296-cd84-4b0c-9316-ef10085cda0a-workload-socket" pod "e185c296-cd84-4b0c-9316-ef10085cda0a" (UID: "e185c296-cd84-4b0c-9316-ef10085cda0a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:34:11.259639 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:11.259610    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e185c296-cd84-4b0c-9316-ef10085cda0a-credential-socket" pod "e185c296-cd84-4b0c-9316-ef10085cda0a" (UID: "e185c296-cd84-4b0c-9316-ef10085cda0a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:34:11.261750 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:11.261712    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e185c296-cd84-4b0c-9316-ef10085cda0a-istio-token" pod "e185c296-cd84-4b0c-9316-ef10085cda0a" (UID: "e185c296-cd84-4b0c-9316-ef10085cda0a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:34:11.261958 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:11.261905    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e185c296-cd84-4b0c-9316-ef10085cda0a-istio-envoy" pod "e185c296-cd84-4b0c-9316-ef10085cda0a" (UID: "e185c296-cd84-4b0c-9316-ef10085cda0a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:34:11.267286 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:11.267259    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e185c296-cd84-4b0c-9316-ef10085cda0a-kube-api-access-j4m4g" pod "e185c296-cd84-4b0c-9316-ef10085cda0a" (UID: "e185c296-cd84-4b0c-9316-ef10085cda0a"). InnerVolumeSpecName "kube-api-access-j4m4g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:34:11.267437 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:11.267284    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e185c296-cd84-4b0c-9316-ef10085cda0a-istio-podinfo" pod "e185c296-cd84-4b0c-9316-ef10085cda0a" (UID: "e185c296-cd84-4b0c-9316-ef10085cda0a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:34:11.359038 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:11.359003    5570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e185c296-cd84-4b0c-9316-ef10085cda0a-workload-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:34:11.359038 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:11.359032    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4m4g\" (UniqueName: \"kubernetes.io/projected/e185c296-cd84-4b0c-9316-ef10085cda0a-kube-api-access-j4m4g\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:34:11.359038 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:11.359042    5570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e185c296-cd84-4b0c-9316-ef10085cda0a-credential-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:34:11.381123 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:11.359052    5570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e185c296-cd84-4b0c-9316-ef10085cda0a-istio-envoy\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:34:11.381123 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:11.359060    5570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e185c296-cd84-4b0c-9316-ef10085cda0a-istio-token\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:34:11.381123 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:11.359067    5570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e185c296-cd84-4b0c-9316-ef10085cda0a-istio-podinfo\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:34:11.965843 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:11.965807    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ad9b048-925c-4d01-a839-a5f25d4dd02d" path="/var/lib/kubelet/pods/4ad9b048-925c-4d01-a839-a5f25d4dd02d/volumes"
Nov 03 20:34:11.966254 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:11.966240    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62" path="/var/lib/kubelet/pods/c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62/volumes"
Nov 03 20:34:12.071500 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:12.071463    5570 generic.go:356] "Generic (PLEG): container finished" podID="f913fe36-7256-4171-ad6f-db25850da8ce" containerID="edeb6af257049566a198a0ed1235fc9c416060dc50a5fd85246d010da5f12ea0" exitCode=0
Nov 03 20:34:12.072018 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:12.071537    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a9ba83-openshift-default-8c675fb59-486wc" event={"ID":"f913fe36-7256-4171-ad6f-db25850da8ce","Type":"ContainerDied","Data":"edeb6af257049566a198a0ed1235fc9c416060dc50a5fd85246d010da5f12ea0"}
Nov 03 20:34:12.072084 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:12.072069    5570 scope.go:122] "RemoveContainer" containerID="edeb6af257049566a198a0ed1235fc9c416060dc50a5fd85246d010da5f12ea0"
Nov 03 20:34:12.073321 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:12.073303    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-z4kzq" event={"ID":"e185c296-cd84-4b0c-9316-ef10085cda0a","Type":"ContainerDied","Data":"9a68094e8967403ffb6b2b568f761790b1fad27597c978030ff8c5badc3d1588"}
Nov 03 20:34:12.073444 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:12.073323    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-z4kzq"
Nov 03 20:34:12.073444 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:12.073352    5570 scope.go:122] "RemoveContainer" containerID="3a3fe6ad7dbd552ebf8f796694c6eddbbde0140bca4dd3da4fe06ba8b279f1bc"
Nov 03 20:34:12.073878 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:12.073841    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:34:12.073969 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:12.073935    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:34:12.074016 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:12.073982    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:34:12.178465 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:12.177035    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-z4kzq"]
Nov 03 20:34:12.178673 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:12.178526    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-9e1bcb-openshift-default-55bccc4c8f-z4kzq"]
Nov 03 20:34:12.927730 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:12.927690    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e40816-548465ddf5-mzdj7" podUID="94a03b52-b293-4475-8b23-fbbbae06fb26" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 20:34:13.084366 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:13.084327    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a9ba83-openshift-default-8c675fb59-486wc" event={"ID":"f913fe36-7256-4171-ad6f-db25850da8ce","Type":"ContainerStarted","Data":"8f879707251cdde1af594cce61f974e2b60d0b8a7e3f888f52f804d150f436ac"}
Nov 03 20:34:13.834821 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:13.834736    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-a9ba83-openshift-default-8c675fb59-486wc"
Nov 03 20:34:13.835165 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:13.835134    5570 patch_prober.go:29] interesting pod/e2e-inf-a-a9ba83-openshift-default-8c675fb59-486wc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body=
Nov 03 20:34:13.835284 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:13.835192    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a9ba83-openshift-default-8c675fb59-486wc" podUID="f913fe36-7256-4171-ad6f-db25850da8ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused"
Nov 03 20:34:13.969490 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:13.969449    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e185c296-cd84-4b0c-9316-ef10085cda0a" path="/var/lib/kubelet/pods/e185c296-cd84-4b0c-9316-ef10085cda0a/volumes"
Nov 03 20:34:14.281911 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:14.280792    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-nz52s"]
Nov 03 20:34:14.281911 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:14.281020    5570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-nz52s" podUID="974badc1-41ad-461c-8719-e1f3f836ccbc" containerName="istio-proxy" containerID="cri-o://f907ae94459b2d9e4d50bb2b9a2f8edfa7b72411d9525c0dd6685e0bad46decd" gracePeriod=30
Nov 03 20:34:14.802830 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:14.802768    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-4b95f3-9dc7f8fb8-glb8b"
Nov 03 20:34:14.803031 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:14.802878    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4b95f3-7544494d7-9j6kp"
Nov 03 20:34:14.834899 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:14.834862    5570 patch_prober.go:29] interesting pod/e2e-inf-a-a9ba83-openshift-default-8c675fb59-486wc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body=
Nov 03 20:34:14.835077 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:14.834920    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a9ba83-openshift-default-8c675fb59-486wc" podUID="f913fe36-7256-4171-ad6f-db25850da8ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused"
Nov 03 20:34:15.022220 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:15.022159    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-c5nf4"]
Nov 03 20:34:15.022852 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:15.022744    5570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-c5nf4" podUID="8a255596-5346-460f-b437-7c48d3d33d54" containerName="istio-proxy" containerID="cri-o://a7bb15afee0634b9ec32b88804532afc13e276d3c21985b7c080bb3aa350a328" gracePeriod=30
Nov 03 20:34:15.835062 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:15.835021    5570 patch_prober.go:29] interesting pod/e2e-inf-a-a9ba83-openshift-default-8c675fb59-486wc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body=
Nov 03 20:34:15.835493 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:15.835107    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a9ba83-openshift-default-8c675fb59-486wc" podUID="f913fe36-7256-4171-ad6f-db25850da8ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused"
Nov 03 20:34:16.835854 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:16.835810    5570 patch_prober.go:29] interesting pod/e2e-inf-a-a9ba83-openshift-default-8c675fb59-486wc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body=
Nov 03 20:34:16.836397 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:16.835914    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a9ba83-openshift-default-8c675fb59-486wc" podUID="f913fe36-7256-4171-ad6f-db25850da8ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused"
Nov 03 20:34:16.922666 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:16.922628    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-jjjr7"]
Nov 03 20:34:16.922882 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:16.922855    5570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-jjjr7" podUID="7ca8412e-c6de-4cb5-944e-ba8afd2fa791" containerName="istio-proxy" containerID="cri-o://0e87fdc31bf499c3143805dba02ed2b9d49780fdf96d7d02f51272999a98282d" gracePeriod=30
Nov 03 20:34:17.834926 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:17.834891    5570 patch_prober.go:29] interesting pod/e2e-inf-a-a9ba83-openshift-default-8c675fb59-486wc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body=
Nov 03 20:34:17.835136 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:17.834946    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a9ba83-openshift-default-8c675fb59-486wc" podUID="f913fe36-7256-4171-ad6f-db25850da8ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused"
Nov 03 20:34:18.840298 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:18.840265    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-a9ba83-openshift-default-8c675fb59-486wc"
Nov 03 20:34:18.840853 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:18.840669    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-a9ba83-openshift-default-8c675fb59-486wc"
Nov 03 20:34:18.841513 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:18.841492    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-a9ba83-openshift-default-8c675fb59-486wc"
Nov 03 20:34:19.586287 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:19.586253    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-nz52s"
Nov 03 20:34:19.787798 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:19.756255    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/974badc1-41ad-461c-8719-e1f3f836ccbc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/974badc1-41ad-461c-8719-e1f3f836ccbc-credential-socket\") pod \"974badc1-41ad-461c-8719-e1f3f836ccbc\" (UID: \"974badc1-41ad-461c-8719-e1f3f836ccbc\") "
Nov 03 20:34:19.787798 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:19.756316    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/974badc1-41ad-461c-8719-e1f3f836ccbc-istio-token\" (UniqueName: \"kubernetes.io/projected/974badc1-41ad-461c-8719-e1f3f836ccbc-istio-token\") pod \"974badc1-41ad-461c-8719-e1f3f836ccbc\" (UID: \"974badc1-41ad-461c-8719-e1f3f836ccbc\") "
Nov 03 20:34:19.787798 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:19.756373    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/974badc1-41ad-461c-8719-e1f3f836ccbc-kube-api-access-tmnc2\" (UniqueName: \"kubernetes.io/projected/974badc1-41ad-461c-8719-e1f3f836ccbc-kube-api-access-tmnc2\") pod \"974badc1-41ad-461c-8719-e1f3f836ccbc\" (UID: \"974badc1-41ad-461c-8719-e1f3f836ccbc\") "
Nov 03 20:34:19.793658 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:19.788376    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/974badc1-41ad-461c-8719-e1f3f836ccbc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/974badc1-41ad-461c-8719-e1f3f836ccbc-istiod-ca-cert\") pod \"974badc1-41ad-461c-8719-e1f3f836ccbc\" (UID: \"974badc1-41ad-461c-8719-e1f3f836ccbc\") "
Nov 03 20:34:19.793658 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:19.788523    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/974badc1-41ad-461c-8719-e1f3f836ccbc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/974badc1-41ad-461c-8719-e1f3f836ccbc-workload-socket\") pod \"974badc1-41ad-461c-8719-e1f3f836ccbc\" (UID: \"974badc1-41ad-461c-8719-e1f3f836ccbc\") "
Nov 03 20:34:19.793658 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:19.788587    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/974badc1-41ad-461c-8719-e1f3f836ccbc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/974badc1-41ad-461c-8719-e1f3f836ccbc-istio-data\") pod \"974badc1-41ad-461c-8719-e1f3f836ccbc\" (UID: \"974badc1-41ad-461c-8719-e1f3f836ccbc\") "
Nov 03 20:34:19.793658 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:19.788665    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/974badc1-41ad-461c-8719-e1f3f836ccbc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/974badc1-41ad-461c-8719-e1f3f836ccbc-istio-envoy\") pod \"974badc1-41ad-461c-8719-e1f3f836ccbc\" (UID: \"974badc1-41ad-461c-8719-e1f3f836ccbc\") "
Nov 03 20:34:19.793658 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:19.788754    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/974badc1-41ad-461c-8719-e1f3f836ccbc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/974badc1-41ad-461c-8719-e1f3f836ccbc-istio-podinfo\") pod \"974badc1-41ad-461c-8719-e1f3f836ccbc\" (UID: \"974badc1-41ad-461c-8719-e1f3f836ccbc\") "
Nov 03 20:34:19.793658 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:19.788847    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/974badc1-41ad-461c-8719-e1f3f836ccbc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/974badc1-41ad-461c-8719-e1f3f836ccbc-workload-certs\") pod \"974badc1-41ad-461c-8719-e1f3f836ccbc\" (UID: \"974badc1-41ad-461c-8719-e1f3f836ccbc\") "
Nov 03 20:34:19.794264 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:19.794232    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/974badc1-41ad-461c-8719-e1f3f836ccbc-credential-socket" pod "974badc1-41ad-461c-8719-e1f3f836ccbc" (UID: "974badc1-41ad-461c-8719-e1f3f836ccbc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:34:19.795542 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:19.795345    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/974badc1-41ad-461c-8719-e1f3f836ccbc-istio-data" pod "974badc1-41ad-461c-8719-e1f3f836ccbc" (UID: "974badc1-41ad-461c-8719-e1f3f836ccbc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:34:19.795898 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:19.795876    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/974badc1-41ad-461c-8719-e1f3f836ccbc-istiod-ca-cert" pod "974badc1-41ad-461c-8719-e1f3f836ccbc" (UID: "974badc1-41ad-461c-8719-e1f3f836ccbc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:34:19.802414 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:19.796600    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/974badc1-41ad-461c-8719-e1f3f836ccbc-workload-socket" pod "974badc1-41ad-461c-8719-e1f3f836ccbc" (UID: "974badc1-41ad-461c-8719-e1f3f836ccbc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:34:19.802414 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:19.797110    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/974badc1-41ad-461c-8719-e1f3f836ccbc-workload-certs" pod "974badc1-41ad-461c-8719-e1f3f836ccbc" (UID: "974badc1-41ad-461c-8719-e1f3f836ccbc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:34:19.820997 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:19.819949    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/974badc1-41ad-461c-8719-e1f3f836ccbc-istio-podinfo" pod "974badc1-41ad-461c-8719-e1f3f836ccbc" (UID: "974badc1-41ad-461c-8719-e1f3f836ccbc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:34:19.823076 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:19.823045    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/974badc1-41ad-461c-8719-e1f3f836ccbc-istio-token" pod "974badc1-41ad-461c-8719-e1f3f836ccbc" (UID: "974badc1-41ad-461c-8719-e1f3f836ccbc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:34:19.824395 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:19.824290    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/974badc1-41ad-461c-8719-e1f3f836ccbc-istio-envoy" pod "974badc1-41ad-461c-8719-e1f3f836ccbc" (UID: "974badc1-41ad-461c-8719-e1f3f836ccbc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:34:19.824395 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:19.824323    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/974badc1-41ad-461c-8719-e1f3f836ccbc-kube-api-access-tmnc2" pod "974badc1-41ad-461c-8719-e1f3f836ccbc" (UID: "974badc1-41ad-461c-8719-e1f3f836ccbc"). InnerVolumeSpecName "kube-api-access-tmnc2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:34:19.891198 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:19.891152    5570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/974badc1-41ad-461c-8719-e1f3f836ccbc-credential-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:34:19.891198 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:19.891189    5570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/974badc1-41ad-461c-8719-e1f3f836ccbc-istio-token\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:34:19.891198 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:19.891202    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmnc2\" (UniqueName: \"kubernetes.io/projected/974badc1-41ad-461c-8719-e1f3f836ccbc-kube-api-access-tmnc2\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:34:19.891736 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:19.891215    5570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/974badc1-41ad-461c-8719-e1f3f836ccbc-istiod-ca-cert\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:34:19.891736 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:19.891229    5570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/974badc1-41ad-461c-8719-e1f3f836ccbc-workload-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:34:19.891736 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:19.891239    5570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/974badc1-41ad-461c-8719-e1f3f836ccbc-istio-data\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:34:19.891736 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:19.891249    5570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/974badc1-41ad-461c-8719-e1f3f836ccbc-istio-envoy\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:34:19.891736 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:19.891259    5570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/974badc1-41ad-461c-8719-e1f3f836ccbc-istio-podinfo\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:34:19.891736 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:19.891270    5570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/974badc1-41ad-461c-8719-e1f3f836ccbc-workload-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:34:20.223955 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:20.223911    5570 generic.go:356] "Generic (PLEG): container finished" podID="974badc1-41ad-461c-8719-e1f3f836ccbc" containerID="f907ae94459b2d9e4d50bb2b9a2f8edfa7b72411d9525c0dd6685e0bad46decd" exitCode=0
Nov 03 20:34:20.224134 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:20.223991    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-nz52s"
Nov 03 20:34:20.224134 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:20.223991    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-nz52s" event={"ID":"974badc1-41ad-461c-8719-e1f3f836ccbc","Type":"ContainerDied","Data":"f907ae94459b2d9e4d50bb2b9a2f8edfa7b72411d9525c0dd6685e0bad46decd"}
Nov 03 20:34:20.224134 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:20.224106    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-nz52s" event={"ID":"974badc1-41ad-461c-8719-e1f3f836ccbc","Type":"ContainerDied","Data":"a84b94fa7ae49c10475785130e3bd1d9c18b45c504b01c0d1035a5901ce6f050"}
Nov 03 20:34:20.224281 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:20.224140    5570 scope.go:122] "RemoveContainer" containerID="f907ae94459b2d9e4d50bb2b9a2f8edfa7b72411d9525c0dd6685e0bad46decd"
Nov 03 20:34:20.227425 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:20.227398    5570 generic.go:356] "Generic (PLEG): container finished" podID="8a255596-5346-460f-b437-7c48d3d33d54" containerID="a7bb15afee0634b9ec32b88804532afc13e276d3c21985b7c080bb3aa350a328" exitCode=0
Nov 03 20:34:20.227569 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:20.227454    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-c5nf4" event={"ID":"8a255596-5346-460f-b437-7c48d3d33d54","Type":"ContainerDied","Data":"a7bb15afee0634b9ec32b88804532afc13e276d3c21985b7c080bb3aa350a328"}
Nov 03 20:34:20.238187 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:20.238130    5570 scope.go:122] "RemoveContainer" containerID="55d50dbccd68644d558fe9e9ce5c19730dc0d11c8b11b77a7e28d9c0dfce32ee"
Nov 03 20:34:20.251215 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:20.251194    5570 scope.go:122] "RemoveContainer" containerID="f907ae94459b2d9e4d50bb2b9a2f8edfa7b72411d9525c0dd6685e0bad46decd"
Nov 03 20:34:20.251621 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:34:20.251594    5570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f907ae94459b2d9e4d50bb2b9a2f8edfa7b72411d9525c0dd6685e0bad46decd\": container with ID starting with f907ae94459b2d9e4d50bb2b9a2f8edfa7b72411d9525c0dd6685e0bad46decd not found: ID does not exist" containerID="f907ae94459b2d9e4d50bb2b9a2f8edfa7b72411d9525c0dd6685e0bad46decd"
Nov 03 20:34:20.251758 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:20.251626    5570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f907ae94459b2d9e4d50bb2b9a2f8edfa7b72411d9525c0dd6685e0bad46decd"} err="failed to get container status \"f907ae94459b2d9e4d50bb2b9a2f8edfa7b72411d9525c0dd6685e0bad46decd\": rpc error: code = NotFound desc = could not find container \"f907ae94459b2d9e4d50bb2b9a2f8edfa7b72411d9525c0dd6685e0bad46decd\": container with ID starting with f907ae94459b2d9e4d50bb2b9a2f8edfa7b72411d9525c0dd6685e0bad46decd not found: ID does not exist"
Nov 03 20:34:20.251758 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:20.251646    5570 scope.go:122] "RemoveContainer" containerID="55d50dbccd68644d558fe9e9ce5c19730dc0d11c8b11b77a7e28d9c0dfce32ee"
Nov 03 20:34:20.252220 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:34:20.252197    5570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55d50dbccd68644d558fe9e9ce5c19730dc0d11c8b11b77a7e28d9c0dfce32ee\": container with ID starting with 55d50dbccd68644d558fe9e9ce5c19730dc0d11c8b11b77a7e28d9c0dfce32ee not found: ID does not exist" containerID="55d50dbccd68644d558fe9e9ce5c19730dc0d11c8b11b77a7e28d9c0dfce32ee"
Nov 03 20:34:20.252370 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:20.252224    5570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55d50dbccd68644d558fe9e9ce5c19730dc0d11c8b11b77a7e28d9c0dfce32ee"} err="failed to get container status \"55d50dbccd68644d558fe9e9ce5c19730dc0d11c8b11b77a7e28d9c0dfce32ee\": rpc error: code = NotFound desc = could not find container \"55d50dbccd68644d558fe9e9ce5c19730dc0d11c8b11b77a7e28d9c0dfce32ee\": container with ID starting with 55d50dbccd68644d558fe9e9ce5c19730dc0d11c8b11b77a7e28d9c0dfce32ee not found: ID does not exist"
Nov 03 20:34:20.252370 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:20.252243    5570 scope.go:122] "RemoveContainer" containerID="425dede10b05299e0b57616be19f61783c0772db42f2d9151301ed07b22f477f"
Nov 03 20:34:20.292968 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:20.292931    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-nz52s"]
Nov 03 20:34:20.301935 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:20.301905    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-c5nf4"
Nov 03 20:34:20.305761 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:20.305732    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-9973d9-openshift-default-ffb6475-nz52s"]
Nov 03 20:34:20.398412 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:20.398317    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a255596-5346-460f-b437-7c48d3d33d54-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a255596-5346-460f-b437-7c48d3d33d54-istiod-ca-cert\") pod \"8a255596-5346-460f-b437-7c48d3d33d54\" (UID: \"8a255596-5346-460f-b437-7c48d3d33d54\") "
Nov 03 20:34:20.398412 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:20.398359    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a255596-5346-460f-b437-7c48d3d33d54-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a255596-5346-460f-b437-7c48d3d33d54-istio-data\") pod \"8a255596-5346-460f-b437-7c48d3d33d54\" (UID: \"8a255596-5346-460f-b437-7c48d3d33d54\") "
Nov 03 20:34:20.398659 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:20.398445    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a255596-5346-460f-b437-7c48d3d33d54-kube-api-access-56ng4\" (UniqueName: \"kubernetes.io/projected/8a255596-5346-460f-b437-7c48d3d33d54-kube-api-access-56ng4\") pod \"8a255596-5346-460f-b437-7c48d3d33d54\" (UID: \"8a255596-5346-460f-b437-7c48d3d33d54\") "
Nov 03 20:34:20.398659 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:20.398586    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a255596-5346-460f-b437-7c48d3d33d54-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a255596-5346-460f-b437-7c48d3d33d54-credential-socket\") pod \"8a255596-5346-460f-b437-7c48d3d33d54\" (UID: \"8a255596-5346-460f-b437-7c48d3d33d54\") "
Nov 03 20:34:20.398659 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:20.398611    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8a255596-5346-460f-b437-7c48d3d33d54-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a255596-5346-460f-b437-7c48d3d33d54-istio-podinfo\") pod \"8a255596-5346-460f-b437-7c48d3d33d54\" (UID: \"8a255596-5346-460f-b437-7c48d3d33d54\") "
Nov 03 20:34:20.398845 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:20.398680    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a255596-5346-460f-b437-7c48d3d33d54-istio-token\" (UniqueName: \"kubernetes.io/projected/8a255596-5346-460f-b437-7c48d3d33d54-istio-token\") pod \"8a255596-5346-460f-b437-7c48d3d33d54\" (UID: \"8a255596-5346-460f-b437-7c48d3d33d54\") "
Nov 03 20:34:20.398845 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:20.398727    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a255596-5346-460f-b437-7c48d3d33d54-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a255596-5346-460f-b437-7c48d3d33d54-workload-socket\") pod \"8a255596-5346-460f-b437-7c48d3d33d54\" (UID: \"8a255596-5346-460f-b437-7c48d3d33d54\") "
Nov 03 20:34:20.398845 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:20.398792    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a255596-5346-460f-b437-7c48d3d33d54-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a255596-5346-460f-b437-7c48d3d33d54-workload-certs\") pod \"8a255596-5346-460f-b437-7c48d3d33d54\" (UID: \"8a255596-5346-460f-b437-7c48d3d33d54\") "
Nov 03 20:34:20.399003 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:20.398860    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a255596-5346-460f-b437-7c48d3d33d54-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a255596-5346-460f-b437-7c48d3d33d54-istio-envoy\") pod \"8a255596-5346-460f-b437-7c48d3d33d54\" (UID: \"8a255596-5346-460f-b437-7c48d3d33d54\") "
Nov 03 20:34:20.399562 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:20.399276    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a255596-5346-460f-b437-7c48d3d33d54-istio-data" pod "8a255596-5346-460f-b437-7c48d3d33d54" (UID: "8a255596-5346-460f-b437-7c48d3d33d54"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:34:20.399562 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:20.399439    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a255596-5346-460f-b437-7c48d3d33d54-credential-socket" pod "8a255596-5346-460f-b437-7c48d3d33d54" (UID: "8a255596-5346-460f-b437-7c48d3d33d54"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:34:20.399562 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:20.399515    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a255596-5346-460f-b437-7c48d3d33d54-workload-socket" pod "8a255596-5346-460f-b437-7c48d3d33d54" (UID: "8a255596-5346-460f-b437-7c48d3d33d54"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:34:20.399945 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:20.399694    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a255596-5346-460f-b437-7c48d3d33d54-workload-certs" pod "8a255596-5346-460f-b437-7c48d3d33d54" (UID: "8a255596-5346-460f-b437-7c48d3d33d54"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:34:20.399945 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:20.399338    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a255596-5346-460f-b437-7c48d3d33d54-istiod-ca-cert" pod "8a255596-5346-460f-b437-7c48d3d33d54" (UID: "8a255596-5346-460f-b437-7c48d3d33d54"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:34:20.399945 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:20.399766    5570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a255596-5346-460f-b437-7c48d3d33d54-istio-data\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:34:20.401173 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:20.401146    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a255596-5346-460f-b437-7c48d3d33d54-kube-api-access-56ng4" pod "8a255596-5346-460f-b437-7c48d3d33d54" (UID: "8a255596-5346-460f-b437-7c48d3d33d54"). InnerVolumeSpecName "kube-api-access-56ng4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:34:20.409952 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:20.409897    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a255596-5346-460f-b437-7c48d3d33d54-istio-envoy" pod "8a255596-5346-460f-b437-7c48d3d33d54" (UID: "8a255596-5346-460f-b437-7c48d3d33d54"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:34:20.409952 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:20.409916    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8a255596-5346-460f-b437-7c48d3d33d54-istio-podinfo" pod "8a255596-5346-460f-b437-7c48d3d33d54" (UID: "8a255596-5346-460f-b437-7c48d3d33d54"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:34:20.410140 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:20.410014    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a255596-5346-460f-b437-7c48d3d33d54-istio-token" pod "8a255596-5346-460f-b437-7c48d3d33d54" (UID: "8a255596-5346-460f-b437-7c48d3d33d54"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:34:20.501614 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:20.501577    5570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a255596-5346-460f-b437-7c48d3d33d54-credential-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:34:20.501614 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:20.501613    5570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a255596-5346-460f-b437-7c48d3d33d54-istio-podinfo\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:34:20.501614 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:20.501625    5570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a255596-5346-460f-b437-7c48d3d33d54-istio-token\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:34:20.501912 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:20.501633    5570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a255596-5346-460f-b437-7c48d3d33d54-workload-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:34:20.501912 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:20.501641    5570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a255596-5346-460f-b437-7c48d3d33d54-workload-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:34:20.501912 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:20.501649    5570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a255596-5346-460f-b437-7c48d3d33d54-istio-envoy\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:34:20.501912 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:20.501657    5570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a255596-5346-460f-b437-7c48d3d33d54-istiod-ca-cert\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:34:20.501912 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:20.501664    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56ng4\" (UniqueName: \"kubernetes.io/projected/8a255596-5346-460f-b437-7c48d3d33d54-kube-api-access-56ng4\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:34:21.236590 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:21.236559    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-c5nf4"
Nov 03 20:34:21.237178 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:21.236599    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-c5nf4" event={"ID":"8a255596-5346-460f-b437-7c48d3d33d54","Type":"ContainerDied","Data":"489c9b309f06d4e2be07db0840600fd8b4cb7a4376291f92672a24215a6aa049"}
Nov 03 20:34:21.237178 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:21.236670    5570 scope.go:122] "RemoveContainer" containerID="a7bb15afee0634b9ec32b88804532afc13e276d3c21985b7c080bb3aa350a328"
Nov 03 20:34:21.310740 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:21.310702    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-c5nf4"]
Nov 03 20:34:21.324002 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:21.323972    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-fd01ad-openshift-default-7c5594dc4c-c5nf4"]
Nov 03 20:34:21.965675 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:21.965637    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a255596-5346-460f-b437-7c48d3d33d54" path="/var/lib/kubelet/pods/8a255596-5346-460f-b437-7c48d3d33d54/volumes"
Nov 03 20:34:21.966323 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:21.966297    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="974badc1-41ad-461c-8719-e1f3f836ccbc" path="/var/lib/kubelet/pods/974badc1-41ad-461c-8719-e1f3f836ccbc/volumes"
Nov 03 20:34:22.209828 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:22.209718    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-jjjr7"
Nov 03 20:34:22.222008 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:22.221978    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ca8412e-c6de-4cb5-944e-ba8afd2fa791-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ca8412e-c6de-4cb5-944e-ba8afd2fa791-workload-certs\") pod \"7ca8412e-c6de-4cb5-944e-ba8afd2fa791\" (UID: \"7ca8412e-c6de-4cb5-944e-ba8afd2fa791\") "
Nov 03 20:34:22.222105 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:22.222051    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ca8412e-c6de-4cb5-944e-ba8afd2fa791-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ca8412e-c6de-4cb5-944e-ba8afd2fa791-istio-envoy\") pod \"7ca8412e-c6de-4cb5-944e-ba8afd2fa791\" (UID: \"7ca8412e-c6de-4cb5-944e-ba8afd2fa791\") "
Nov 03 20:34:22.222161 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:22.222107    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7ca8412e-c6de-4cb5-944e-ba8afd2fa791-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ca8412e-c6de-4cb5-944e-ba8afd2fa791-istio-podinfo\") pod \"7ca8412e-c6de-4cb5-944e-ba8afd2fa791\" (UID: \"7ca8412e-c6de-4cb5-944e-ba8afd2fa791\") "
Nov 03 20:34:22.222161 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:22.222141    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ca8412e-c6de-4cb5-944e-ba8afd2fa791-istio-token\" (UniqueName: \"kubernetes.io/projected/7ca8412e-c6de-4cb5-944e-ba8afd2fa791-istio-token\") pod \"7ca8412e-c6de-4cb5-944e-ba8afd2fa791\" (UID: \"7ca8412e-c6de-4cb5-944e-ba8afd2fa791\") "
Nov 03 20:34:22.222271 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:22.222200    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ca8412e-c6de-4cb5-944e-ba8afd2fa791-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ca8412e-c6de-4cb5-944e-ba8afd2fa791-istiod-ca-cert\") pod \"7ca8412e-c6de-4cb5-944e-ba8afd2fa791\" (UID: \"7ca8412e-c6de-4cb5-944e-ba8afd2fa791\") "
Nov 03 20:34:22.222271 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:22.222216    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ca8412e-c6de-4cb5-944e-ba8afd2fa791-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ca8412e-c6de-4cb5-944e-ba8afd2fa791-credential-socket\") pod \"7ca8412e-c6de-4cb5-944e-ba8afd2fa791\" (UID: \"7ca8412e-c6de-4cb5-944e-ba8afd2fa791\") "
Nov 03 20:34:22.222271 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:22.222268    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ca8412e-c6de-4cb5-944e-ba8afd2fa791-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ca8412e-c6de-4cb5-944e-ba8afd2fa791-workload-socket\") pod \"7ca8412e-c6de-4cb5-944e-ba8afd2fa791\" (UID: \"7ca8412e-c6de-4cb5-944e-ba8afd2fa791\") "
Nov 03 20:34:22.222438 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:22.222307    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ca8412e-c6de-4cb5-944e-ba8afd2fa791-kube-api-access-hg92x\" (UniqueName: \"kubernetes.io/projected/7ca8412e-c6de-4cb5-944e-ba8afd2fa791-kube-api-access-hg92x\") pod \"7ca8412e-c6de-4cb5-944e-ba8afd2fa791\" (UID: \"7ca8412e-c6de-4cb5-944e-ba8afd2fa791\") "
Nov 03 20:34:22.222438 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:22.222313    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ca8412e-c6de-4cb5-944e-ba8afd2fa791-workload-certs" pod "7ca8412e-c6de-4cb5-944e-ba8afd2fa791" (UID: "7ca8412e-c6de-4cb5-944e-ba8afd2fa791"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:34:22.222438 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:22.222368    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ca8412e-c6de-4cb5-944e-ba8afd2fa791-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ca8412e-c6de-4cb5-944e-ba8afd2fa791-istio-data\") pod \"7ca8412e-c6de-4cb5-944e-ba8afd2fa791\" (UID: \"7ca8412e-c6de-4cb5-944e-ba8afd2fa791\") "
Nov 03 20:34:22.222639 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:22.222591    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ca8412e-c6de-4cb5-944e-ba8afd2fa791-credential-socket" pod "7ca8412e-c6de-4cb5-944e-ba8afd2fa791" (UID: "7ca8412e-c6de-4cb5-944e-ba8afd2fa791"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:34:22.222639 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:22.222609    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ca8412e-c6de-4cb5-944e-ba8afd2fa791-workload-socket" pod "7ca8412e-c6de-4cb5-944e-ba8afd2fa791" (UID: "7ca8412e-c6de-4cb5-944e-ba8afd2fa791"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:34:22.222746 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:22.222639    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ca8412e-c6de-4cb5-944e-ba8afd2fa791-istiod-ca-cert" pod "7ca8412e-c6de-4cb5-944e-ba8afd2fa791" (UID: "7ca8412e-c6de-4cb5-944e-ba8afd2fa791"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:34:22.222857 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:22.222836    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ca8412e-c6de-4cb5-944e-ba8afd2fa791-istio-data" pod "7ca8412e-c6de-4cb5-944e-ba8afd2fa791" (UID: "7ca8412e-c6de-4cb5-944e-ba8afd2fa791"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:34:22.223104 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:22.223078    5570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ca8412e-c6de-4cb5-944e-ba8afd2fa791-istiod-ca-cert\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:34:22.223196 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:22.223109    5570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ca8412e-c6de-4cb5-944e-ba8afd2fa791-credential-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:34:22.223196 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:22.223125    5570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ca8412e-c6de-4cb5-944e-ba8afd2fa791-workload-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:34:22.223196 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:22.223138    5570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ca8412e-c6de-4cb5-944e-ba8afd2fa791-istio-data\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:34:22.223196 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:22.223151    5570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ca8412e-c6de-4cb5-944e-ba8afd2fa791-workload-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:34:22.224453 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:22.224430    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7ca8412e-c6de-4cb5-944e-ba8afd2fa791-istio-podinfo" pod "7ca8412e-c6de-4cb5-944e-ba8afd2fa791" (UID: "7ca8412e-c6de-4cb5-944e-ba8afd2fa791"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:34:22.224541 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:22.224467    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ca8412e-c6de-4cb5-944e-ba8afd2fa791-istio-envoy" pod "7ca8412e-c6de-4cb5-944e-ba8afd2fa791" (UID: "7ca8412e-c6de-4cb5-944e-ba8afd2fa791"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:34:22.224608 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:22.224550    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ca8412e-c6de-4cb5-944e-ba8afd2fa791-istio-token" pod "7ca8412e-c6de-4cb5-944e-ba8afd2fa791" (UID: "7ca8412e-c6de-4cb5-944e-ba8afd2fa791"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:34:22.224661 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:22.224615    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ca8412e-c6de-4cb5-944e-ba8afd2fa791-kube-api-access-hg92x" pod "7ca8412e-c6de-4cb5-944e-ba8afd2fa791" (UID: "7ca8412e-c6de-4cb5-944e-ba8afd2fa791"). InnerVolumeSpecName "kube-api-access-hg92x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:34:22.245570 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:22.245535    5570 generic.go:356] "Generic (PLEG): container finished" podID="7ca8412e-c6de-4cb5-944e-ba8afd2fa791" containerID="0e87fdc31bf499c3143805dba02ed2b9d49780fdf96d7d02f51272999a98282d" exitCode=0
Nov 03 20:34:22.246013 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:22.245579    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-jjjr7"
Nov 03 20:34:22.246013 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:22.245611    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-jjjr7" event={"ID":"7ca8412e-c6de-4cb5-944e-ba8afd2fa791","Type":"ContainerDied","Data":"0e87fdc31bf499c3143805dba02ed2b9d49780fdf96d7d02f51272999a98282d"}
Nov 03 20:34:22.246013 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:22.245696    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-jjjr7" event={"ID":"7ca8412e-c6de-4cb5-944e-ba8afd2fa791","Type":"ContainerDied","Data":"4ab71c224982bd60dcd2bbc0ab9f10ef89374f3563e8b3718ebca4210186952a"}
Nov 03 20:34:22.246013 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:22.245725    5570 scope.go:122] "RemoveContainer" containerID="0e87fdc31bf499c3143805dba02ed2b9d49780fdf96d7d02f51272999a98282d"
Nov 03 20:34:22.256943 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:22.256919    5570 scope.go:122] "RemoveContainer" containerID="2ea2028d5fcd730c4d6c7fb86bd31a166bf09e340eba28a808236e578dcc3ffa"
Nov 03 20:34:22.267271 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:22.267248    5570 scope.go:122] "RemoveContainer" containerID="0e87fdc31bf499c3143805dba02ed2b9d49780fdf96d7d02f51272999a98282d"
Nov 03 20:34:22.267569 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:34:22.267551    5570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e87fdc31bf499c3143805dba02ed2b9d49780fdf96d7d02f51272999a98282d\": container with ID starting with 0e87fdc31bf499c3143805dba02ed2b9d49780fdf96d7d02f51272999a98282d not found: ID does not exist" containerID="0e87fdc31bf499c3143805dba02ed2b9d49780fdf96d7d02f51272999a98282d"
Nov 03 20:34:22.267618 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:22.267580    5570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e87fdc31bf499c3143805dba02ed2b9d49780fdf96d7d02f51272999a98282d"} err="failed to get container status \"0e87fdc31bf499c3143805dba02ed2b9d49780fdf96d7d02f51272999a98282d\": rpc error: code = NotFound desc = could not find container \"0e87fdc31bf499c3143805dba02ed2b9d49780fdf96d7d02f51272999a98282d\": container with ID starting with 0e87fdc31bf499c3143805dba02ed2b9d49780fdf96d7d02f51272999a98282d not found: ID does not exist"
Nov 03 20:34:22.267618 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:22.267598    5570 scope.go:122] "RemoveContainer" containerID="2ea2028d5fcd730c4d6c7fb86bd31a166bf09e340eba28a808236e578dcc3ffa"
Nov 03 20:34:22.267850 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:34:22.267829    5570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ea2028d5fcd730c4d6c7fb86bd31a166bf09e340eba28a808236e578dcc3ffa\": container with ID starting with 2ea2028d5fcd730c4d6c7fb86bd31a166bf09e340eba28a808236e578dcc3ffa not found: ID does not exist" containerID="2ea2028d5fcd730c4d6c7fb86bd31a166bf09e340eba28a808236e578dcc3ffa"
Nov 03 20:34:22.267916 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:22.267875    5570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ea2028d5fcd730c4d6c7fb86bd31a166bf09e340eba28a808236e578dcc3ffa"} err="failed to get container status \"2ea2028d5fcd730c4d6c7fb86bd31a166bf09e340eba28a808236e578dcc3ffa\": rpc error: code = NotFound desc = could not find container \"2ea2028d5fcd730c4d6c7fb86bd31a166bf09e340eba28a808236e578dcc3ffa\": container with ID starting with 2ea2028d5fcd730c4d6c7fb86bd31a166bf09e340eba28a808236e578dcc3ffa not found: ID does not exist"
Nov 03 20:34:22.315613 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:22.315580    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-jjjr7"]
Nov 03 20:34:22.319975 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:22.319942    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-jjjr7"]
Nov 03 20:34:22.324567 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:22.324538    5570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ca8412e-c6de-4cb5-944e-ba8afd2fa791-istio-envoy\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:34:22.324567 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:22.324566    5570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ca8412e-c6de-4cb5-944e-ba8afd2fa791-istio-podinfo\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:34:22.324760 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:22.324575    5570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ca8412e-c6de-4cb5-944e-ba8afd2fa791-istio-token\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:34:22.324760 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:22.324584    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hg92x\" (UniqueName: \"kubernetes.io/projected/7ca8412e-c6de-4cb5-944e-ba8afd2fa791-kube-api-access-hg92x\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:34:22.927505 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:22.927468    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e40816-548465ddf5-mzdj7" podUID="94a03b52-b293-4475-8b23-fbbbae06fb26" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 20:34:23.966431 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:23.966395    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ca8412e-c6de-4cb5-944e-ba8afd2fa791" path="/var/lib/kubelet/pods/7ca8412e-c6de-4cb5-944e-ba8afd2fa791/volumes"
Nov 03 20:34:25.282388 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:25.282346    5570 generic.go:356] "Generic (PLEG): container finished" podID="94a03b52-b293-4475-8b23-fbbbae06fb26" containerID="e8cc79d0834cf7d7b1ed1f4463ba238eecea9b687962905be6718222cd5de000" exitCode=0
Nov 03 20:34:25.282811 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:25.282407    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e40816-548465ddf5-mzdj7" event={"ID":"94a03b52-b293-4475-8b23-fbbbae06fb26","Type":"ContainerDied","Data":"e8cc79d0834cf7d7b1ed1f4463ba238eecea9b687962905be6718222cd5de000"}
Nov 03 20:34:25.282811 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:25.282469    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e40816-548465ddf5-mzdj7" event={"ID":"94a03b52-b293-4475-8b23-fbbbae06fb26","Type":"ContainerDied","Data":"9b39545510de8f1b362e4a18192b5b839e958609ffb23ae806f52080b7a9f6b4"}
Nov 03 20:34:25.282811 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:25.282493    5570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9b39545510de8f1b362e4a18192b5b839e958609ffb23ae806f52080b7a9f6b4"
Nov 03 20:34:25.291575 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:25.291547    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e40816-548465ddf5-mzdj7"
Nov 03 20:34:25.369198 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:25.369151    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94a03b52-b293-4475-8b23-fbbbae06fb26-kube-api-access-nx8k9\" (UniqueName: \"kubernetes.io/projected/94a03b52-b293-4475-8b23-fbbbae06fb26-kube-api-access-nx8k9\") pod \"94a03b52-b293-4475-8b23-fbbbae06fb26\" (UID: \"94a03b52-b293-4475-8b23-fbbbae06fb26\") "
Nov 03 20:34:25.369418 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:25.369244    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94a03b52-b293-4475-8b23-fbbbae06fb26-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94a03b52-b293-4475-8b23-fbbbae06fb26-plugins-config-volume\") pod \"94a03b52-b293-4475-8b23-fbbbae06fb26\" (UID: \"94a03b52-b293-4475-8b23-fbbbae06fb26\") "
Nov 03 20:34:25.369701 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:25.369675    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94a03b52-b293-4475-8b23-fbbbae06fb26-plugins-config-volume" pod "94a03b52-b293-4475-8b23-fbbbae06fb26" (UID: "94a03b52-b293-4475-8b23-fbbbae06fb26"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:34:25.369867 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:25.369851    5570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94a03b52-b293-4475-8b23-fbbbae06fb26-plugins-config-volume\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:34:25.371603 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:25.371568    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94a03b52-b293-4475-8b23-fbbbae06fb26-kube-api-access-nx8k9" pod "94a03b52-b293-4475-8b23-fbbbae06fb26" (UID: "94a03b52-b293-4475-8b23-fbbbae06fb26"). InnerVolumeSpecName "kube-api-access-nx8k9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:34:25.472460 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:25.471348    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nx8k9\" (UniqueName: \"kubernetes.io/projected/94a03b52-b293-4475-8b23-fbbbae06fb26-kube-api-access-nx8k9\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:34:26.289399 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:26.289306    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e40816-548465ddf5-mzdj7"
Nov 03 20:34:26.343327 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:26.343293    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-e40816-548465ddf5-mzdj7"]
Nov 03 20:34:26.356851 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:26.356813    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-e40816-548465ddf5-mzdj7"]
Nov 03 20:34:27.230088 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:34:27.230055    5570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8bwhz" containerName="extractor"
Nov 03 20:34:27.965280 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:27.965245    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94a03b52-b293-4475-8b23-fbbbae06fb26" path="/var/lib/kubelet/pods/94a03b52-b293-4475-8b23-fbbbae06fb26/volumes"
Nov 03 20:34:31.324915 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:31.324536    5570 scope.go:122] "RemoveContainer" containerID="e8cc79d0834cf7d7b1ed1f4463ba238eecea9b687962905be6718222cd5de000"
Nov 03 20:34:34.687952 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:34.687908    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-xt4z5"]
Nov 03 20:34:34.688497 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:34.688242    5570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-xt4z5" podUID="e026bfa1-f899-46e8-9309-be16eea226bb" containerName="istio-proxy" containerID="cri-o://b4c63954b88fed3fa4e0f508daa689ef20d457a83e2b12c3f5d3a8c3e34f6308" gracePeriod=30
Nov 03 20:34:35.140098 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:35.139993    5570 patch_prober.go:29] interesting pod/e2e-worker-w0-e40816-openshift-default-744c57967c-xt4z5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 20:34:35.140286 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:35.140091    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-xt4z5" podUID="e026bfa1-f899-46e8-9309-be16eea226bb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 20:34:39.960621 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:39.960544    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-xt4z5"
Nov 03 20:34:39.980221 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:39.980179    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e026bfa1-f899-46e8-9309-be16eea226bb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e026bfa1-f899-46e8-9309-be16eea226bb-credential-socket\") pod \"e026bfa1-f899-46e8-9309-be16eea226bb\" (UID: \"e026bfa1-f899-46e8-9309-be16eea226bb\") "
Nov 03 20:34:39.980433 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:39.980277    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e026bfa1-f899-46e8-9309-be16eea226bb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e026bfa1-f899-46e8-9309-be16eea226bb-workload-certs\") pod \"e026bfa1-f899-46e8-9309-be16eea226bb\" (UID: \"e026bfa1-f899-46e8-9309-be16eea226bb\") "
Nov 03 20:34:39.980433 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:39.980308    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e026bfa1-f899-46e8-9309-be16eea226bb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e026bfa1-f899-46e8-9309-be16eea226bb-istio-data\") pod \"e026bfa1-f899-46e8-9309-be16eea226bb\" (UID: \"e026bfa1-f899-46e8-9309-be16eea226bb\") "
Nov 03 20:34:39.980433 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:39.980355    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e026bfa1-f899-46e8-9309-be16eea226bb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e026bfa1-f899-46e8-9309-be16eea226bb-istio-envoy\") pod \"e026bfa1-f899-46e8-9309-be16eea226bb\" (UID: \"e026bfa1-f899-46e8-9309-be16eea226bb\") "
Nov 03 20:34:39.980433 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:39.980397    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e026bfa1-f899-46e8-9309-be16eea226bb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e026bfa1-f899-46e8-9309-be16eea226bb-istio-podinfo\") pod \"e026bfa1-f899-46e8-9309-be16eea226bb\" (UID: \"e026bfa1-f899-46e8-9309-be16eea226bb\") "
Nov 03 20:34:39.980663 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:39.980491    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e026bfa1-f899-46e8-9309-be16eea226bb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e026bfa1-f899-46e8-9309-be16eea226bb-workload-socket\") pod \"e026bfa1-f899-46e8-9309-be16eea226bb\" (UID: \"e026bfa1-f899-46e8-9309-be16eea226bb\") "
Nov 03 20:34:39.980663 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:39.980539    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e026bfa1-f899-46e8-9309-be16eea226bb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e026bfa1-f899-46e8-9309-be16eea226bb-istiod-ca-cert\") pod \"e026bfa1-f899-46e8-9309-be16eea226bb\" (UID: \"e026bfa1-f899-46e8-9309-be16eea226bb\") "
Nov 03 20:34:39.980663 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:39.980554    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e026bfa1-f899-46e8-9309-be16eea226bb-credential-socket" pod "e026bfa1-f899-46e8-9309-be16eea226bb" (UID: "e026bfa1-f899-46e8-9309-be16eea226bb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:34:39.980663 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:39.980604    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e026bfa1-f899-46e8-9309-be16eea226bb-istio-token\" (UniqueName: \"kubernetes.io/projected/e026bfa1-f899-46e8-9309-be16eea226bb-istio-token\") pod \"e026bfa1-f899-46e8-9309-be16eea226bb\" (UID: \"e026bfa1-f899-46e8-9309-be16eea226bb\") "
Nov 03 20:34:39.980663 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:39.980639    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e026bfa1-f899-46e8-9309-be16eea226bb-kube-api-access-m5zsf\" (UniqueName: \"kubernetes.io/projected/e026bfa1-f899-46e8-9309-be16eea226bb-kube-api-access-m5zsf\") pod \"e026bfa1-f899-46e8-9309-be16eea226bb\" (UID: \"e026bfa1-f899-46e8-9309-be16eea226bb\") "
Nov 03 20:34:39.980663 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:39.980647    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e026bfa1-f899-46e8-9309-be16eea226bb-workload-certs" pod "e026bfa1-f899-46e8-9309-be16eea226bb" (UID: "e026bfa1-f899-46e8-9309-be16eea226bb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:34:39.981006 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:39.980663    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e026bfa1-f899-46e8-9309-be16eea226bb-istio-data" pod "e026bfa1-f899-46e8-9309-be16eea226bb" (UID: "e026bfa1-f899-46e8-9309-be16eea226bb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:34:39.981006 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:39.980828    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e026bfa1-f899-46e8-9309-be16eea226bb-workload-socket" pod "e026bfa1-f899-46e8-9309-be16eea226bb" (UID: "e026bfa1-f899-46e8-9309-be16eea226bb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:34:39.981126 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:39.981032    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e026bfa1-f899-46e8-9309-be16eea226bb-istiod-ca-cert" pod "e026bfa1-f899-46e8-9309-be16eea226bb" (UID: "e026bfa1-f899-46e8-9309-be16eea226bb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:34:39.981399 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:39.981377    5570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e026bfa1-f899-46e8-9309-be16eea226bb-credential-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:34:39.981503 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:39.981402    5570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e026bfa1-f899-46e8-9309-be16eea226bb-workload-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:34:39.981503 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:39.981416    5570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e026bfa1-f899-46e8-9309-be16eea226bb-istio-data\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:34:39.981503 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:39.981428    5570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e026bfa1-f899-46e8-9309-be16eea226bb-workload-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:34:39.981503 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:39.981440    5570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e026bfa1-f899-46e8-9309-be16eea226bb-istiod-ca-cert\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:34:39.982978 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:39.982954    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e026bfa1-f899-46e8-9309-be16eea226bb-kube-api-access-m5zsf" pod "e026bfa1-f899-46e8-9309-be16eea226bb" (UID: "e026bfa1-f899-46e8-9309-be16eea226bb"). InnerVolumeSpecName "kube-api-access-m5zsf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:34:39.983157 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:39.983139    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e026bfa1-f899-46e8-9309-be16eea226bb-istio-token" pod "e026bfa1-f899-46e8-9309-be16eea226bb" (UID: "e026bfa1-f899-46e8-9309-be16eea226bb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:34:39.983224 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:39.983159    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e026bfa1-f899-46e8-9309-be16eea226bb-istio-podinfo" pod "e026bfa1-f899-46e8-9309-be16eea226bb" (UID: "e026bfa1-f899-46e8-9309-be16eea226bb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:34:39.983260 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:39.983242    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e026bfa1-f899-46e8-9309-be16eea226bb-istio-envoy" pod "e026bfa1-f899-46e8-9309-be16eea226bb" (UID: "e026bfa1-f899-46e8-9309-be16eea226bb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:34:40.095953 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:40.095904    5570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e026bfa1-f899-46e8-9309-be16eea226bb-istio-envoy\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:34:40.095953 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:40.095952    5570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e026bfa1-f899-46e8-9309-be16eea226bb-istio-podinfo\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:34:40.096232 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:40.095967    5570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e026bfa1-f899-46e8-9309-be16eea226bb-istio-token\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:34:40.096232 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:40.095981    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5zsf\" (UniqueName: \"kubernetes.io/projected/e026bfa1-f899-46e8-9309-be16eea226bb-kube-api-access-m5zsf\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:34:40.605042 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:40.604998    5570 generic.go:356] "Generic (PLEG): container finished" podID="e026bfa1-f899-46e8-9309-be16eea226bb" containerID="b4c63954b88fed3fa4e0f508daa689ef20d457a83e2b12c3f5d3a8c3e34f6308" exitCode=0
Nov 03 20:34:40.605250 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:40.605066    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-xt4z5"
Nov 03 20:34:40.605250 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:40.605063    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-xt4z5" event={"ID":"e026bfa1-f899-46e8-9309-be16eea226bb","Type":"ContainerDied","Data":"b4c63954b88fed3fa4e0f508daa689ef20d457a83e2b12c3f5d3a8c3e34f6308"}
Nov 03 20:34:40.605250 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:40.605148    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-xt4z5" event={"ID":"e026bfa1-f899-46e8-9309-be16eea226bb","Type":"ContainerDied","Data":"66ed62de0da2e8c8f541b3358d6cfe7819efcd760b7e252f5bb2732f5400f29a"}
Nov 03 20:34:40.605250 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:40.605177    5570 scope.go:122] "RemoveContainer" containerID="b4c63954b88fed3fa4e0f508daa689ef20d457a83e2b12c3f5d3a8c3e34f6308"
Nov 03 20:34:40.618030 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:40.617887    5570 scope.go:122] "RemoveContainer" containerID="b4c63954b88fed3fa4e0f508daa689ef20d457a83e2b12c3f5d3a8c3e34f6308"
Nov 03 20:34:40.621008 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:34:40.620975    5570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4c63954b88fed3fa4e0f508daa689ef20d457a83e2b12c3f5d3a8c3e34f6308\": container with ID starting with b4c63954b88fed3fa4e0f508daa689ef20d457a83e2b12c3f5d3a8c3e34f6308 not found: ID does not exist" containerID="b4c63954b88fed3fa4e0f508daa689ef20d457a83e2b12c3f5d3a8c3e34f6308"
Nov 03 20:34:40.621133 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:40.621017    5570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4c63954b88fed3fa4e0f508daa689ef20d457a83e2b12c3f5d3a8c3e34f6308"} err="failed to get container status \"b4c63954b88fed3fa4e0f508daa689ef20d457a83e2b12c3f5d3a8c3e34f6308\": rpc error: code = NotFound desc = could not find container \"b4c63954b88fed3fa4e0f508daa689ef20d457a83e2b12c3f5d3a8c3e34f6308\": container with ID starting with b4c63954b88fed3fa4e0f508daa689ef20d457a83e2b12c3f5d3a8c3e34f6308 not found: ID does not exist"
Nov 03 20:34:40.674820 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:40.674767    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-xt4z5"]
Nov 03 20:34:40.677683 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:40.677659    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-e40816-openshift-default-744c57967c-xt4z5"]
Nov 03 20:34:41.965610 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:41.965571    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e026bfa1-f899-46e8-9309-be16eea226bb" path="/var/lib/kubelet/pods/e026bfa1-f899-46e8-9309-be16eea226bb/volumes"
Nov 03 20:34:43.259335 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:43.259302    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g69f8"]
Nov 03 20:34:47.843179 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:47.843137    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-a9ba83/test-model-a9ba83-kserve-5f6985cb6c-d6pfg"]
Nov 03 20:34:47.844086 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:47.844063    5570 cpu_manager.go:414] "containerMap: removing container" podUID="8a255596-5346-460f-b437-7c48d3d33d54" containerName="istio-proxy"
Nov 03 20:34:47.844154 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:47.844090    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a255596-5346-460f-b437-7c48d3d33d54" containerName="istio-proxy"
Nov 03 20:34:47.844154 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:47.844108    5570 cpu_manager.go:414] "containerMap: removing container" podUID="94a03b52-b293-4475-8b23-fbbbae06fb26" containerName="payload-pre-processing"
Nov 03 20:34:47.844154 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:47.844116    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94a03b52-b293-4475-8b23-fbbbae06fb26" containerName="payload-pre-processing"
Nov 03 20:34:47.844154 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:47.844135    5570 cpu_manager.go:414] "containerMap: removing container" podUID="7ca8412e-c6de-4cb5-944e-ba8afd2fa791" containerName="istio-proxy"
Nov 03 20:34:47.844154 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:47.844144    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ca8412e-c6de-4cb5-944e-ba8afd2fa791" containerName="istio-proxy"
Nov 03 20:34:47.844312 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:47.844175    5570 cpu_manager.go:414] "containerMap: removing container" podUID="c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62" containerName="main"
Nov 03 20:34:47.844312 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:47.844182    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62" containerName="main"
Nov 03 20:34:47.844312 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:47.844193    5570 cpu_manager.go:414] "containerMap: removing container" podUID="974badc1-41ad-461c-8719-e1f3f836ccbc" containerName="istio-proxy"
Nov 03 20:34:47.844312 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:47.844200    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="974badc1-41ad-461c-8719-e1f3f836ccbc" containerName="istio-proxy"
Nov 03 20:34:47.844312 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:47.844218    5570 cpu_manager.go:414] "containerMap: removing container" podUID="8a255596-5346-460f-b437-7c48d3d33d54" containerName="istio-proxy"
Nov 03 20:34:47.844312 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:47.844225    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a255596-5346-460f-b437-7c48d3d33d54" containerName="istio-proxy"
Nov 03 20:34:47.844312 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:47.844236    5570 cpu_manager.go:414] "containerMap: removing container" podUID="7ca8412e-c6de-4cb5-944e-ba8afd2fa791" containerName="istio-proxy"
Nov 03 20:34:47.844312 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:47.844245    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ca8412e-c6de-4cb5-944e-ba8afd2fa791" containerName="istio-proxy"
Nov 03 20:34:47.844312 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:47.844254    5570 cpu_manager.go:414] "containerMap: removing container" podUID="4ad9b048-925c-4d01-a839-a5f25d4dd02d" containerName="main"
Nov 03 20:34:47.844312 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:47.844263    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ad9b048-925c-4d01-a839-a5f25d4dd02d" containerName="main"
Nov 03 20:34:47.844312 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:47.844274    5570 cpu_manager.go:414] "containerMap: removing container" podUID="89266b89-4697-45b3-af99-c2dddf7034bc" containerName="revoke-keys"
Nov 03 20:34:47.844312 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:47.844281    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89266b89-4697-45b3-af99-c2dddf7034bc" containerName="revoke-keys"
Nov 03 20:34:47.844312 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:47.844297    5570 cpu_manager.go:414] "containerMap: removing container" podUID="974badc1-41ad-461c-8719-e1f3f836ccbc" containerName="istio-proxy"
Nov 03 20:34:47.844312 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:47.844304    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="974badc1-41ad-461c-8719-e1f3f836ccbc" containerName="istio-proxy"
Nov 03 20:34:47.844726 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:47.844330    5570 cpu_manager.go:414] "containerMap: removing container" podUID="e026bfa1-f899-46e8-9309-be16eea226bb" containerName="istio-proxy"
Nov 03 20:34:47.844726 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:47.844337    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e026bfa1-f899-46e8-9309-be16eea226bb" containerName="istio-proxy"
Nov 03 20:34:47.844726 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:47.844361    5570 cpu_manager.go:414] "containerMap: removing container" podUID="e185c296-cd84-4b0c-9316-ef10085cda0a" containerName="istio-proxy"
Nov 03 20:34:47.844726 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:47.844368    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e185c296-cd84-4b0c-9316-ef10085cda0a" containerName="istio-proxy"
Nov 03 20:34:47.844726 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:47.844514    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a255596-5346-460f-b437-7c48d3d33d54" containerName="istio-proxy"
Nov 03 20:34:47.844726 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:47.844531    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="94a03b52-b293-4475-8b23-fbbbae06fb26" containerName="payload-pre-processing"
Nov 03 20:34:47.844726 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:47.844542    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ca8412e-c6de-4cb5-944e-ba8afd2fa791" containerName="istio-proxy"
Nov 03 20:34:47.844726 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:47.844556    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0e6d9ca-7944-44d2-8bbd-cc6fc3019f62" containerName="main"
Nov 03 20:34:47.844726 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:47.844566    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="974badc1-41ad-461c-8719-e1f3f836ccbc" containerName="istio-proxy"
Nov 03 20:34:47.844726 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:47.844576    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a255596-5346-460f-b437-7c48d3d33d54" containerName="istio-proxy"
Nov 03 20:34:47.844726 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:47.844583    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ad9b048-925c-4d01-a839-a5f25d4dd02d" containerName="main"
Nov 03 20:34:47.844726 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:47.844592    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="89266b89-4697-45b3-af99-c2dddf7034bc" containerName="revoke-keys"
Nov 03 20:34:47.844726 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:47.844604    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="e026bfa1-f899-46e8-9309-be16eea226bb" containerName="istio-proxy"
Nov 03 20:34:47.844726 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:47.844614    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="e185c296-cd84-4b0c-9316-ef10085cda0a" containerName="istio-proxy"
Nov 03 20:34:47.848625 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:47.848595    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a9ba83/test-model-a9ba83-kserve-5f6985cb6c-d6pfg"
Nov 03 20:34:47.867174 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:47.867140    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-a9ba83/test-model-a9ba83-kserve-5f6985cb6c-d6pfg"]
Nov 03 20:34:47.875388 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:47.875356    5570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-a9ba83\"/\"openshift-service-ca.crt\""
Nov 03 20:34:47.875539 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:47.875417    5570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-a9ba83\"/\"kube-root-ca.crt\""
Nov 03 20:34:47.875585 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:47.875550    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-a9ba83\"/\"test-model-a9ba83-kserve-self-signed-certs\""
Nov 03 20:34:47.878238 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:47.878212    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-a9ba83\"/\"default-dockercfg-54qw2\""
Nov 03 20:34:47.918135 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:47.918095    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wp6gx\" (UniqueName: \"kubernetes.io/projected/b2d9f055-1380-4eb4-84ee-48dc75855852-kube-api-access-wp6gx\") pod \"test-model-a9ba83-kserve-5f6985cb6c-d6pfg\" (UID: \"b2d9f055-1380-4eb4-84ee-48dc75855852\") " pod="ai-tenant-e2e-inf-a-a9ba83/test-model-a9ba83-kserve-5f6985cb6c-d6pfg"
Nov 03 20:34:47.918315 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:47.918151    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2d9f055-1380-4eb4-84ee-48dc75855852-tls-certs\") pod \"test-model-a9ba83-kserve-5f6985cb6c-d6pfg\" (UID: \"b2d9f055-1380-4eb4-84ee-48dc75855852\") " pod="ai-tenant-e2e-inf-a-a9ba83/test-model-a9ba83-kserve-5f6985cb6c-d6pfg"
Nov 03 20:34:47.918315 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:47.918299    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2d9f055-1380-4eb4-84ee-48dc75855852-home\") pod \"test-model-a9ba83-kserve-5f6985cb6c-d6pfg\" (UID: \"b2d9f055-1380-4eb4-84ee-48dc75855852\") " pod="ai-tenant-e2e-inf-a-a9ba83/test-model-a9ba83-kserve-5f6985cb6c-d6pfg"
Nov 03 20:34:47.918485 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:47.918467    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2d9f055-1380-4eb4-84ee-48dc75855852-model-cache\") pod \"test-model-a9ba83-kserve-5f6985cb6c-d6pfg\" (UID: \"b2d9f055-1380-4eb4-84ee-48dc75855852\") " pod="ai-tenant-e2e-inf-a-a9ba83/test-model-a9ba83-kserve-5f6985cb6c-d6pfg"
Nov 03 20:34:47.918549 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:47.918526    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2d9f055-1380-4eb4-84ee-48dc75855852-tmp-dir\") pod \"test-model-a9ba83-kserve-5f6985cb6c-d6pfg\" (UID: \"b2d9f055-1380-4eb4-84ee-48dc75855852\") " pod="ai-tenant-e2e-inf-a-a9ba83/test-model-a9ba83-kserve-5f6985cb6c-d6pfg"
Nov 03 20:34:47.918675 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:47.918628    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2d9f055-1380-4eb4-84ee-48dc75855852-dshm\") pod \"test-model-a9ba83-kserve-5f6985cb6c-d6pfg\" (UID: \"b2d9f055-1380-4eb4-84ee-48dc75855852\") " pod="ai-tenant-e2e-inf-a-a9ba83/test-model-a9ba83-kserve-5f6985cb6c-d6pfg"
Nov 03 20:34:48.020452 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:48.020412    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2d9f055-1380-4eb4-84ee-48dc75855852-model-cache\") pod \"test-model-a9ba83-kserve-5f6985cb6c-d6pfg\" (UID: \"b2d9f055-1380-4eb4-84ee-48dc75855852\") " pod="ai-tenant-e2e-inf-a-a9ba83/test-model-a9ba83-kserve-5f6985cb6c-d6pfg"
Nov 03 20:34:48.020452 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:48.020460    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2d9f055-1380-4eb4-84ee-48dc75855852-tmp-dir\") pod \"test-model-a9ba83-kserve-5f6985cb6c-d6pfg\" (UID: \"b2d9f055-1380-4eb4-84ee-48dc75855852\") " pod="ai-tenant-e2e-inf-a-a9ba83/test-model-a9ba83-kserve-5f6985cb6c-d6pfg"
Nov 03 20:34:48.020751 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:48.020484    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2d9f055-1380-4eb4-84ee-48dc75855852-dshm\") pod \"test-model-a9ba83-kserve-5f6985cb6c-d6pfg\" (UID: \"b2d9f055-1380-4eb4-84ee-48dc75855852\") " pod="ai-tenant-e2e-inf-a-a9ba83/test-model-a9ba83-kserve-5f6985cb6c-d6pfg"
Nov 03 20:34:48.020751 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:48.020573    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wp6gx\" (UniqueName: \"kubernetes.io/projected/b2d9f055-1380-4eb4-84ee-48dc75855852-kube-api-access-wp6gx\") pod \"test-model-a9ba83-kserve-5f6985cb6c-d6pfg\" (UID: \"b2d9f055-1380-4eb4-84ee-48dc75855852\") " pod="ai-tenant-e2e-inf-a-a9ba83/test-model-a9ba83-kserve-5f6985cb6c-d6pfg"
Nov 03 20:34:48.020751 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:48.020620    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2d9f055-1380-4eb4-84ee-48dc75855852-tls-certs\") pod \"test-model-a9ba83-kserve-5f6985cb6c-d6pfg\" (UID: \"b2d9f055-1380-4eb4-84ee-48dc75855852\") " pod="ai-tenant-e2e-inf-a-a9ba83/test-model-a9ba83-kserve-5f6985cb6c-d6pfg"
Nov 03 20:34:48.020751 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:48.020699    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2d9f055-1380-4eb4-84ee-48dc75855852-home\") pod \"test-model-a9ba83-kserve-5f6985cb6c-d6pfg\" (UID: \"b2d9f055-1380-4eb4-84ee-48dc75855852\") " pod="ai-tenant-e2e-inf-a-a9ba83/test-model-a9ba83-kserve-5f6985cb6c-d6pfg"
Nov 03 20:34:48.021081 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:48.020953    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2d9f055-1380-4eb4-84ee-48dc75855852-model-cache\") pod \"test-model-a9ba83-kserve-5f6985cb6c-d6pfg\" (UID: \"b2d9f055-1380-4eb4-84ee-48dc75855852\") " pod="ai-tenant-e2e-inf-a-a9ba83/test-model-a9ba83-kserve-5f6985cb6c-d6pfg"
Nov 03 20:34:48.021081 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:48.020966    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2d9f055-1380-4eb4-84ee-48dc75855852-tmp-dir\") pod \"test-model-a9ba83-kserve-5f6985cb6c-d6pfg\" (UID: \"b2d9f055-1380-4eb4-84ee-48dc75855852\") " pod="ai-tenant-e2e-inf-a-a9ba83/test-model-a9ba83-kserve-5f6985cb6c-d6pfg"
Nov 03 20:34:48.021181 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:48.021160    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2d9f055-1380-4eb4-84ee-48dc75855852-home\") pod \"test-model-a9ba83-kserve-5f6985cb6c-d6pfg\" (UID: \"b2d9f055-1380-4eb4-84ee-48dc75855852\") " pod="ai-tenant-e2e-inf-a-a9ba83/test-model-a9ba83-kserve-5f6985cb6c-d6pfg"
Nov 03 20:34:48.023001 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:48.022977    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2d9f055-1380-4eb4-84ee-48dc75855852-dshm\") pod \"test-model-a9ba83-kserve-5f6985cb6c-d6pfg\" (UID: \"b2d9f055-1380-4eb4-84ee-48dc75855852\") " pod="ai-tenant-e2e-inf-a-a9ba83/test-model-a9ba83-kserve-5f6985cb6c-d6pfg"
Nov 03 20:34:48.023224 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:48.023206    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2d9f055-1380-4eb4-84ee-48dc75855852-tls-certs\") pod \"test-model-a9ba83-kserve-5f6985cb6c-d6pfg\" (UID: \"b2d9f055-1380-4eb4-84ee-48dc75855852\") " pod="ai-tenant-e2e-inf-a-a9ba83/test-model-a9ba83-kserve-5f6985cb6c-d6pfg"
Nov 03 20:34:48.042945 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:48.042913    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wp6gx\" (UniqueName: \"kubernetes.io/projected/b2d9f055-1380-4eb4-84ee-48dc75855852-kube-api-access-wp6gx\") pod \"test-model-a9ba83-kserve-5f6985cb6c-d6pfg\" (UID: \"b2d9f055-1380-4eb4-84ee-48dc75855852\") " pod="ai-tenant-e2e-inf-a-a9ba83/test-model-a9ba83-kserve-5f6985cb6c-d6pfg"
Nov 03 20:34:48.170849 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:48.170747    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a9ba83/test-model-a9ba83-kserve-5f6985cb6c-d6pfg"
Nov 03 20:34:48.471973 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:48.471932    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-a9ba83/test-model-a9ba83-kserve-5f6985cb6c-d6pfg"]
Nov 03 20:34:48.473761 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:34:48.472479    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb2d9f055_1380_4eb4_84ee_48dc75855852.slice/crio-4730f9b56db16e727239799d26df90df41d167ff9f1be1e4ec4467147ff758b6 WatchSource:0}: Error finding container 4730f9b56db16e727239799d26df90df41d167ff9f1be1e4ec4467147ff758b6: Status 404 returned error can't find the container with id 4730f9b56db16e727239799d26df90df41d167ff9f1be1e4ec4467147ff758b6
Nov 03 20:34:48.658221 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:48.658182    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a9ba83/test-model-a9ba83-kserve-5f6985cb6c-d6pfg" event={"ID":"b2d9f055-1380-4eb4-84ee-48dc75855852","Type":"ContainerStarted","Data":"69369f1cd566da840ae692406efc55dc6741ddcc3e5ea8288eadc64406d7033c"}
Nov 03 20:34:48.658407 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:48.658231    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a9ba83/test-model-a9ba83-kserve-5f6985cb6c-d6pfg" event={"ID":"b2d9f055-1380-4eb4-84ee-48dc75855852","Type":"ContainerStarted","Data":"4730f9b56db16e727239799d26df90df41d167ff9f1be1e4ec4467147ff758b6"}
Nov 03 20:34:48.700129 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:48.700063    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-a9ba83/test-model-a9ba83-kserve-5f6985cb6c-d6pfg" podStartSLOduration=1.700047369 podStartE2EDuration="1.700047369s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:34:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:34:48.698163013 +0000 UTC m=+1822.850824899" watchObservedRunningTime="2026-11-03 20:34:48.700047369 +0000 UTC m=+1822.852709256"
Nov 03 20:34:49.170870 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:49.170832    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-a9ba83/test-model-a9ba83-kserve-5f6985cb6c-d6pfg"
Nov 03 20:34:53.363758 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:53.363721    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-4b95f3/test-model-4b95f3-kserve-54555ccc49-ncvhw"]
Nov 03 20:34:53.364609 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:53.364587    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="974badc1-41ad-461c-8719-e1f3f836ccbc" containerName="istio-proxy"
Nov 03 20:34:53.364737 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:53.364619    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ca8412e-c6de-4cb5-944e-ba8afd2fa791" containerName="istio-proxy"
Nov 03 20:34:53.370462 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:53.369238    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-4b95f3/test-model-4b95f3-kserve-54555ccc49-ncvhw"
Nov 03 20:34:53.404111 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:53.403363    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-4b95f3/test-model-4b95f3-kserve-54555ccc49-ncvhw"]
Nov 03 20:34:53.406641 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:53.406134    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-4b95f3\"/\"test-model-4b95f3-kserve-self-signed-certs\""
Nov 03 20:34:53.406641 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:53.406276    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-4b95f3\"/\"default-dockercfg-gbn6w\""
Nov 03 20:34:53.478026 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:53.477992    5570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-4b95f3\"/\"kube-root-ca.crt\""
Nov 03 20:34:53.478026 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:53.478016    5570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-4b95f3\"/\"openshift-service-ca.crt\""
Nov 03 20:34:53.508327 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:53.508287    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/11cd87af-5c07-4e83-80ad-72c4f7e2482b-tls-certs\") pod \"test-model-4b95f3-kserve-54555ccc49-ncvhw\" (UID: \"11cd87af-5c07-4e83-80ad-72c4f7e2482b\") " pod="ai-tenant-e2e-inf-b-4b95f3/test-model-4b95f3-kserve-54555ccc49-ncvhw"
Nov 03 20:34:53.508516 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:53.508443    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/11cd87af-5c07-4e83-80ad-72c4f7e2482b-home\") pod \"test-model-4b95f3-kserve-54555ccc49-ncvhw\" (UID: \"11cd87af-5c07-4e83-80ad-72c4f7e2482b\") " pod="ai-tenant-e2e-inf-b-4b95f3/test-model-4b95f3-kserve-54555ccc49-ncvhw"
Nov 03 20:34:53.508516 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:53.508486    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/11cd87af-5c07-4e83-80ad-72c4f7e2482b-dshm\") pod \"test-model-4b95f3-kserve-54555ccc49-ncvhw\" (UID: \"11cd87af-5c07-4e83-80ad-72c4f7e2482b\") " pod="ai-tenant-e2e-inf-b-4b95f3/test-model-4b95f3-kserve-54555ccc49-ncvhw"
Nov 03 20:34:53.508689 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:53.508675    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/11cd87af-5c07-4e83-80ad-72c4f7e2482b-model-cache\") pod \"test-model-4b95f3-kserve-54555ccc49-ncvhw\" (UID: \"11cd87af-5c07-4e83-80ad-72c4f7e2482b\") " pod="ai-tenant-e2e-inf-b-4b95f3/test-model-4b95f3-kserve-54555ccc49-ncvhw"
Nov 03 20:34:53.508737 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:53.508710    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/11cd87af-5c07-4e83-80ad-72c4f7e2482b-tmp-dir\") pod \"test-model-4b95f3-kserve-54555ccc49-ncvhw\" (UID: \"11cd87af-5c07-4e83-80ad-72c4f7e2482b\") " pod="ai-tenant-e2e-inf-b-4b95f3/test-model-4b95f3-kserve-54555ccc49-ncvhw"
Nov 03 20:34:53.508737 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:53.508732    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghmvw\" (UniqueName: \"kubernetes.io/projected/11cd87af-5c07-4e83-80ad-72c4f7e2482b-kube-api-access-ghmvw\") pod \"test-model-4b95f3-kserve-54555ccc49-ncvhw\" (UID: \"11cd87af-5c07-4e83-80ad-72c4f7e2482b\") " pod="ai-tenant-e2e-inf-b-4b95f3/test-model-4b95f3-kserve-54555ccc49-ncvhw"
Nov 03 20:34:53.610567 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:53.610527    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/11cd87af-5c07-4e83-80ad-72c4f7e2482b-tmp-dir\") pod \"test-model-4b95f3-kserve-54555ccc49-ncvhw\" (UID: \"11cd87af-5c07-4e83-80ad-72c4f7e2482b\") " pod="ai-tenant-e2e-inf-b-4b95f3/test-model-4b95f3-kserve-54555ccc49-ncvhw"
Nov 03 20:34:53.610766 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:53.610580    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghmvw\" (UniqueName: \"kubernetes.io/projected/11cd87af-5c07-4e83-80ad-72c4f7e2482b-kube-api-access-ghmvw\") pod \"test-model-4b95f3-kserve-54555ccc49-ncvhw\" (UID: \"11cd87af-5c07-4e83-80ad-72c4f7e2482b\") " pod="ai-tenant-e2e-inf-b-4b95f3/test-model-4b95f3-kserve-54555ccc49-ncvhw"
Nov 03 20:34:53.610766 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:53.610658    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/11cd87af-5c07-4e83-80ad-72c4f7e2482b-tls-certs\") pod \"test-model-4b95f3-kserve-54555ccc49-ncvhw\" (UID: \"11cd87af-5c07-4e83-80ad-72c4f7e2482b\") " pod="ai-tenant-e2e-inf-b-4b95f3/test-model-4b95f3-kserve-54555ccc49-ncvhw"
Nov 03 20:34:53.610766 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:53.610741    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/11cd87af-5c07-4e83-80ad-72c4f7e2482b-home\") pod \"test-model-4b95f3-kserve-54555ccc49-ncvhw\" (UID: \"11cd87af-5c07-4e83-80ad-72c4f7e2482b\") " pod="ai-tenant-e2e-inf-b-4b95f3/test-model-4b95f3-kserve-54555ccc49-ncvhw"
Nov 03 20:34:53.610766 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:53.610793    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/11cd87af-5c07-4e83-80ad-72c4f7e2482b-dshm\") pod \"test-model-4b95f3-kserve-54555ccc49-ncvhw\" (UID: \"11cd87af-5c07-4e83-80ad-72c4f7e2482b\") " pod="ai-tenant-e2e-inf-b-4b95f3/test-model-4b95f3-kserve-54555ccc49-ncvhw"
Nov 03 20:34:53.611112 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:53.611075    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/11cd87af-5c07-4e83-80ad-72c4f7e2482b-tmp-dir\") pod \"test-model-4b95f3-kserve-54555ccc49-ncvhw\" (UID: \"11cd87af-5c07-4e83-80ad-72c4f7e2482b\") " pod="ai-tenant-e2e-inf-b-4b95f3/test-model-4b95f3-kserve-54555ccc49-ncvhw"
Nov 03 20:34:53.611237 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:53.611214    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/11cd87af-5c07-4e83-80ad-72c4f7e2482b-home\") pod \"test-model-4b95f3-kserve-54555ccc49-ncvhw\" (UID: \"11cd87af-5c07-4e83-80ad-72c4f7e2482b\") " pod="ai-tenant-e2e-inf-b-4b95f3/test-model-4b95f3-kserve-54555ccc49-ncvhw"
Nov 03 20:34:53.611601 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:53.611574    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/11cd87af-5c07-4e83-80ad-72c4f7e2482b-model-cache\") pod \"test-model-4b95f3-kserve-54555ccc49-ncvhw\" (UID: \"11cd87af-5c07-4e83-80ad-72c4f7e2482b\") " pod="ai-tenant-e2e-inf-b-4b95f3/test-model-4b95f3-kserve-54555ccc49-ncvhw"
Nov 03 20:34:53.611925 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:53.611903    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/11cd87af-5c07-4e83-80ad-72c4f7e2482b-model-cache\") pod \"test-model-4b95f3-kserve-54555ccc49-ncvhw\" (UID: \"11cd87af-5c07-4e83-80ad-72c4f7e2482b\") " pod="ai-tenant-e2e-inf-b-4b95f3/test-model-4b95f3-kserve-54555ccc49-ncvhw"
Nov 03 20:34:53.613459 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:53.613441    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/11cd87af-5c07-4e83-80ad-72c4f7e2482b-dshm\") pod \"test-model-4b95f3-kserve-54555ccc49-ncvhw\" (UID: \"11cd87af-5c07-4e83-80ad-72c4f7e2482b\") " pod="ai-tenant-e2e-inf-b-4b95f3/test-model-4b95f3-kserve-54555ccc49-ncvhw"
Nov 03 20:34:53.614098 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:53.614058    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/11cd87af-5c07-4e83-80ad-72c4f7e2482b-tls-certs\") pod \"test-model-4b95f3-kserve-54555ccc49-ncvhw\" (UID: \"11cd87af-5c07-4e83-80ad-72c4f7e2482b\") " pod="ai-tenant-e2e-inf-b-4b95f3/test-model-4b95f3-kserve-54555ccc49-ncvhw"
Nov 03 20:34:53.631934 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:53.631904    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghmvw\" (UniqueName: \"kubernetes.io/projected/11cd87af-5c07-4e83-80ad-72c4f7e2482b-kube-api-access-ghmvw\") pod \"test-model-4b95f3-kserve-54555ccc49-ncvhw\" (UID: \"11cd87af-5c07-4e83-80ad-72c4f7e2482b\") " pod="ai-tenant-e2e-inf-b-4b95f3/test-model-4b95f3-kserve-54555ccc49-ncvhw"
Nov 03 20:34:53.719805 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:53.711239    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-4b95f3/test-model-4b95f3-kserve-54555ccc49-ncvhw"
Nov 03 20:34:54.173300 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:54.172670    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-4b95f3/test-model-4b95f3-kserve-54555ccc49-ncvhw"]
Nov 03 20:34:54.752452 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:54.752411    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-4b95f3/test-model-4b95f3-kserve-54555ccc49-ncvhw" event={"ID":"11cd87af-5c07-4e83-80ad-72c4f7e2482b","Type":"ContainerStarted","Data":"c2d718a731911d49f885ce4a3e274c1b444f601ddca4fb57b7cfa54d63d73154"}
Nov 03 20:34:54.752898 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:54.752462    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-4b95f3/test-model-4b95f3-kserve-54555ccc49-ncvhw" event={"ID":"11cd87af-5c07-4e83-80ad-72c4f7e2482b","Type":"ContainerStarted","Data":"6834c83c13b04814877ab18d3610e4a28602d6ecf5ea20db49d826a6fce7287d"}
Nov 03 20:34:54.796648 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:54.796583    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-4b95f3/test-model-4b95f3-kserve-54555ccc49-ncvhw" podStartSLOduration=1.7965687240000001 podStartE2EDuration="1.796568724s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:34:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:34:54.793576995 +0000 UTC m=+1828.946238880" watchObservedRunningTime="2026-11-03 20:34:54.796568724 +0000 UTC m=+1828.949230610"
Nov 03 20:34:55.721255 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:55.721219    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-4b95f3/test-model-4b95f3-kserve-54555ccc49-ncvhw"
Nov 03 20:34:57.229118 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:34:57.229067    5570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8bwhz" containerName="extractor"
Nov 03 20:34:58.171934 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:58.171880    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-a9ba83/test-model-a9ba83-kserve-5f6985cb6c-d6pfg"
Nov 03 20:34:58.184530 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:58.184503    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-a9ba83/test-model-a9ba83-kserve-5f6985cb6c-d6pfg"
Nov 03 20:34:58.789522 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:34:58.789483    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-a9ba83/test-model-a9ba83-kserve-5f6985cb6c-d6pfg"
Nov 03 20:35:03.720635 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:35:03.720591    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-4b95f3/test-model-4b95f3-kserve-54555ccc49-ncvhw"
Nov 03 20:35:03.733930 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:35:03.733898    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-4b95f3/test-model-4b95f3-kserve-54555ccc49-ncvhw"
Nov 03 20:35:03.821428 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:35:03.821388    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-4b95f3/test-model-4b95f3-kserve-54555ccc49-ncvhw"
Nov 03 20:35:07.163084 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:35:07.163046    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g69f8"]
Nov 03 20:35:18.030841 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:35:18.030791    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a42b52f-j678v"]
Nov 03 20:35:18.072147 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:35:18.072078    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a42b52f-j678v"]
Nov 03 20:35:19.976521 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:35:19.976472    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="742793f0-4170-46dd-aa89-31df5f833c45" path="/var/lib/kubelet/pods/742793f0-4170-46dd-aa89-31df5f833c45/volumes"
Nov 03 20:35:27.230511 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:35:27.230464    5570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8bwhz" containerName="extractor"
Nov 03 20:35:28.749726 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:35:28.749690    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g69f8"]
Nov 03 20:35:50.351454 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:35:50.351419    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g69f8"]
Nov 03 20:35:57.231985 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:35:57.231944    5570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8bwhz" containerName="extractor"
Nov 03 20:36:11.774680 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:11.774637    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g69f8"]
Nov 03 20:36:12.786814 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:12.786732    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-60efdf/test-model-60efdf-kserve-7fb9cb65bf-9774z"]
Nov 03 20:36:24.725491 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:24.725452    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-1d159a/test-model-1d159a-kserve-5766d9c789-lqs6f"]
Nov 03 20:36:24.985941 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:24.985838    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-60efdf-4gbsk"]
Nov 03 20:36:24.997193 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:24.997167    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-60efdf-4gbsk"
Nov 03 20:36:24.998117 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:24.998075    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-60efdf-4gbsk"]
Nov 03 20:36:25.015458 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:25.015420    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:36:25.015645 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:25.015493    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l7s5w\""
Nov 03 20:36:25.156016 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:25.155970    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/933a606e-1a38-47dc-9781-3a5730b0c0c3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-60efdf-4gbsk\" (UID: \"933a606e-1a38-47dc-9781-3a5730b0c0c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-60efdf-4gbsk"
Nov 03 20:36:25.156228 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:25.156027    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xn7h6\" (UniqueName: \"kubernetes.io/projected/933a606e-1a38-47dc-9781-3a5730b0c0c3-kube-api-access-xn7h6\") pod \"maas-api-revoke-keys-e2e-inf-a-60efdf-4gbsk\" (UID: \"933a606e-1a38-47dc-9781-3a5730b0c0c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-60efdf-4gbsk"
Nov 03 20:36:25.156281 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:25.156236    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/933a606e-1a38-47dc-9781-3a5730b0c0c3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-60efdf-4gbsk\" (UID: \"933a606e-1a38-47dc-9781-3a5730b0c0c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-60efdf-4gbsk"
Nov 03 20:36:25.257543 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:25.257437    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/933a606e-1a38-47dc-9781-3a5730b0c0c3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-60efdf-4gbsk\" (UID: \"933a606e-1a38-47dc-9781-3a5730b0c0c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-60efdf-4gbsk"
Nov 03 20:36:25.257543 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:25.257496    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xn7h6\" (UniqueName: \"kubernetes.io/projected/933a606e-1a38-47dc-9781-3a5730b0c0c3-kube-api-access-xn7h6\") pod \"maas-api-revoke-keys-e2e-inf-a-60efdf-4gbsk\" (UID: \"933a606e-1a38-47dc-9781-3a5730b0c0c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-60efdf-4gbsk"
Nov 03 20:36:25.257847 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:25.257664    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/933a606e-1a38-47dc-9781-3a5730b0c0c3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-60efdf-4gbsk\" (UID: \"933a606e-1a38-47dc-9781-3a5730b0c0c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-60efdf-4gbsk"
Nov 03 20:36:25.258316 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:25.258286    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/933a606e-1a38-47dc-9781-3a5730b0c0c3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-60efdf-4gbsk\" (UID: \"933a606e-1a38-47dc-9781-3a5730b0c0c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-60efdf-4gbsk"
Nov 03 20:36:25.259905 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:25.259862    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/933a606e-1a38-47dc-9781-3a5730b0c0c3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-60efdf-4gbsk\" (UID: \"933a606e-1a38-47dc-9781-3a5730b0c0c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-60efdf-4gbsk"
Nov 03 20:36:25.277718 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:25.277692    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xn7h6\" (UniqueName: \"kubernetes.io/projected/933a606e-1a38-47dc-9781-3a5730b0c0c3-kube-api-access-xn7h6\") pod \"maas-api-revoke-keys-e2e-inf-a-60efdf-4gbsk\" (UID: \"933a606e-1a38-47dc-9781-3a5730b0c0c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-60efdf-4gbsk"
Nov 03 20:36:25.319831 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:25.319769    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-60efdf-4gbsk"
Nov 03 20:36:25.623707 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:25.623672    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-60efdf-4gbsk"]
Nov 03 20:36:25.625156 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:36:25.625115    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod933a606e_1a38_47dc_9781_3a5730b0c0c3.slice/crio-32c859173c4ddffbc01d54fbba68bc99b0a9335507a4b4579dfc1774538f8e29 WatchSource:0}: Error finding container 32c859173c4ddffbc01d54fbba68bc99b0a9335507a4b4579dfc1774538f8e29: Status 404 returned error can't find the container with id 32c859173c4ddffbc01d54fbba68bc99b0a9335507a4b4579dfc1774538f8e29
Nov 03 20:36:26.127363 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:26.127326    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g69f8"]
Nov 03 20:36:26.379527 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:26.379435    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-60efdf-4gbsk" event={"ID":"933a606e-1a38-47dc-9781-3a5730b0c0c3","Type":"ContainerStarted","Data":"155122ce70bbb1e1a1de2f68df5cf6b21b6860787f71654a7080ff6bea9b0bc4"}
Nov 03 20:36:26.379527 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:26.379494    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-60efdf-4gbsk" event={"ID":"933a606e-1a38-47dc-9781-3a5730b0c0c3","Type":"ContainerStarted","Data":"32c859173c4ddffbc01d54fbba68bc99b0a9335507a4b4579dfc1774538f8e29"}
Nov 03 20:36:26.421824 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:26.421719    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-60efdf-4gbsk" podStartSLOduration=2.421700408 podStartE2EDuration="2.421700408s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:36:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:36:26.418547901 +0000 UTC m=+1920.571209790" watchObservedRunningTime="2026-11-03 20:36:26.421700408 +0000 UTC m=+1920.574362296"
Nov 03 20:36:27.233293 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:36:27.233260    5570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8bwhz" containerName="extractor"
Nov 03 20:36:27.387971 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:27.387941    5570 generic.go:356] "Generic (PLEG): container finished" podID="933a606e-1a38-47dc-9781-3a5730b0c0c3" containerID="155122ce70bbb1e1a1de2f68df5cf6b21b6860787f71654a7080ff6bea9b0bc4" exitCode=0
Nov 03 20:36:27.388177 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:27.387992    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-60efdf-4gbsk" event={"ID":"933a606e-1a38-47dc-9781-3a5730b0c0c3","Type":"ContainerDied","Data":"155122ce70bbb1e1a1de2f68df5cf6b21b6860787f71654a7080ff6bea9b0bc4"}
Nov 03 20:36:27.820868 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:27.820820    5570 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-60efdf/test-model-60efdf-kserve-7fb9cb65bf-9774z" podUID="543a8a92-e2b1-4d88-9b3f-4473806a9a3d" containerName="main" containerID="cri-o://c3dad758f122394a07e2faf548e93e71df95738b6e01e1bfcf3c442d86914f6d" gracePeriod=45
Nov 03 20:36:28.147952 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:28.147920    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-60efdf/test-model-60efdf-kserve-7fb9cb65bf-9774z"
Nov 03 20:36:28.302177 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:28.302137    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/543a8a92-e2b1-4d88-9b3f-4473806a9a3d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/543a8a92-e2b1-4d88-9b3f-4473806a9a3d-dshm\") pod \"543a8a92-e2b1-4d88-9b3f-4473806a9a3d\" (UID: \"543a8a92-e2b1-4d88-9b3f-4473806a9a3d\") "
Nov 03 20:36:28.302676 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:28.302223    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/543a8a92-e2b1-4d88-9b3f-4473806a9a3d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/543a8a92-e2b1-4d88-9b3f-4473806a9a3d-model-cache\") pod \"543a8a92-e2b1-4d88-9b3f-4473806a9a3d\" (UID: \"543a8a92-e2b1-4d88-9b3f-4473806a9a3d\") "
Nov 03 20:36:28.302676 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:28.302390    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/543a8a92-e2b1-4d88-9b3f-4473806a9a3d-tls-certs\" (UniqueName: \"kubernetes.io/secret/543a8a92-e2b1-4d88-9b3f-4473806a9a3d-tls-certs\") pod \"543a8a92-e2b1-4d88-9b3f-4473806a9a3d\" (UID: \"543a8a92-e2b1-4d88-9b3f-4473806a9a3d\") "
Nov 03 20:36:28.302676 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:28.302425    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/543a8a92-e2b1-4d88-9b3f-4473806a9a3d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/543a8a92-e2b1-4d88-9b3f-4473806a9a3d-tmp-dir\") pod \"543a8a92-e2b1-4d88-9b3f-4473806a9a3d\" (UID: \"543a8a92-e2b1-4d88-9b3f-4473806a9a3d\") "
Nov 03 20:36:28.302676 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:28.302481    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/543a8a92-e2b1-4d88-9b3f-4473806a9a3d-model-cache" pod "543a8a92-e2b1-4d88-9b3f-4473806a9a3d" (UID: "543a8a92-e2b1-4d88-9b3f-4473806a9a3d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:36:28.302676 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:28.302531    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/543a8a92-e2b1-4d88-9b3f-4473806a9a3d-home\" (UniqueName: \"kubernetes.io/empty-dir/543a8a92-e2b1-4d88-9b3f-4473806a9a3d-home\") pod \"543a8a92-e2b1-4d88-9b3f-4473806a9a3d\" (UID: \"543a8a92-e2b1-4d88-9b3f-4473806a9a3d\") "
Nov 03 20:36:28.302676 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:28.302601    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/543a8a92-e2b1-4d88-9b3f-4473806a9a3d-kube-api-access-5h78z\" (UniqueName: \"kubernetes.io/projected/543a8a92-e2b1-4d88-9b3f-4473806a9a3d-kube-api-access-5h78z\") pod \"543a8a92-e2b1-4d88-9b3f-4473806a9a3d\" (UID: \"543a8a92-e2b1-4d88-9b3f-4473806a9a3d\") "
Nov 03 20:36:28.303231 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:28.302802    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/543a8a92-e2b1-4d88-9b3f-4473806a9a3d-tmp-dir" pod "543a8a92-e2b1-4d88-9b3f-4473806a9a3d" (UID: "543a8a92-e2b1-4d88-9b3f-4473806a9a3d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:36:28.303231 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:28.302807    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/543a8a92-e2b1-4d88-9b3f-4473806a9a3d-home" pod "543a8a92-e2b1-4d88-9b3f-4473806a9a3d" (UID: "543a8a92-e2b1-4d88-9b3f-4473806a9a3d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:36:28.303231 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:28.303183    5570 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/543a8a92-e2b1-4d88-9b3f-4473806a9a3d-tmp-dir\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:36:28.303231 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:28.303205    5570 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/543a8a92-e2b1-4d88-9b3f-4473806a9a3d-home\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:36:28.303231 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:28.303218    5570 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/543a8a92-e2b1-4d88-9b3f-4473806a9a3d-model-cache\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:36:28.304642 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:28.304613    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/543a8a92-e2b1-4d88-9b3f-4473806a9a3d-kube-api-access-5h78z" pod "543a8a92-e2b1-4d88-9b3f-4473806a9a3d" (UID: "543a8a92-e2b1-4d88-9b3f-4473806a9a3d"). InnerVolumeSpecName "kube-api-access-5h78z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:36:28.304761 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:28.304660    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/543a8a92-e2b1-4d88-9b3f-4473806a9a3d-tls-certs" pod "543a8a92-e2b1-4d88-9b3f-4473806a9a3d" (UID: "543a8a92-e2b1-4d88-9b3f-4473806a9a3d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:36:28.304761 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:28.304671    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/543a8a92-e2b1-4d88-9b3f-4473806a9a3d-dshm" pod "543a8a92-e2b1-4d88-9b3f-4473806a9a3d" (UID: "543a8a92-e2b1-4d88-9b3f-4473806a9a3d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:36:28.395988 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:28.395876    5570 generic.go:356] "Generic (PLEG): container finished" podID="543a8a92-e2b1-4d88-9b3f-4473806a9a3d" containerID="c3dad758f122394a07e2faf548e93e71df95738b6e01e1bfcf3c442d86914f6d" exitCode=0
Nov 03 20:36:28.395988 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:28.395939    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-60efdf/test-model-60efdf-kserve-7fb9cb65bf-9774z"
Nov 03 20:36:28.395988 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:28.395954    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-60efdf/test-model-60efdf-kserve-7fb9cb65bf-9774z" event={"ID":"543a8a92-e2b1-4d88-9b3f-4473806a9a3d","Type":"ContainerDied","Data":"c3dad758f122394a07e2faf548e93e71df95738b6e01e1bfcf3c442d86914f6d"}
Nov 03 20:36:28.396273 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:28.396001    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-60efdf/test-model-60efdf-kserve-7fb9cb65bf-9774z" event={"ID":"543a8a92-e2b1-4d88-9b3f-4473806a9a3d","Type":"ContainerDied","Data":"3ca1594bd9a0812176dd8b2dbfb67883c5b624103286ac4cd3bd73dd3f208db0"}
Nov 03 20:36:28.396273 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:28.396020    5570 scope.go:122] "RemoveContainer" containerID="c3dad758f122394a07e2faf548e93e71df95738b6e01e1bfcf3c442d86914f6d"
Nov 03 20:36:28.405361 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:28.405073    5570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/543a8a92-e2b1-4d88-9b3f-4473806a9a3d-tls-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:36:28.405361 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:28.405110    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5h78z\" (UniqueName: \"kubernetes.io/projected/543a8a92-e2b1-4d88-9b3f-4473806a9a3d-kube-api-access-5h78z\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:36:28.405361 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:28.405126    5570 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/543a8a92-e2b1-4d88-9b3f-4473806a9a3d-dshm\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:36:28.407916 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:28.407878    5570 scope.go:122] "RemoveContainer" containerID="c3dad758f122394a07e2faf548e93e71df95738b6e01e1bfcf3c442d86914f6d"
Nov 03 20:36:28.408303 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:36:28.408283    5570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3dad758f122394a07e2faf548e93e71df95738b6e01e1bfcf3c442d86914f6d\": container with ID starting with c3dad758f122394a07e2faf548e93e71df95738b6e01e1bfcf3c442d86914f6d not found: ID does not exist" containerID="c3dad758f122394a07e2faf548e93e71df95738b6e01e1bfcf3c442d86914f6d"
Nov 03 20:36:28.408380 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:28.408318    5570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3dad758f122394a07e2faf548e93e71df95738b6e01e1bfcf3c442d86914f6d"} err="failed to get container status \"c3dad758f122394a07e2faf548e93e71df95738b6e01e1bfcf3c442d86914f6d\": rpc error: code = NotFound desc = could not find container \"c3dad758f122394a07e2faf548e93e71df95738b6e01e1bfcf3c442d86914f6d\": container with ID starting with c3dad758f122394a07e2faf548e93e71df95738b6e01e1bfcf3c442d86914f6d not found: ID does not exist"
Nov 03 20:36:28.450440 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:28.450381    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-60efdf/test-model-60efdf-kserve-7fb9cb65bf-9774z"]
Nov 03 20:36:28.453559 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:28.453528    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-60efdf/test-model-60efdf-kserve-7fb9cb65bf-9774z"]
Nov 03 20:36:28.535164 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:28.535134    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-60efdf-4gbsk"
Nov 03 20:36:28.708935 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:28.708899    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/933a606e-1a38-47dc-9781-3a5730b0c0c3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/933a606e-1a38-47dc-9781-3a5730b0c0c3-maas-api-xks-ca\") pod \"933a606e-1a38-47dc-9781-3a5730b0c0c3\" (UID: \"933a606e-1a38-47dc-9781-3a5730b0c0c3\") "
Nov 03 20:36:28.708935 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:28.708938    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/933a606e-1a38-47dc-9781-3a5730b0c0c3-kube-api-access-xn7h6\" (UniqueName: \"kubernetes.io/projected/933a606e-1a38-47dc-9781-3a5730b0c0c3-kube-api-access-xn7h6\") pod \"933a606e-1a38-47dc-9781-3a5730b0c0c3\" (UID: \"933a606e-1a38-47dc-9781-3a5730b0c0c3\") "
Nov 03 20:36:28.709189 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:28.709148    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/933a606e-1a38-47dc-9781-3a5730b0c0c3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/933a606e-1a38-47dc-9781-3a5730b0c0c3-maas-api-openshift-ca\") pod \"933a606e-1a38-47dc-9781-3a5730b0c0c3\" (UID: \"933a606e-1a38-47dc-9781-3a5730b0c0c3\") "
Nov 03 20:36:28.709542 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:28.709513    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/933a606e-1a38-47dc-9781-3a5730b0c0c3-maas-api-openshift-ca" pod "933a606e-1a38-47dc-9781-3a5730b0c0c3" (UID: "933a606e-1a38-47dc-9781-3a5730b0c0c3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:36:28.709655 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:28.709613    5570 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/933a606e-1a38-47dc-9781-3a5730b0c0c3-maas-api-openshift-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:36:28.711131 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:28.711103    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/933a606e-1a38-47dc-9781-3a5730b0c0c3-maas-api-xks-ca" pod "933a606e-1a38-47dc-9781-3a5730b0c0c3" (UID: "933a606e-1a38-47dc-9781-3a5730b0c0c3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:36:28.711256 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:28.711177    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/933a606e-1a38-47dc-9781-3a5730b0c0c3-kube-api-access-xn7h6" pod "933a606e-1a38-47dc-9781-3a5730b0c0c3" (UID: "933a606e-1a38-47dc-9781-3a5730b0c0c3"). InnerVolumeSpecName "kube-api-access-xn7h6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:36:28.811489 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:28.811450    5570 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/933a606e-1a38-47dc-9781-3a5730b0c0c3-maas-api-xks-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:36:28.811489 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:28.811482    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xn7h6\" (UniqueName: \"kubernetes.io/projected/933a606e-1a38-47dc-9781-3a5730b0c0c3-kube-api-access-xn7h6\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:36:29.405244 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:29.405210    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-60efdf-4gbsk"
Nov 03 20:36:29.405787 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:29.405211    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-60efdf-4gbsk" event={"ID":"933a606e-1a38-47dc-9781-3a5730b0c0c3","Type":"ContainerDied","Data":"32c859173c4ddffbc01d54fbba68bc99b0a9335507a4b4579dfc1774538f8e29"}
Nov 03 20:36:29.405787 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:29.405352    5570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="32c859173c4ddffbc01d54fbba68bc99b0a9335507a4b4579dfc1774538f8e29"
Nov 03 20:36:29.966419 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:29.966383    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="543a8a92-e2b1-4d88-9b3f-4473806a9a3d" path="/var/lib/kubelet/pods/543a8a92-e2b1-4d88-9b3f-4473806a9a3d/volumes"
Nov 03 20:36:31.584659 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:31.584625    5570 scope.go:122] "RemoveContainer" containerID="47143fc84098b81ae0a30abe3c762f33ac146636ea15729c1d08e8282b2558a4"
Nov 03 20:36:35.405415 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:35.405375    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-60efdf-7cb6d9fbc8-fwx62"]
Nov 03 20:36:35.437656 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:35.437618    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-60efdf-7bfdc67bd7-j6ctn"]
Nov 03 20:36:35.532658 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:35.532624    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-60efdf-57848f5c8c-qjzhb"]
Nov 03 20:36:35.533923 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:35.533885    5570 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-60efdf-57848f5c8c-qjzhb" podUID="ca0be3e9-2414-4d9c-ae41-fcbe4dddb52a" containerName="maas-api" containerID="cri-o://1e461e7c4090b27d313b295321bafd3383e35d71c468c4df753a994ba95cdf41" gracePeriod=30
Nov 03 20:36:36.150483 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:36.150455    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-60efdf-57848f5c8c-qjzhb"
Nov 03 20:36:36.250558 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:36.250515    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-a9ba83/test-model-a9ba83-kserve-5f6985cb6c-d6pfg"]
Nov 03 20:36:36.368806 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:36.358731    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca0be3e9-2414-4d9c-ae41-fcbe4dddb52a-kube-api-access-6v6n6\" (UniqueName: \"kubernetes.io/projected/ca0be3e9-2414-4d9c-ae41-fcbe4dddb52a-kube-api-access-6v6n6\") pod \"ca0be3e9-2414-4d9c-ae41-fcbe4dddb52a\" (UID: \"ca0be3e9-2414-4d9c-ae41-fcbe4dddb52a\") "
Nov 03 20:36:36.369704 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:36.369667    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca0be3e9-2414-4d9c-ae41-fcbe4dddb52a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca0be3e9-2414-4d9c-ae41-fcbe4dddb52a-openshift-service-ca\") pod \"ca0be3e9-2414-4d9c-ae41-fcbe4dddb52a\" (UID: \"ca0be3e9-2414-4d9c-ae41-fcbe4dddb52a\") "
Nov 03 20:36:36.370689 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:36.370575    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca0be3e9-2414-4d9c-ae41-fcbe4dddb52a-openshift-service-ca" pod "ca0be3e9-2414-4d9c-ae41-fcbe4dddb52a" (UID: "ca0be3e9-2414-4d9c-ae41-fcbe4dddb52a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:36:36.371074 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:36.371056    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ca0be3e9-2414-4d9c-ae41-fcbe4dddb52a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca0be3e9-2414-4d9c-ae41-fcbe4dddb52a-maas-api-tls\") pod \"ca0be3e9-2414-4d9c-ae41-fcbe4dddb52a\" (UID: \"ca0be3e9-2414-4d9c-ae41-fcbe4dddb52a\") "
Nov 03 20:36:36.371565 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:36.371547    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ca0be3e9-2414-4d9c-ae41-fcbe4dddb52a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca0be3e9-2414-4d9c-ae41-fcbe4dddb52a-metrics-certs\") pod \"ca0be3e9-2414-4d9c-ae41-fcbe4dddb52a\" (UID: \"ca0be3e9-2414-4d9c-ae41-fcbe4dddb52a\") "
Nov 03 20:36:36.372702 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:36.372643    5570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca0be3e9-2414-4d9c-ae41-fcbe4dddb52a-openshift-service-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:36:36.384335 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:36.384277    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca0be3e9-2414-4d9c-ae41-fcbe4dddb52a-metrics-certs" pod "ca0be3e9-2414-4d9c-ae41-fcbe4dddb52a" (UID: "ca0be3e9-2414-4d9c-ae41-fcbe4dddb52a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:36:36.388024 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:36.387223    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca0be3e9-2414-4d9c-ae41-fcbe4dddb52a-maas-api-tls" pod "ca0be3e9-2414-4d9c-ae41-fcbe4dddb52a" (UID: "ca0be3e9-2414-4d9c-ae41-fcbe4dddb52a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:36:36.402616 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:36.400034    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca0be3e9-2414-4d9c-ae41-fcbe4dddb52a-kube-api-access-6v6n6" pod "ca0be3e9-2414-4d9c-ae41-fcbe4dddb52a" (UID: "ca0be3e9-2414-4d9c-ae41-fcbe4dddb52a"). InnerVolumeSpecName "kube-api-access-6v6n6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:36:36.462632 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:36.462599    5570 generic.go:356] "Generic (PLEG): container finished" podID="ca0be3e9-2414-4d9c-ae41-fcbe4dddb52a" containerID="1e461e7c4090b27d313b295321bafd3383e35d71c468c4df753a994ba95cdf41" exitCode=0
Nov 03 20:36:36.463087 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:36.462646    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-60efdf-57848f5c8c-qjzhb"
Nov 03 20:36:36.463087 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:36.462683    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-60efdf-57848f5c8c-qjzhb" event={"ID":"ca0be3e9-2414-4d9c-ae41-fcbe4dddb52a","Type":"ContainerDied","Data":"1e461e7c4090b27d313b295321bafd3383e35d71c468c4df753a994ba95cdf41"}
Nov 03 20:36:36.463087 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:36.462743    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-60efdf-57848f5c8c-qjzhb" event={"ID":"ca0be3e9-2414-4d9c-ae41-fcbe4dddb52a","Type":"ContainerDied","Data":"fcd179dae740b6a6ec64118dedae08cd35fef554e78dc03b798b1aa37f8baedd"}
Nov 03 20:36:36.463087 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:36.462786    5570 scope.go:122] "RemoveContainer" containerID="1e461e7c4090b27d313b295321bafd3383e35d71c468c4df753a994ba95cdf41"
Nov 03 20:36:36.473371 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:36.473349    5570 scope.go:122] "RemoveContainer" containerID="1e461e7c4090b27d313b295321bafd3383e35d71c468c4df753a994ba95cdf41"
Nov 03 20:36:36.473793 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:36:36.473754    5570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e461e7c4090b27d313b295321bafd3383e35d71c468c4df753a994ba95cdf41\": container with ID starting with 1e461e7c4090b27d313b295321bafd3383e35d71c468c4df753a994ba95cdf41 not found: ID does not exist" containerID="1e461e7c4090b27d313b295321bafd3383e35d71c468c4df753a994ba95cdf41"
Nov 03 20:36:36.473882 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:36.473803    5570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e461e7c4090b27d313b295321bafd3383e35d71c468c4df753a994ba95cdf41"} err="failed to get container status \"1e461e7c4090b27d313b295321bafd3383e35d71c468c4df753a994ba95cdf41\": rpc error: code = NotFound desc = could not find container \"1e461e7c4090b27d313b295321bafd3383e35d71c468c4df753a994ba95cdf41\": container with ID starting with 1e461e7c4090b27d313b295321bafd3383e35d71c468c4df753a994ba95cdf41 not found: ID does not exist"
Nov 03 20:36:36.474247 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:36.474228    5570 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca0be3e9-2414-4d9c-ae41-fcbe4dddb52a-maas-api-tls\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:36:36.474284 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:36.474250    5570 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca0be3e9-2414-4d9c-ae41-fcbe4dddb52a-metrics-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:36:36.474284 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:36.474259    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6v6n6\" (UniqueName: \"kubernetes.io/projected/ca0be3e9-2414-4d9c-ae41-fcbe4dddb52a-kube-api-access-6v6n6\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:36:36.518757 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:36.518708    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-60efdf-57848f5c8c-qjzhb"]
Nov 03 20:36:36.526141 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:36.526108    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-60efdf-57848f5c8c-qjzhb"]
Nov 03 20:36:36.792200 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:36.792165    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e33e59be-fmxqt"]
Nov 03 20:36:36.792858 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:36.792836    5570 cpu_manager.go:414] "containerMap: removing container" podUID="543a8a92-e2b1-4d88-9b3f-4473806a9a3d" containerName="main"
Nov 03 20:36:36.792858 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:36.792857    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="543a8a92-e2b1-4d88-9b3f-4473806a9a3d" containerName="main"
Nov 03 20:36:36.793062 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:36.792877    5570 cpu_manager.go:414] "containerMap: removing container" podUID="ca0be3e9-2414-4d9c-ae41-fcbe4dddb52a" containerName="maas-api"
Nov 03 20:36:36.793062 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:36.792885    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca0be3e9-2414-4d9c-ae41-fcbe4dddb52a" containerName="maas-api"
Nov 03 20:36:36.793062 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:36.792930    5570 cpu_manager.go:414] "containerMap: removing container" podUID="933a606e-1a38-47dc-9781-3a5730b0c0c3" containerName="revoke-keys"
Nov 03 20:36:36.793062 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:36.792940    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="933a606e-1a38-47dc-9781-3a5730b0c0c3" containerName="revoke-keys"
Nov 03 20:36:36.793271 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:36.793108    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="933a606e-1a38-47dc-9781-3a5730b0c0c3" containerName="revoke-keys"
Nov 03 20:36:36.793271 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:36.793133    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="543a8a92-e2b1-4d88-9b3f-4473806a9a3d" containerName="main"
Nov 03 20:36:36.793271 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:36.793144    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca0be3e9-2414-4d9c-ae41-fcbe4dddb52a" containerName="maas-api"
Nov 03 20:36:36.798069 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:36.798044    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e33e59be-fmxqt"
Nov 03 20:36:36.814148 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:36.814116    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:36:36.814340 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:36.814170    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l7s5w\""
Nov 03 20:36:36.817458 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:36.817432    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e33e59be-fmxqt"]
Nov 03 20:36:36.879460 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:36.879418    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/426afdc6-cfdd-47c9-9ba9-6da128c31913-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e33e59be-fmxqt\" (UID: \"426afdc6-cfdd-47c9-9ba9-6da128c31913\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e33e59be-fmxqt"
Nov 03 20:36:36.879660 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:36.879550    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/426afdc6-cfdd-47c9-9ba9-6da128c31913-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e33e59be-fmxqt\" (UID: \"426afdc6-cfdd-47c9-9ba9-6da128c31913\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e33e59be-fmxqt"
Nov 03 20:36:36.879730 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:36.879712    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwsxs\" (UniqueName: \"kubernetes.io/projected/426afdc6-cfdd-47c9-9ba9-6da128c31913-kube-api-access-kwsxs\") pod \"maas-api-revoke-sub-e33e59be-fmxqt\" (UID: \"426afdc6-cfdd-47c9-9ba9-6da128c31913\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e33e59be-fmxqt"
Nov 03 20:36:36.981408 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:36.981368    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwsxs\" (UniqueName: \"kubernetes.io/projected/426afdc6-cfdd-47c9-9ba9-6da128c31913-kube-api-access-kwsxs\") pod \"maas-api-revoke-sub-e33e59be-fmxqt\" (UID: \"426afdc6-cfdd-47c9-9ba9-6da128c31913\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e33e59be-fmxqt"
Nov 03 20:36:36.981607 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:36.981475    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/426afdc6-cfdd-47c9-9ba9-6da128c31913-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e33e59be-fmxqt\" (UID: \"426afdc6-cfdd-47c9-9ba9-6da128c31913\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e33e59be-fmxqt"
Nov 03 20:36:36.981607 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:36.981558    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/426afdc6-cfdd-47c9-9ba9-6da128c31913-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e33e59be-fmxqt\" (UID: \"426afdc6-cfdd-47c9-9ba9-6da128c31913\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e33e59be-fmxqt"
Nov 03 20:36:36.982291 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:36.982261    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/426afdc6-cfdd-47c9-9ba9-6da128c31913-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e33e59be-fmxqt\" (UID: \"426afdc6-cfdd-47c9-9ba9-6da128c31913\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e33e59be-fmxqt"
Nov 03 20:36:36.984052 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:36.984030    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/426afdc6-cfdd-47c9-9ba9-6da128c31913-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e33e59be-fmxqt\" (UID: \"426afdc6-cfdd-47c9-9ba9-6da128c31913\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e33e59be-fmxqt"
Nov 03 20:36:37.002389 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:37.002352    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwsxs\" (UniqueName: \"kubernetes.io/projected/426afdc6-cfdd-47c9-9ba9-6da128c31913-kube-api-access-kwsxs\") pod \"maas-api-revoke-sub-e33e59be-fmxqt\" (UID: \"426afdc6-cfdd-47c9-9ba9-6da128c31913\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e33e59be-fmxqt"
Nov 03 20:36:37.116729 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:37.116636    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e33e59be-fmxqt"
Nov 03 20:36:37.420291 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:37.420261    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e33e59be-fmxqt"]
Nov 03 20:36:37.421542 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:36:37.421503    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod426afdc6_cfdd_47c9_9ba9_6da128c31913.slice/crio-ffcf53876004c76e1c9d422df81727206bdbf97ed1801dc51156875766c253b2 WatchSource:0}: Error finding container ffcf53876004c76e1c9d422df81727206bdbf97ed1801dc51156875766c253b2: Status 404 returned error can't find the container with id ffcf53876004c76e1c9d422df81727206bdbf97ed1801dc51156875766c253b2
Nov 03 20:36:37.471510 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:37.471476    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e33e59be-fmxqt" event={"ID":"426afdc6-cfdd-47c9-9ba9-6da128c31913","Type":"ContainerStarted","Data":"ffcf53876004c76e1c9d422df81727206bdbf97ed1801dc51156875766c253b2"}
Nov 03 20:36:37.966229 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:37.966193    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca0be3e9-2414-4d9c-ae41-fcbe4dddb52a" path="/var/lib/kubelet/pods/ca0be3e9-2414-4d9c-ae41-fcbe4dddb52a/volumes"
Nov 03 20:36:38.430825 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:38.430768    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-60efdf-4gbsk"]
Nov 03 20:36:38.443298 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:38.442649    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-60efdf-4gbsk"]
Nov 03 20:36:38.479611 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:38.479577    5570 generic.go:356] "Generic (PLEG): container finished" podID="426afdc6-cfdd-47c9-9ba9-6da128c31913" containerID="b88ff525475e91c8387174eb572c2b939820a5b2c8fac17dc14258e4bbca7c25" exitCode=0
Nov 03 20:36:38.480099 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:38.479638    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e33e59be-fmxqt" event={"ID":"426afdc6-cfdd-47c9-9ba9-6da128c31913","Type":"ContainerDied","Data":"b88ff525475e91c8387174eb572c2b939820a5b2c8fac17dc14258e4bbca7c25"}
Nov 03 20:36:39.685075 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:39.685047    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e33e59be-fmxqt"
Nov 03 20:36:39.747184 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:39.747087    5570 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-1d159a/test-model-1d159a-kserve-5766d9c789-lqs6f" podUID="6a03d044-d7ba-4c94-b408-a7cdeaea9273" containerName="main" containerID="cri-o://941895b3742e544fedb84438fe2779fa5a8a1143cc792b26b0ad7d80ef6b1915" gracePeriod=45
Nov 03 20:36:39.764111 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:39.764075    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/426afdc6-cfdd-47c9-9ba9-6da128c31913-kube-api-access-kwsxs\" (UniqueName: \"kubernetes.io/projected/426afdc6-cfdd-47c9-9ba9-6da128c31913-kube-api-access-kwsxs\") pod \"426afdc6-cfdd-47c9-9ba9-6da128c31913\" (UID: \"426afdc6-cfdd-47c9-9ba9-6da128c31913\") "
Nov 03 20:36:39.764300 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:39.764143    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/426afdc6-cfdd-47c9-9ba9-6da128c31913-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/426afdc6-cfdd-47c9-9ba9-6da128c31913-maas-api-openshift-ca\") pod \"426afdc6-cfdd-47c9-9ba9-6da128c31913\" (UID: \"426afdc6-cfdd-47c9-9ba9-6da128c31913\") "
Nov 03 20:36:39.764300 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:39.764181    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/426afdc6-cfdd-47c9-9ba9-6da128c31913-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/426afdc6-cfdd-47c9-9ba9-6da128c31913-maas-api-xks-ca\") pod \"426afdc6-cfdd-47c9-9ba9-6da128c31913\" (UID: \"426afdc6-cfdd-47c9-9ba9-6da128c31913\") "
Nov 03 20:36:39.764573 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:39.764545    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/426afdc6-cfdd-47c9-9ba9-6da128c31913-maas-api-openshift-ca" pod "426afdc6-cfdd-47c9-9ba9-6da128c31913" (UID: "426afdc6-cfdd-47c9-9ba9-6da128c31913"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:36:39.764734 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:39.764713    5570 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/426afdc6-cfdd-47c9-9ba9-6da128c31913-maas-api-openshift-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:36:39.767147 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:39.766448    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/426afdc6-cfdd-47c9-9ba9-6da128c31913-maas-api-xks-ca" pod "426afdc6-cfdd-47c9-9ba9-6da128c31913" (UID: "426afdc6-cfdd-47c9-9ba9-6da128c31913"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:36:39.767147 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:39.766620    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/426afdc6-cfdd-47c9-9ba9-6da128c31913-kube-api-access-kwsxs" pod "426afdc6-cfdd-47c9-9ba9-6da128c31913" (UID: "426afdc6-cfdd-47c9-9ba9-6da128c31913"). InnerVolumeSpecName "kube-api-access-kwsxs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:36:39.865606 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:39.865580    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwsxs\" (UniqueName: \"kubernetes.io/projected/426afdc6-cfdd-47c9-9ba9-6da128c31913-kube-api-access-kwsxs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:36:39.865671 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:39.865617    5570 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/426afdc6-cfdd-47c9-9ba9-6da128c31913-maas-api-xks-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:36:39.966334 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:39.966299    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="933a606e-1a38-47dc-9781-3a5730b0c0c3" path="/var/lib/kubelet/pods/933a606e-1a38-47dc-9781-3a5730b0c0c3/volumes"
Nov 03 20:36:40.024514 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:40.024485    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-1d159a/test-model-1d159a-kserve-5766d9c789-lqs6f"
Nov 03 20:36:40.170416 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:40.170378    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a03d044-d7ba-4c94-b408-a7cdeaea9273-tls-certs\" (UniqueName: \"kubernetes.io/secret/6a03d044-d7ba-4c94-b408-a7cdeaea9273-tls-certs\") pod \"6a03d044-d7ba-4c94-b408-a7cdeaea9273\" (UID: \"6a03d044-d7ba-4c94-b408-a7cdeaea9273\") "
Nov 03 20:36:40.170416 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:40.170423    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a03d044-d7ba-4c94-b408-a7cdeaea9273-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a03d044-d7ba-4c94-b408-a7cdeaea9273-tmp-dir\") pod \"6a03d044-d7ba-4c94-b408-a7cdeaea9273\" (UID: \"6a03d044-d7ba-4c94-b408-a7cdeaea9273\") "
Nov 03 20:36:40.170683 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:40.170452    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a03d044-d7ba-4c94-b408-a7cdeaea9273-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a03d044-d7ba-4c94-b408-a7cdeaea9273-model-cache\") pod \"6a03d044-d7ba-4c94-b408-a7cdeaea9273\" (UID: \"6a03d044-d7ba-4c94-b408-a7cdeaea9273\") "
Nov 03 20:36:40.170683 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:40.170533    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a03d044-d7ba-4c94-b408-a7cdeaea9273-kube-api-access-fgrpl\" (UniqueName: \"kubernetes.io/projected/6a03d044-d7ba-4c94-b408-a7cdeaea9273-kube-api-access-fgrpl\") pod \"6a03d044-d7ba-4c94-b408-a7cdeaea9273\" (UID: \"6a03d044-d7ba-4c94-b408-a7cdeaea9273\") "
Nov 03 20:36:40.170683 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:40.170629    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a03d044-d7ba-4c94-b408-a7cdeaea9273-home\" (UniqueName: \"kubernetes.io/empty-dir/6a03d044-d7ba-4c94-b408-a7cdeaea9273-home\") pod \"6a03d044-d7ba-4c94-b408-a7cdeaea9273\" (UID: \"6a03d044-d7ba-4c94-b408-a7cdeaea9273\") "
Nov 03 20:36:40.170876 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:40.170730    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a03d044-d7ba-4c94-b408-a7cdeaea9273-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a03d044-d7ba-4c94-b408-a7cdeaea9273-dshm\") pod \"6a03d044-d7ba-4c94-b408-a7cdeaea9273\" (UID: \"6a03d044-d7ba-4c94-b408-a7cdeaea9273\") "
Nov 03 20:36:40.170876 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:40.170735    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a03d044-d7ba-4c94-b408-a7cdeaea9273-tmp-dir" pod "6a03d044-d7ba-4c94-b408-a7cdeaea9273" (UID: "6a03d044-d7ba-4c94-b408-a7cdeaea9273"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:36:40.170876 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:40.170858    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a03d044-d7ba-4c94-b408-a7cdeaea9273-model-cache" pod "6a03d044-d7ba-4c94-b408-a7cdeaea9273" (UID: "6a03d044-d7ba-4c94-b408-a7cdeaea9273"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:36:40.171025 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:40.170999    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a03d044-d7ba-4c94-b408-a7cdeaea9273-home" pod "6a03d044-d7ba-4c94-b408-a7cdeaea9273" (UID: "6a03d044-d7ba-4c94-b408-a7cdeaea9273"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:36:40.171278 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:40.171261    5570 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a03d044-d7ba-4c94-b408-a7cdeaea9273-home\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:36:40.171356 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:40.171278    5570 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a03d044-d7ba-4c94-b408-a7cdeaea9273-tmp-dir\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:36:40.171356 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:40.171291    5570 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a03d044-d7ba-4c94-b408-a7cdeaea9273-model-cache\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:36:40.172889 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:40.172864    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a03d044-d7ba-4c94-b408-a7cdeaea9273-dshm" pod "6a03d044-d7ba-4c94-b408-a7cdeaea9273" (UID: "6a03d044-d7ba-4c94-b408-a7cdeaea9273"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:36:40.172889 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:40.172875    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a03d044-d7ba-4c94-b408-a7cdeaea9273-tls-certs" pod "6a03d044-d7ba-4c94-b408-a7cdeaea9273" (UID: "6a03d044-d7ba-4c94-b408-a7cdeaea9273"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:36:40.173032 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:40.172913    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a03d044-d7ba-4c94-b408-a7cdeaea9273-kube-api-access-fgrpl" pod "6a03d044-d7ba-4c94-b408-a7cdeaea9273" (UID: "6a03d044-d7ba-4c94-b408-a7cdeaea9273"). InnerVolumeSpecName "kube-api-access-fgrpl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:36:40.272919 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:40.272819    5570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a03d044-d7ba-4c94-b408-a7cdeaea9273-tls-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:36:40.272919 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:40.272863    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgrpl\" (UniqueName: \"kubernetes.io/projected/6a03d044-d7ba-4c94-b408-a7cdeaea9273-kube-api-access-fgrpl\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:36:40.272919 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:40.272873    5570 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a03d044-d7ba-4c94-b408-a7cdeaea9273-dshm\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:36:40.406363 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:40.406312    5570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-60efdf-7cb6d9fbc8-fwx62" podUID="c03d73cd-af2e-4af4-88cb-bca5f26dda01" containerName="payload-processing" containerID="cri-o://61c5e6d2f06cee85a4001247ff9b4756cdb25498d04e3d9a402ac4eaf6e6b593" gracePeriod=25
Nov 03 20:36:40.438563 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:40.438508    5570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-60efdf-7bfdc67bd7-j6ctn" podUID="effc03b7-0b9e-4239-baa0-5c6cd120f479" containerName="payload-pre-processing" containerID="cri-o://2ec3f67934e86a94a7ef166e62c2b6e9cc6fcf63f2ac547d75064af34674793a" gracePeriod=25
Nov 03 20:36:40.492960 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:40.492921    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e33e59be-fmxqt"
Nov 03 20:36:40.493162 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:40.492932    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e33e59be-fmxqt" event={"ID":"426afdc6-cfdd-47c9-9ba9-6da128c31913","Type":"ContainerDied","Data":"ffcf53876004c76e1c9d422df81727206bdbf97ed1801dc51156875766c253b2"}
Nov 03 20:36:40.493162 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:40.493060    5570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ffcf53876004c76e1c9d422df81727206bdbf97ed1801dc51156875766c253b2"
Nov 03 20:36:40.494390 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:40.494366    5570 generic.go:356] "Generic (PLEG): container finished" podID="6a03d044-d7ba-4c94-b408-a7cdeaea9273" containerID="941895b3742e544fedb84438fe2779fa5a8a1143cc792b26b0ad7d80ef6b1915" exitCode=0
Nov 03 20:36:40.494538 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:40.494405    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-1d159a/test-model-1d159a-kserve-5766d9c789-lqs6f"
Nov 03 20:36:40.494538 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:40.494451    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-1d159a/test-model-1d159a-kserve-5766d9c789-lqs6f" event={"ID":"6a03d044-d7ba-4c94-b408-a7cdeaea9273","Type":"ContainerDied","Data":"941895b3742e544fedb84438fe2779fa5a8a1143cc792b26b0ad7d80ef6b1915"}
Nov 03 20:36:40.494538 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:40.494496    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-1d159a/test-model-1d159a-kserve-5766d9c789-lqs6f" event={"ID":"6a03d044-d7ba-4c94-b408-a7cdeaea9273","Type":"ContainerDied","Data":"da909e275ff2c6c652d883b8b3bf910af795054cb560fe5cede3571cf6a89943"}
Nov 03 20:36:40.494538 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:40.494520    5570 scope.go:122] "RemoveContainer" containerID="941895b3742e544fedb84438fe2779fa5a8a1143cc792b26b0ad7d80ef6b1915"
Nov 03 20:36:40.518503 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:40.518479    5570 scope.go:122] "RemoveContainer" containerID="941895b3742e544fedb84438fe2779fa5a8a1143cc792b26b0ad7d80ef6b1915"
Nov 03 20:36:40.518873 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:36:40.518850    5570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"941895b3742e544fedb84438fe2779fa5a8a1143cc792b26b0ad7d80ef6b1915\": container with ID starting with 941895b3742e544fedb84438fe2779fa5a8a1143cc792b26b0ad7d80ef6b1915 not found: ID does not exist" containerID="941895b3742e544fedb84438fe2779fa5a8a1143cc792b26b0ad7d80ef6b1915"
Nov 03 20:36:40.518914 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:40.518886    5570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"941895b3742e544fedb84438fe2779fa5a8a1143cc792b26b0ad7d80ef6b1915"} err="failed to get container status \"941895b3742e544fedb84438fe2779fa5a8a1143cc792b26b0ad7d80ef6b1915\": rpc error: code = NotFound desc = could not find container \"941895b3742e544fedb84438fe2779fa5a8a1143cc792b26b0ad7d80ef6b1915\": container with ID starting with 941895b3742e544fedb84438fe2779fa5a8a1143cc792b26b0ad7d80ef6b1915 not found: ID does not exist"
Nov 03 20:36:40.554734 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:40.554641    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-1d159a/test-model-1d159a-kserve-5766d9c789-lqs6f"]
Nov 03 20:36:40.566218 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:40.566183    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-1d159a/test-model-1d159a-kserve-5766d9c789-lqs6f"]
Nov 03 20:36:40.691517 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:40.691493    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-60efdf-7cb6d9fbc8-fwx62"
Nov 03 20:36:40.779108 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:40.779068    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c03d73cd-af2e-4af4-88cb-bca5f26dda01-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c03d73cd-af2e-4af4-88cb-bca5f26dda01-plugins-config-volume\") pod \"c03d73cd-af2e-4af4-88cb-bca5f26dda01\" (UID: \"c03d73cd-af2e-4af4-88cb-bca5f26dda01\") "
Nov 03 20:36:40.779306 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:40.779221    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c03d73cd-af2e-4af4-88cb-bca5f26dda01-kube-api-access-dmnqz\" (UniqueName: \"kubernetes.io/projected/c03d73cd-af2e-4af4-88cb-bca5f26dda01-kube-api-access-dmnqz\") pod \"c03d73cd-af2e-4af4-88cb-bca5f26dda01\" (UID: \"c03d73cd-af2e-4af4-88cb-bca5f26dda01\") "
Nov 03 20:36:40.779453 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:40.779428    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c03d73cd-af2e-4af4-88cb-bca5f26dda01-plugins-config-volume" pod "c03d73cd-af2e-4af4-88cb-bca5f26dda01" (UID: "c03d73cd-af2e-4af4-88cb-bca5f26dda01"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:36:40.779863 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:40.779834    5570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c03d73cd-af2e-4af4-88cb-bca5f26dda01-plugins-config-volume\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:36:40.781691 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:40.781671    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c03d73cd-af2e-4af4-88cb-bca5f26dda01-kube-api-access-dmnqz" pod "c03d73cd-af2e-4af4-88cb-bca5f26dda01" (UID: "c03d73cd-af2e-4af4-88cb-bca5f26dda01"). InnerVolumeSpecName "kube-api-access-dmnqz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:36:40.881483 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:40.881388    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmnqz\" (UniqueName: \"kubernetes.io/projected/c03d73cd-af2e-4af4-88cb-bca5f26dda01-kube-api-access-dmnqz\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:36:41.502278 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:41.502244    5570 generic.go:356] "Generic (PLEG): container finished" podID="c03d73cd-af2e-4af4-88cb-bca5f26dda01" containerID="61c5e6d2f06cee85a4001247ff9b4756cdb25498d04e3d9a402ac4eaf6e6b593" exitCode=0
Nov 03 20:36:41.502514 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:41.502297    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-60efdf-7cb6d9fbc8-fwx62"
Nov 03 20:36:41.502514 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:41.502333    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-60efdf-7cb6d9fbc8-fwx62" event={"ID":"c03d73cd-af2e-4af4-88cb-bca5f26dda01","Type":"ContainerDied","Data":"61c5e6d2f06cee85a4001247ff9b4756cdb25498d04e3d9a402ac4eaf6e6b593"}
Nov 03 20:36:41.502514 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:41.502389    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-60efdf-7cb6d9fbc8-fwx62" event={"ID":"c03d73cd-af2e-4af4-88cb-bca5f26dda01","Type":"ContainerDied","Data":"4e385de3d756ba5db012c4f0bf9e339ecba196cb3c6a71088aeb32c2cc89c67b"}
Nov 03 20:36:41.502514 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:41.502420    5570 scope.go:122] "RemoveContainer" containerID="61c5e6d2f06cee85a4001247ff9b4756cdb25498d04e3d9a402ac4eaf6e6b593"
Nov 03 20:36:41.513056 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:41.513035    5570 scope.go:122] "RemoveContainer" containerID="61c5e6d2f06cee85a4001247ff9b4756cdb25498d04e3d9a402ac4eaf6e6b593"
Nov 03 20:36:41.513407 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:36:41.513387    5570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61c5e6d2f06cee85a4001247ff9b4756cdb25498d04e3d9a402ac4eaf6e6b593\": container with ID starting with 61c5e6d2f06cee85a4001247ff9b4756cdb25498d04e3d9a402ac4eaf6e6b593 not found: ID does not exist" containerID="61c5e6d2f06cee85a4001247ff9b4756cdb25498d04e3d9a402ac4eaf6e6b593"
Nov 03 20:36:41.513480 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:41.513415    5570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61c5e6d2f06cee85a4001247ff9b4756cdb25498d04e3d9a402ac4eaf6e6b593"} err="failed to get container status \"61c5e6d2f06cee85a4001247ff9b4756cdb25498d04e3d9a402ac4eaf6e6b593\": rpc error: code = NotFound desc = could not find container \"61c5e6d2f06cee85a4001247ff9b4756cdb25498d04e3d9a402ac4eaf6e6b593\": container with ID starting with 61c5e6d2f06cee85a4001247ff9b4756cdb25498d04e3d9a402ac4eaf6e6b593 not found: ID does not exist"
Nov 03 20:36:41.557939 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:41.557899    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-60efdf-7cb6d9fbc8-fwx62"]
Nov 03 20:36:41.570629 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:41.570600    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-60efdf-7cb6d9fbc8-fwx62"]
Nov 03 20:36:41.965381 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:41.965347    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a03d044-d7ba-4c94-b408-a7cdeaea9273" path="/var/lib/kubelet/pods/6a03d044-d7ba-4c94-b408-a7cdeaea9273/volumes"
Nov 03 20:36:41.965847 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:41.965831    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c03d73cd-af2e-4af4-88cb-bca5f26dda01" path="/var/lib/kubelet/pods/c03d73cd-af2e-4af4-88cb-bca5f26dda01/volumes"
Nov 03 20:36:43.394810 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:43.394756    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g69f8"]
Nov 03 20:36:44.314978 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:44.314940    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-60efdf-7bfdc67bd7-j6ctn" podUID="effc03b7-0b9e-4239-baa0-5c6cd120f479" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 20:36:45.969619 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:45.969575    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-60efdf-openshift-default-5dfc484849-k6445"]
Nov 03 20:36:45.970152 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:45.969866    5570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-60efdf-openshift-default-5dfc484849-k6445" podUID="c6fb7442-714a-4e4a-a6cf-19326b50a929" containerName="istio-proxy" containerID="cri-o://f7d8105e38072ca5fdd84f547f0b0fed18c8772f8ce6f4e0cc52c9f7cb80e806" gracePeriod=30
Nov 03 20:36:46.532805 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:46.532754    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1d159a-49svt"]
Nov 03 20:36:46.533487 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:46.533466    5570 cpu_manager.go:414] "containerMap: removing container" podUID="6a03d044-d7ba-4c94-b408-a7cdeaea9273" containerName="main"
Nov 03 20:36:46.533620 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:46.533488    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a03d044-d7ba-4c94-b408-a7cdeaea9273" containerName="main"
Nov 03 20:36:46.533620 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:46.533511    5570 cpu_manager.go:414] "containerMap: removing container" podUID="426afdc6-cfdd-47c9-9ba9-6da128c31913" containerName="revoke-keys"
Nov 03 20:36:46.533620 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:46.533519    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="426afdc6-cfdd-47c9-9ba9-6da128c31913" containerName="revoke-keys"
Nov 03 20:36:46.533620 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:46.533539    5570 cpu_manager.go:414] "containerMap: removing container" podUID="c03d73cd-af2e-4af4-88cb-bca5f26dda01" containerName="payload-processing"
Nov 03 20:36:46.533620 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:46.533547    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c03d73cd-af2e-4af4-88cb-bca5f26dda01" containerName="payload-processing"
Nov 03 20:36:46.533901 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:46.533706    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a03d044-d7ba-4c94-b408-a7cdeaea9273" containerName="main"
Nov 03 20:36:46.533901 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:46.533722    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="426afdc6-cfdd-47c9-9ba9-6da128c31913" containerName="revoke-keys"
Nov 03 20:36:46.533901 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:46.533733    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="c03d73cd-af2e-4af4-88cb-bca5f26dda01" containerName="payload-processing"
Nov 03 20:36:46.538647 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:46.538622    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1d159a-49svt"
Nov 03 20:36:46.553434 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:46.553397    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1d159a-49svt"]
Nov 03 20:36:46.563085 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:46.563045    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:36:46.563288 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:46.563268    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l7s5w\""
Nov 03 20:36:46.697741 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:46.697707    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5bbecbfa-d55a-4ec2-9db5-77e5d9675955-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-1d159a-49svt\" (UID: \"5bbecbfa-d55a-4ec2-9db5-77e5d9675955\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1d159a-49svt"
Nov 03 20:36:46.697970 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:46.697748    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5bbecbfa-d55a-4ec2-9db5-77e5d9675955-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-1d159a-49svt\" (UID: \"5bbecbfa-d55a-4ec2-9db5-77e5d9675955\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1d159a-49svt"
Nov 03 20:36:46.697970 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:46.697909    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2lfw\" (UniqueName: \"kubernetes.io/projected/5bbecbfa-d55a-4ec2-9db5-77e5d9675955-kube-api-access-p2lfw\") pod \"maas-api-revoke-keys-e2e-inf-b-1d159a-49svt\" (UID: \"5bbecbfa-d55a-4ec2-9db5-77e5d9675955\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1d159a-49svt"
Nov 03 20:36:46.799331 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:46.799222    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5bbecbfa-d55a-4ec2-9db5-77e5d9675955-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-1d159a-49svt\" (UID: \"5bbecbfa-d55a-4ec2-9db5-77e5d9675955\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1d159a-49svt"
Nov 03 20:36:46.799517 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:46.799337    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2lfw\" (UniqueName: \"kubernetes.io/projected/5bbecbfa-d55a-4ec2-9db5-77e5d9675955-kube-api-access-p2lfw\") pod \"maas-api-revoke-keys-e2e-inf-b-1d159a-49svt\" (UID: \"5bbecbfa-d55a-4ec2-9db5-77e5d9675955\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1d159a-49svt"
Nov 03 20:36:46.799579 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:46.799560    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5bbecbfa-d55a-4ec2-9db5-77e5d9675955-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-1d159a-49svt\" (UID: \"5bbecbfa-d55a-4ec2-9db5-77e5d9675955\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1d159a-49svt"
Nov 03 20:36:46.800139 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:46.800111    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5bbecbfa-d55a-4ec2-9db5-77e5d9675955-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-1d159a-49svt\" (UID: \"5bbecbfa-d55a-4ec2-9db5-77e5d9675955\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1d159a-49svt"
Nov 03 20:36:46.802095 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:46.802070    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5bbecbfa-d55a-4ec2-9db5-77e5d9675955-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-1d159a-49svt\" (UID: \"5bbecbfa-d55a-4ec2-9db5-77e5d9675955\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1d159a-49svt"
Nov 03 20:36:46.819909 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:46.819852    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2lfw\" (UniqueName: \"kubernetes.io/projected/5bbecbfa-d55a-4ec2-9db5-77e5d9675955-kube-api-access-p2lfw\") pod \"maas-api-revoke-keys-e2e-inf-b-1d159a-49svt\" (UID: \"5bbecbfa-d55a-4ec2-9db5-77e5d9675955\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1d159a-49svt"
Nov 03 20:36:46.855278 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:46.855234    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1d159a-49svt"
Nov 03 20:36:47.153499 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:47.153458    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1d159a-49svt"]
Nov 03 20:36:47.153892 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:36:47.153721    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bbecbfa_d55a_4ec2_9db5_77e5d9675955.slice/crio-754e04d617ff8a27f851bca4d0cf602991e321ac19a939e5407181ad259eaef6 WatchSource:0}: Error finding container 754e04d617ff8a27f851bca4d0cf602991e321ac19a939e5407181ad259eaef6: Status 404 returned error can't find the container with id 754e04d617ff8a27f851bca4d0cf602991e321ac19a939e5407181ad259eaef6
Nov 03 20:36:47.547428 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:47.547387    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1d159a-49svt" event={"ID":"5bbecbfa-d55a-4ec2-9db5-77e5d9675955","Type":"ContainerStarted","Data":"c208803397eb8d4054c1b188174da01748b3d7cad8a5135b673477bfe42743f9"}
Nov 03 20:36:47.547428 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:47.547456    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1d159a-49svt" event={"ID":"5bbecbfa-d55a-4ec2-9db5-77e5d9675955","Type":"ContainerStarted","Data":"754e04d617ff8a27f851bca4d0cf602991e321ac19a939e5407181ad259eaef6"}
Nov 03 20:36:47.589765 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:47.589712    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1d159a-49svt" podStartSLOduration=1.5896975150000001 podStartE2EDuration="1.589697515s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:36:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:36:47.58888202 +0000 UTC m=+1941.741543910" watchObservedRunningTime="2026-11-03 20:36:47.589697515 +0000 UTC m=+1941.742359400"
Nov 03 20:36:47.663861 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:47.663807    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-4b95f3/test-model-4b95f3-kserve-54555ccc49-ncvhw"]
Nov 03 20:36:48.555577 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:48.555485    5570 generic.go:356] "Generic (PLEG): container finished" podID="5bbecbfa-d55a-4ec2-9db5-77e5d9675955" containerID="c208803397eb8d4054c1b188174da01748b3d7cad8a5135b673477bfe42743f9" exitCode=0
Nov 03 20:36:48.555577 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:48.555554    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1d159a-49svt" event={"ID":"5bbecbfa-d55a-4ec2-9db5-77e5d9675955","Type":"ContainerDied","Data":"c208803397eb8d4054c1b188174da01748b3d7cad8a5135b673477bfe42743f9"}
Nov 03 20:36:49.730228 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:49.730197    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1d159a-49svt"
Nov 03 20:36:49.772684 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:49.772649    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5bbecbfa-d55a-4ec2-9db5-77e5d9675955-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5bbecbfa-d55a-4ec2-9db5-77e5d9675955-maas-api-openshift-ca\") pod \"5bbecbfa-d55a-4ec2-9db5-77e5d9675955\" (UID: \"5bbecbfa-d55a-4ec2-9db5-77e5d9675955\") "
Nov 03 20:36:49.772877 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:49.772833    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5bbecbfa-d55a-4ec2-9db5-77e5d9675955-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5bbecbfa-d55a-4ec2-9db5-77e5d9675955-maas-api-xks-ca\") pod \"5bbecbfa-d55a-4ec2-9db5-77e5d9675955\" (UID: \"5bbecbfa-d55a-4ec2-9db5-77e5d9675955\") "
Nov 03 20:36:49.773013 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:49.772991    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5bbecbfa-d55a-4ec2-9db5-77e5d9675955-kube-api-access-p2lfw\" (UniqueName: \"kubernetes.io/projected/5bbecbfa-d55a-4ec2-9db5-77e5d9675955-kube-api-access-p2lfw\") pod \"5bbecbfa-d55a-4ec2-9db5-77e5d9675955\" (UID: \"5bbecbfa-d55a-4ec2-9db5-77e5d9675955\") "
Nov 03 20:36:49.773642 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:49.773615    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bbecbfa-d55a-4ec2-9db5-77e5d9675955-maas-api-openshift-ca" pod "5bbecbfa-d55a-4ec2-9db5-77e5d9675955" (UID: "5bbecbfa-d55a-4ec2-9db5-77e5d9675955"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:36:49.773948 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:49.773929    5570 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5bbecbfa-d55a-4ec2-9db5-77e5d9675955-maas-api-openshift-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:36:49.775384 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:49.775362    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5bbecbfa-d55a-4ec2-9db5-77e5d9675955-maas-api-xks-ca" pod "5bbecbfa-d55a-4ec2-9db5-77e5d9675955" (UID: "5bbecbfa-d55a-4ec2-9db5-77e5d9675955"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:36:49.775640 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:49.775615    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bbecbfa-d55a-4ec2-9db5-77e5d9675955-kube-api-access-p2lfw" pod "5bbecbfa-d55a-4ec2-9db5-77e5d9675955" (UID: "5bbecbfa-d55a-4ec2-9db5-77e5d9675955"). InnerVolumeSpecName "kube-api-access-p2lfw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:36:49.875349 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:49.875264    5570 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5bbecbfa-d55a-4ec2-9db5-77e5d9675955-maas-api-xks-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:36:49.875349 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:49.875293    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2lfw\" (UniqueName: \"kubernetes.io/projected/5bbecbfa-d55a-4ec2-9db5-77e5d9675955-kube-api-access-p2lfw\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:36:50.570750 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:50.570715    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1d159a-49svt" event={"ID":"5bbecbfa-d55a-4ec2-9db5-77e5d9675955","Type":"ContainerDied","Data":"754e04d617ff8a27f851bca4d0cf602991e321ac19a939e5407181ad259eaef6"}
Nov 03 20:36:50.570945 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:50.570757    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1d159a-49svt"
Nov 03 20:36:50.570945 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:50.570767    5570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="754e04d617ff8a27f851bca4d0cf602991e321ac19a939e5407181ad259eaef6"
Nov 03 20:36:51.254289 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:51.254257    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-60efdf-openshift-default-5dfc484849-k6445"
Nov 03 20:36:51.279711 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:51.279668    5570 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-a9ba83/test-model-a9ba83-kserve-5f6985cb6c-d6pfg" podUID="b2d9f055-1380-4eb4-84ee-48dc75855852" containerName="main" containerID="cri-o://69369f1cd566da840ae692406efc55dc6741ddcc3e5ea8288eadc64406d7033c" gracePeriod=45
Nov 03 20:36:51.292929 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:51.292891    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c6fb7442-714a-4e4a-a6cf-19326b50a929-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6fb7442-714a-4e4a-a6cf-19326b50a929-istio-podinfo\") pod \"c6fb7442-714a-4e4a-a6cf-19326b50a929\" (UID: \"c6fb7442-714a-4e4a-a6cf-19326b50a929\") "
Nov 03 20:36:51.293089 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:51.292981    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6fb7442-714a-4e4a-a6cf-19326b50a929-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6fb7442-714a-4e4a-a6cf-19326b50a929-istiod-ca-cert\") pod \"c6fb7442-714a-4e4a-a6cf-19326b50a929\" (UID: \"c6fb7442-714a-4e4a-a6cf-19326b50a929\") "
Nov 03 20:36:51.293089 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:51.293042    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6fb7442-714a-4e4a-a6cf-19326b50a929-kube-api-access-lbmj4\" (UniqueName: \"kubernetes.io/projected/c6fb7442-714a-4e4a-a6cf-19326b50a929-kube-api-access-lbmj4\") pod \"c6fb7442-714a-4e4a-a6cf-19326b50a929\" (UID: \"c6fb7442-714a-4e4a-a6cf-19326b50a929\") "
Nov 03 20:36:51.293089 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:51.293077    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6fb7442-714a-4e4a-a6cf-19326b50a929-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6fb7442-714a-4e4a-a6cf-19326b50a929-credential-socket\") pod \"c6fb7442-714a-4e4a-a6cf-19326b50a929\" (UID: \"c6fb7442-714a-4e4a-a6cf-19326b50a929\") "
Nov 03 20:36:51.293260 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:51.293129    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6fb7442-714a-4e4a-a6cf-19326b50a929-istio-token\" (UniqueName: \"kubernetes.io/projected/c6fb7442-714a-4e4a-a6cf-19326b50a929-istio-token\") pod \"c6fb7442-714a-4e4a-a6cf-19326b50a929\" (UID: \"c6fb7442-714a-4e4a-a6cf-19326b50a929\") "
Nov 03 20:36:51.293260 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:51.293206    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6fb7442-714a-4e4a-a6cf-19326b50a929-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6fb7442-714a-4e4a-a6cf-19326b50a929-workload-certs\") pod \"c6fb7442-714a-4e4a-a6cf-19326b50a929\" (UID: \"c6fb7442-714a-4e4a-a6cf-19326b50a929\") "
Nov 03 20:36:51.293390 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:51.293294    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6fb7442-714a-4e4a-a6cf-19326b50a929-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6fb7442-714a-4e4a-a6cf-19326b50a929-istio-envoy\") pod \"c6fb7442-714a-4e4a-a6cf-19326b50a929\" (UID: \"c6fb7442-714a-4e4a-a6cf-19326b50a929\") "
Nov 03 20:36:51.293452 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:51.293410    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6fb7442-714a-4e4a-a6cf-19326b50a929-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6fb7442-714a-4e4a-a6cf-19326b50a929-workload-socket\") pod \"c6fb7442-714a-4e4a-a6cf-19326b50a929\" (UID: \"c6fb7442-714a-4e4a-a6cf-19326b50a929\") "
Nov 03 20:36:51.293512 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:51.293451    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6fb7442-714a-4e4a-a6cf-19326b50a929-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6fb7442-714a-4e4a-a6cf-19326b50a929-istio-data\") pod \"c6fb7442-714a-4e4a-a6cf-19326b50a929\" (UID: \"c6fb7442-714a-4e4a-a6cf-19326b50a929\") "
Nov 03 20:36:51.293573 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:51.293528    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6fb7442-714a-4e4a-a6cf-19326b50a929-workload-certs" pod "c6fb7442-714a-4e4a-a6cf-19326b50a929" (UID: "c6fb7442-714a-4e4a-a6cf-19326b50a929"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:36:51.293573 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:51.293542    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6fb7442-714a-4e4a-a6cf-19326b50a929-credential-socket" pod "c6fb7442-714a-4e4a-a6cf-19326b50a929" (UID: "c6fb7442-714a-4e4a-a6cf-19326b50a929"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:36:51.293761 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:51.293743    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6fb7442-714a-4e4a-a6cf-19326b50a929-workload-socket" pod "c6fb7442-714a-4e4a-a6cf-19326b50a929" (UID: "c6fb7442-714a-4e4a-a6cf-19326b50a929"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:36:51.294056 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:51.294033    5570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6fb7442-714a-4e4a-a6cf-19326b50a929-credential-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:36:51.294127 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:51.294062    5570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6fb7442-714a-4e4a-a6cf-19326b50a929-workload-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:36:51.294127 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:51.294076    5570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6fb7442-714a-4e4a-a6cf-19326b50a929-workload-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:36:51.294127 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:51.294045    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6fb7442-714a-4e4a-a6cf-19326b50a929-istio-data" pod "c6fb7442-714a-4e4a-a6cf-19326b50a929" (UID: "c6fb7442-714a-4e4a-a6cf-19326b50a929"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:36:51.294458 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:51.294424    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6fb7442-714a-4e4a-a6cf-19326b50a929-istiod-ca-cert" pod "c6fb7442-714a-4e4a-a6cf-19326b50a929" (UID: "c6fb7442-714a-4e4a-a6cf-19326b50a929"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:36:51.295646 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:51.295619    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c6fb7442-714a-4e4a-a6cf-19326b50a929-istio-podinfo" pod "c6fb7442-714a-4e4a-a6cf-19326b50a929" (UID: "c6fb7442-714a-4e4a-a6cf-19326b50a929"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:36:51.296370 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:51.296347    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6fb7442-714a-4e4a-a6cf-19326b50a929-kube-api-access-lbmj4" pod "c6fb7442-714a-4e4a-a6cf-19326b50a929" (UID: "c6fb7442-714a-4e4a-a6cf-19326b50a929"). InnerVolumeSpecName "kube-api-access-lbmj4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:36:51.296461 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:51.296364    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6fb7442-714a-4e4a-a6cf-19326b50a929-istio-envoy" pod "c6fb7442-714a-4e4a-a6cf-19326b50a929" (UID: "c6fb7442-714a-4e4a-a6cf-19326b50a929"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:36:51.296461 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:51.296436    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6fb7442-714a-4e4a-a6cf-19326b50a929-istio-token" pod "c6fb7442-714a-4e4a-a6cf-19326b50a929" (UID: "c6fb7442-714a-4e4a-a6cf-19326b50a929"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:36:51.395137 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:51.395102    5570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6fb7442-714a-4e4a-a6cf-19326b50a929-istio-envoy\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:36:51.395137 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:51.395135    5570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6fb7442-714a-4e4a-a6cf-19326b50a929-istio-data\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:36:51.395339 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:51.395149    5570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6fb7442-714a-4e4a-a6cf-19326b50a929-istio-podinfo\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:36:51.395339 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:51.395161    5570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6fb7442-714a-4e4a-a6cf-19326b50a929-istiod-ca-cert\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:36:51.395339 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:51.395174    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbmj4\" (UniqueName: \"kubernetes.io/projected/c6fb7442-714a-4e4a-a6cf-19326b50a929-kube-api-access-lbmj4\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:36:51.395339 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:51.395187    5570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6fb7442-714a-4e4a-a6cf-19326b50a929-istio-token\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:36:51.540913 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:51.540886    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a9ba83/test-model-a9ba83-kserve-5f6985cb6c-d6pfg"
Nov 03 20:36:51.578649 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:51.578612    5570 generic.go:356] "Generic (PLEG): container finished" podID="b2d9f055-1380-4eb4-84ee-48dc75855852" containerID="69369f1cd566da840ae692406efc55dc6741ddcc3e5ea8288eadc64406d7033c" exitCode=0
Nov 03 20:36:51.578893 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:51.578662    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a9ba83/test-model-a9ba83-kserve-5f6985cb6c-d6pfg"
Nov 03 20:36:51.578893 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:51.578695    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a9ba83/test-model-a9ba83-kserve-5f6985cb6c-d6pfg" event={"ID":"b2d9f055-1380-4eb4-84ee-48dc75855852","Type":"ContainerDied","Data":"69369f1cd566da840ae692406efc55dc6741ddcc3e5ea8288eadc64406d7033c"}
Nov 03 20:36:51.578893 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:51.578740    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a9ba83/test-model-a9ba83-kserve-5f6985cb6c-d6pfg" event={"ID":"b2d9f055-1380-4eb4-84ee-48dc75855852","Type":"ContainerDied","Data":"4730f9b56db16e727239799d26df90df41d167ff9f1be1e4ec4467147ff758b6"}
Nov 03 20:36:51.578893 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:51.578762    5570 scope.go:122] "RemoveContainer" containerID="69369f1cd566da840ae692406efc55dc6741ddcc3e5ea8288eadc64406d7033c"
Nov 03 20:36:51.580583 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:51.580552    5570 generic.go:356] "Generic (PLEG): container finished" podID="c6fb7442-714a-4e4a-a6cf-19326b50a929" containerID="f7d8105e38072ca5fdd84f547f0b0fed18c8772f8ce6f4e0cc52c9f7cb80e806" exitCode=0
Nov 03 20:36:51.580714 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:51.580608    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-60efdf-openshift-default-5dfc484849-k6445"
Nov 03 20:36:51.580714 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:51.580620    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-60efdf-openshift-default-5dfc484849-k6445" event={"ID":"c6fb7442-714a-4e4a-a6cf-19326b50a929","Type":"ContainerDied","Data":"f7d8105e38072ca5fdd84f547f0b0fed18c8772f8ce6f4e0cc52c9f7cb80e806"}
Nov 03 20:36:51.580714 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:51.580654    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-60efdf-openshift-default-5dfc484849-k6445" event={"ID":"c6fb7442-714a-4e4a-a6cf-19326b50a929","Type":"ContainerDied","Data":"3142a43f29671e032e8ff70ba749cd8c00c7f3930d96979dbe7456d9b8ca9461"}
Nov 03 20:36:51.597065 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:51.597001    5570 scope.go:122] "RemoveContainer" containerID="69369f1cd566da840ae692406efc55dc6741ddcc3e5ea8288eadc64406d7033c"
Nov 03 20:36:51.597219 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:51.597202    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2d9f055-1380-4eb4-84ee-48dc75855852-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2d9f055-1380-4eb4-84ee-48dc75855852-tmp-dir\") pod \"b2d9f055-1380-4eb4-84ee-48dc75855852\" (UID: \"b2d9f055-1380-4eb4-84ee-48dc75855852\") "
Nov 03 20:36:51.597296 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:51.597256    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b2d9f055-1380-4eb4-84ee-48dc75855852-tls-certs\" (UniqueName: \"kubernetes.io/secret/b2d9f055-1380-4eb4-84ee-48dc75855852-tls-certs\") pod \"b2d9f055-1380-4eb4-84ee-48dc75855852\" (UID: \"b2d9f055-1380-4eb4-84ee-48dc75855852\") "
Nov 03 20:36:51.597380 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:51.597360    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2d9f055-1380-4eb4-84ee-48dc75855852-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2d9f055-1380-4eb4-84ee-48dc75855852-model-cache\") pod \"b2d9f055-1380-4eb4-84ee-48dc75855852\" (UID: \"b2d9f055-1380-4eb4-84ee-48dc75855852\") "
Nov 03 20:36:51.597489 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:51.597448    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2d9f055-1380-4eb4-84ee-48dc75855852-home\" (UniqueName: \"kubernetes.io/empty-dir/b2d9f055-1380-4eb4-84ee-48dc75855852-home\") pod \"b2d9f055-1380-4eb4-84ee-48dc75855852\" (UID: \"b2d9f055-1380-4eb4-84ee-48dc75855852\") "
Nov 03 20:36:51.597489 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:51.597478    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2d9f055-1380-4eb4-84ee-48dc75855852-tmp-dir" pod "b2d9f055-1380-4eb4-84ee-48dc75855852" (UID: "b2d9f055-1380-4eb4-84ee-48dc75855852"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:36:51.597617 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:51.597601    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2d9f055-1380-4eb4-84ee-48dc75855852-kube-api-access-wp6gx\" (UniqueName: \"kubernetes.io/projected/b2d9f055-1380-4eb4-84ee-48dc75855852-kube-api-access-wp6gx\") pod \"b2d9f055-1380-4eb4-84ee-48dc75855852\" (UID: \"b2d9f055-1380-4eb4-84ee-48dc75855852\") "
Nov 03 20:36:51.597617 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:36:51.597576    5570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69369f1cd566da840ae692406efc55dc6741ddcc3e5ea8288eadc64406d7033c\": container with ID starting with 69369f1cd566da840ae692406efc55dc6741ddcc3e5ea8288eadc64406d7033c not found: ID does not exist" containerID="69369f1cd566da840ae692406efc55dc6741ddcc3e5ea8288eadc64406d7033c"
Nov 03 20:36:51.597720 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:51.597636    5570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69369f1cd566da840ae692406efc55dc6741ddcc3e5ea8288eadc64406d7033c"} err="failed to get container status \"69369f1cd566da840ae692406efc55dc6741ddcc3e5ea8288eadc64406d7033c\": rpc error: code = NotFound desc = could not find container \"69369f1cd566da840ae692406efc55dc6741ddcc3e5ea8288eadc64406d7033c\": container with ID starting with 69369f1cd566da840ae692406efc55dc6741ddcc3e5ea8288eadc64406d7033c not found: ID does not exist"
Nov 03 20:36:51.597720 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:51.597652    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2d9f055-1380-4eb4-84ee-48dc75855852-model-cache" pod "b2d9f055-1380-4eb4-84ee-48dc75855852" (UID: "b2d9f055-1380-4eb4-84ee-48dc75855852"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:36:51.597720 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:51.597663    5570 scope.go:122] "RemoveContainer" containerID="f7d8105e38072ca5fdd84f547f0b0fed18c8772f8ce6f4e0cc52c9f7cb80e806"
Nov 03 20:36:51.598918 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:51.597648    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2d9f055-1380-4eb4-84ee-48dc75855852-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2d9f055-1380-4eb4-84ee-48dc75855852-dshm\") pod \"b2d9f055-1380-4eb4-84ee-48dc75855852\" (UID: \"b2d9f055-1380-4eb4-84ee-48dc75855852\") "
Nov 03 20:36:51.598918 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:51.597891    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2d9f055-1380-4eb4-84ee-48dc75855852-home" pod "b2d9f055-1380-4eb4-84ee-48dc75855852" (UID: "b2d9f055-1380-4eb4-84ee-48dc75855852"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:36:51.598918 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:51.598357    5570 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2d9f055-1380-4eb4-84ee-48dc75855852-home\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:36:51.598918 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:51.598377    5570 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2d9f055-1380-4eb4-84ee-48dc75855852-tmp-dir\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:36:51.598918 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:51.598389    5570 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2d9f055-1380-4eb4-84ee-48dc75855852-model-cache\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:36:51.616211 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:51.616159    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2d9f055-1380-4eb4-84ee-48dc75855852-dshm" pod "b2d9f055-1380-4eb4-84ee-48dc75855852" (UID: "b2d9f055-1380-4eb4-84ee-48dc75855852"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:36:51.616673 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:51.616640    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2d9f055-1380-4eb4-84ee-48dc75855852-tls-certs" pod "b2d9f055-1380-4eb4-84ee-48dc75855852" (UID: "b2d9f055-1380-4eb4-84ee-48dc75855852"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:36:51.616847 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:51.616675    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2d9f055-1380-4eb4-84ee-48dc75855852-kube-api-access-wp6gx" pod "b2d9f055-1380-4eb4-84ee-48dc75855852" (UID: "b2d9f055-1380-4eb4-84ee-48dc75855852"). InnerVolumeSpecName "kube-api-access-wp6gx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:36:51.617121 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:51.617085    5570 scope.go:122] "RemoveContainer" containerID="4537e19896b788980331af4346ff30e28e6227d9410b794ab6558c526b1fc121"
Nov 03 20:36:51.627713 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:51.627687    5570 scope.go:122] "RemoveContainer" containerID="f7d8105e38072ca5fdd84f547f0b0fed18c8772f8ce6f4e0cc52c9f7cb80e806"
Nov 03 20:36:51.628108 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:36:51.628084    5570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7d8105e38072ca5fdd84f547f0b0fed18c8772f8ce6f4e0cc52c9f7cb80e806\": container with ID starting with f7d8105e38072ca5fdd84f547f0b0fed18c8772f8ce6f4e0cc52c9f7cb80e806 not found: ID does not exist" containerID="f7d8105e38072ca5fdd84f547f0b0fed18c8772f8ce6f4e0cc52c9f7cb80e806"
Nov 03 20:36:51.628184 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:51.628118    5570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7d8105e38072ca5fdd84f547f0b0fed18c8772f8ce6f4e0cc52c9f7cb80e806"} err="failed to get container status \"f7d8105e38072ca5fdd84f547f0b0fed18c8772f8ce6f4e0cc52c9f7cb80e806\": rpc error: code = NotFound desc = could not find container \"f7d8105e38072ca5fdd84f547f0b0fed18c8772f8ce6f4e0cc52c9f7cb80e806\": container with ID starting with f7d8105e38072ca5fdd84f547f0b0fed18c8772f8ce6f4e0cc52c9f7cb80e806 not found: ID does not exist"
Nov 03 20:36:51.628184 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:51.628138    5570 scope.go:122] "RemoveContainer" containerID="4537e19896b788980331af4346ff30e28e6227d9410b794ab6558c526b1fc121"
Nov 03 20:36:51.628460 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:36:51.628431    5570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4537e19896b788980331af4346ff30e28e6227d9410b794ab6558c526b1fc121\": container with ID starting with 4537e19896b788980331af4346ff30e28e6227d9410b794ab6558c526b1fc121 not found: ID does not exist" containerID="4537e19896b788980331af4346ff30e28e6227d9410b794ab6558c526b1fc121"
Nov 03 20:36:51.628507 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:51.628471    5570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4537e19896b788980331af4346ff30e28e6227d9410b794ab6558c526b1fc121"} err="failed to get container status \"4537e19896b788980331af4346ff30e28e6227d9410b794ab6558c526b1fc121\": rpc error: code = NotFound desc = could not find container \"4537e19896b788980331af4346ff30e28e6227d9410b794ab6558c526b1fc121\": container with ID starting with 4537e19896b788980331af4346ff30e28e6227d9410b794ab6558c526b1fc121 not found: ID does not exist"
Nov 03 20:36:51.662165 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:51.662131    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-60efdf-openshift-default-5dfc484849-k6445"]
Nov 03 20:36:51.665473 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:51.665445    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-60efdf-openshift-default-5dfc484849-k6445"]
Nov 03 20:36:51.699530 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:51.699490    5570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2d9f055-1380-4eb4-84ee-48dc75855852-tls-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:36:51.699530 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:51.699524    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wp6gx\" (UniqueName: \"kubernetes.io/projected/b2d9f055-1380-4eb4-84ee-48dc75855852-kube-api-access-wp6gx\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:36:51.699530 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:51.699537    5570 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2d9f055-1380-4eb4-84ee-48dc75855852-dshm\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:36:51.927883 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:51.927830    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-a9ba83/test-model-a9ba83-kserve-5f6985cb6c-d6pfg"]
Nov 03 20:36:51.932354 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:51.932323    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-a9ba83/test-model-a9ba83-kserve-5f6985cb6c-d6pfg"]
Nov 03 20:36:51.965733 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:51.965694    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2d9f055-1380-4eb4-84ee-48dc75855852" path="/var/lib/kubelet/pods/b2d9f055-1380-4eb4-84ee-48dc75855852/volumes"
Nov 03 20:36:51.966240 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:51.966225    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6fb7442-714a-4e4a-a6cf-19326b50a929" path="/var/lib/kubelet/pods/c6fb7442-714a-4e4a-a6cf-19326b50a929/volumes"
Nov 03 20:36:54.315160 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:54.315120    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-60efdf-7bfdc67bd7-j6ctn" podUID="effc03b7-0b9e-4239-baa0-5c6cd120f479" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 20:36:56.963768 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:56.963733    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-1d159a-5497649584-zvjng"]
Nov 03 20:36:56.995271 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:56.995229    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-1d159a-5c484658dc-8qvxq"]
Nov 03 20:36:57.229288 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:36:57.229200    5570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8bwhz" containerName="extractor"
Nov 03 20:36:58.169335 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:58.169290    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a9ba83-67b98df76d-24dtv"]
Nov 03 20:36:58.201653 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:58.201612    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a9ba83-67d4dd56f9-22nhs"]
Nov 03 20:36:58.469573 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:58.469534    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1d159a-49svt"]
Nov 03 20:36:58.472618 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:58.472593    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1d159a-49svt"]
Nov 03 20:36:59.965441 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:36:59.965405    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5bbecbfa-d55a-4ec2-9db5-77e5d9675955" path="/var/lib/kubelet/pods/5bbecbfa-d55a-4ec2-9db5-77e5d9675955/volumes"
Nov 03 20:37:00.673449 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:00.673406    5570 generic.go:356] "Generic (PLEG): container finished" podID="effc03b7-0b9e-4239-baa0-5c6cd120f479" containerID="2ec3f67934e86a94a7ef166e62c2b6e9cc6fcf63f2ac547d75064af34674793a" exitCode=0
Nov 03 20:37:00.673667 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:00.673483    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-60efdf-7bfdc67bd7-j6ctn" event={"ID":"effc03b7-0b9e-4239-baa0-5c6cd120f479","Type":"ContainerDied","Data":"2ec3f67934e86a94a7ef166e62c2b6e9cc6fcf63f2ac547d75064af34674793a"}
Nov 03 20:37:00.712409 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:00.712333    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-60efdf-7bfdc67bd7-j6ctn"
Nov 03 20:37:00.833290 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:00.833255    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/effc03b7-0b9e-4239-baa0-5c6cd120f479-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/effc03b7-0b9e-4239-baa0-5c6cd120f479-plugins-config-volume\") pod \"effc03b7-0b9e-4239-baa0-5c6cd120f479\" (UID: \"effc03b7-0b9e-4239-baa0-5c6cd120f479\") "
Nov 03 20:37:00.833500 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:00.833370    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/effc03b7-0b9e-4239-baa0-5c6cd120f479-kube-api-access-jkvrq\" (UniqueName: \"kubernetes.io/projected/effc03b7-0b9e-4239-baa0-5c6cd120f479-kube-api-access-jkvrq\") pod \"effc03b7-0b9e-4239-baa0-5c6cd120f479\" (UID: \"effc03b7-0b9e-4239-baa0-5c6cd120f479\") "
Nov 03 20:37:00.833738 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:00.833710    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/effc03b7-0b9e-4239-baa0-5c6cd120f479-plugins-config-volume" pod "effc03b7-0b9e-4239-baa0-5c6cd120f479" (UID: "effc03b7-0b9e-4239-baa0-5c6cd120f479"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:37:00.834059 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:00.834040    5570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/effc03b7-0b9e-4239-baa0-5c6cd120f479-plugins-config-volume\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:37:00.835554 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:00.835516    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/effc03b7-0b9e-4239-baa0-5c6cd120f479-kube-api-access-jkvrq" pod "effc03b7-0b9e-4239-baa0-5c6cd120f479" (UID: "effc03b7-0b9e-4239-baa0-5c6cd120f479"). InnerVolumeSpecName "kube-api-access-jkvrq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:37:00.934886 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:00.934839    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkvrq\" (UniqueName: \"kubernetes.io/projected/effc03b7-0b9e-4239-baa0-5c6cd120f479-kube-api-access-jkvrq\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:37:01.682504 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:01.682477    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-60efdf-7bfdc67bd7-j6ctn"
Nov 03 20:37:01.683017 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:01.682516    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-60efdf-7bfdc67bd7-j6ctn" event={"ID":"effc03b7-0b9e-4239-baa0-5c6cd120f479","Type":"ContainerDied","Data":"5e3b8247ca9ebf4f87360412db37c15b8ae96e11d5f859986b7cdf1ddc56aa15"}
Nov 03 20:37:01.683017 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:01.682580    5570 scope.go:122] "RemoveContainer" containerID="2ec3f67934e86a94a7ef166e62c2b6e9cc6fcf63f2ac547d75064af34674793a"
Nov 03 20:37:01.737719 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:01.737675    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-60efdf-7bfdc67bd7-j6ctn"]
Nov 03 20:37:01.751228 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:01.751195    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-60efdf-7bfdc67bd7-j6ctn"]
Nov 03 20:37:01.968946 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:01.968906    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g69f8"]
Nov 03 20:37:01.981177 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:01.981125    5570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-1d159a-5497649584-zvjng" podUID="96055efa-780d-46b1-8d17-31587157b508" containerName="payload-processing" containerID="cri-o://4acb55c63c600be5f8c437ba91538b91b5cdd1506ef232360c71451b0e035676" gracePeriod=25
Nov 03 20:37:01.995545 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:01.995506    5570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1d159a-5c484658dc-8qvxq" podUID="1ed630bf-cf60-4e37-9113-a1707b99937f" containerName="payload-pre-processing" containerID="cri-o://dd61f44cdbdbff9a9b8c70ea8c8cc2997d1eae214f81d484aaf387a2325fffee" gracePeriod=25
Nov 03 20:37:02.000752 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:02.000720    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="effc03b7-0b9e-4239-baa0-5c6cd120f479" path="/var/lib/kubelet/pods/effc03b7-0b9e-4239-baa0-5c6cd120f479/volumes"
Nov 03 20:37:02.280143 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:02.280114    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-1d159a-5497649584-zvjng"
Nov 03 20:37:02.389806 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:02.376614    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96055efa-780d-46b1-8d17-31587157b508-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96055efa-780d-46b1-8d17-31587157b508-plugins-config-volume\") pod \"96055efa-780d-46b1-8d17-31587157b508\" (UID: \"96055efa-780d-46b1-8d17-31587157b508\") "
Nov 03 20:37:02.389806 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:02.376763    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96055efa-780d-46b1-8d17-31587157b508-kube-api-access-qk6mv\" (UniqueName: \"kubernetes.io/projected/96055efa-780d-46b1-8d17-31587157b508-kube-api-access-qk6mv\") pod \"96055efa-780d-46b1-8d17-31587157b508\" (UID: \"96055efa-780d-46b1-8d17-31587157b508\") "
Nov 03 20:37:02.389806 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:02.383829    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96055efa-780d-46b1-8d17-31587157b508-plugins-config-volume" pod "96055efa-780d-46b1-8d17-31587157b508" (UID: "96055efa-780d-46b1-8d17-31587157b508"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:37:02.413797 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:02.402314    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96055efa-780d-46b1-8d17-31587157b508-kube-api-access-qk6mv" pod "96055efa-780d-46b1-8d17-31587157b508" (UID: "96055efa-780d-46b1-8d17-31587157b508"). InnerVolumeSpecName "kube-api-access-qk6mv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:37:02.478666 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:02.478626    5570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96055efa-780d-46b1-8d17-31587157b508-plugins-config-volume\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:37:02.478666 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:02.478663    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qk6mv\" (UniqueName: \"kubernetes.io/projected/96055efa-780d-46b1-8d17-31587157b508-kube-api-access-qk6mv\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:37:02.694415 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:02.694370    5570 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-4b95f3/test-model-4b95f3-kserve-54555ccc49-ncvhw" podUID="11cd87af-5c07-4e83-80ad-72c4f7e2482b" containerName="main" containerID="cri-o://c2d718a731911d49f885ce4a3e274c1b444f601ddca4fb57b7cfa54d63d73154" gracePeriod=45
Nov 03 20:37:02.698326 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:02.698294    5570 generic.go:356] "Generic (PLEG): container finished" podID="96055efa-780d-46b1-8d17-31587157b508" containerID="4acb55c63c600be5f8c437ba91538b91b5cdd1506ef232360c71451b0e035676" exitCode=0
Nov 03 20:37:02.698477 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:02.698358    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-1d159a-5497649584-zvjng"
Nov 03 20:37:02.698477 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:02.698370    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-1d159a-5497649584-zvjng" event={"ID":"96055efa-780d-46b1-8d17-31587157b508","Type":"ContainerDied","Data":"4acb55c63c600be5f8c437ba91538b91b5cdd1506ef232360c71451b0e035676"}
Nov 03 20:37:02.698477 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:02.698468    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-1d159a-5497649584-zvjng" event={"ID":"96055efa-780d-46b1-8d17-31587157b508","Type":"ContainerDied","Data":"1f4419d15f67647080e121042acf1150dfd460859516e223f90a74c2f4e8a6eb"}
Nov 03 20:37:02.698761 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:02.698497    5570 scope.go:122] "RemoveContainer" containerID="4acb55c63c600be5f8c437ba91538b91b5cdd1506ef232360c71451b0e035676"
Nov 03 20:37:02.787646 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:02.787615    5570 scope.go:122] "RemoveContainer" containerID="4acb55c63c600be5f8c437ba91538b91b5cdd1506ef232360c71451b0e035676"
Nov 03 20:37:02.788397 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:37:02.788352    5570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4acb55c63c600be5f8c437ba91538b91b5cdd1506ef232360c71451b0e035676\": container with ID starting with 4acb55c63c600be5f8c437ba91538b91b5cdd1506ef232360c71451b0e035676 not found: ID does not exist" containerID="4acb55c63c600be5f8c437ba91538b91b5cdd1506ef232360c71451b0e035676"
Nov 03 20:37:02.788636 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:02.788585    5570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4acb55c63c600be5f8c437ba91538b91b5cdd1506ef232360c71451b0e035676"} err="failed to get container status \"4acb55c63c600be5f8c437ba91538b91b5cdd1506ef232360c71451b0e035676\": rpc error: code = NotFound desc = could not find container \"4acb55c63c600be5f8c437ba91538b91b5cdd1506ef232360c71451b0e035676\": container with ID starting with 4acb55c63c600be5f8c437ba91538b91b5cdd1506ef232360c71451b0e035676 not found: ID does not exist"
Nov 03 20:37:02.811209 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:02.811171    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-4b95f3/test-model-4b95f3-kserve-54555ccc49-ncvhw" podUID="11cd87af-5c07-4e83-80ad-72c4f7e2482b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.174:8000/ready\": dial tcp 10.134.0.174:8000: connect: connection refused"
Nov 03 20:37:02.814693 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:02.814668    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-1d159a-5497649584-zvjng"]
Nov 03 20:37:02.828217 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:02.828188    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-1d159a-5497649584-zvjng"]
Nov 03 20:37:02.946559 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:02.946494    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-4b95f3/test-model-4b95f3-kserve-54555ccc49-ncvhw"
Nov 03 20:37:02.985600 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:02.985568    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11cd87af-5c07-4e83-80ad-72c4f7e2482b-kube-api-access-ghmvw\" (UniqueName: \"kubernetes.io/projected/11cd87af-5c07-4e83-80ad-72c4f7e2482b-kube-api-access-ghmvw\") pod \"11cd87af-5c07-4e83-80ad-72c4f7e2482b\" (UID: \"11cd87af-5c07-4e83-80ad-72c4f7e2482b\") "
Nov 03 20:37:02.985814 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:02.985633    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11cd87af-5c07-4e83-80ad-72c4f7e2482b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/11cd87af-5c07-4e83-80ad-72c4f7e2482b-model-cache\") pod \"11cd87af-5c07-4e83-80ad-72c4f7e2482b\" (UID: \"11cd87af-5c07-4e83-80ad-72c4f7e2482b\") "
Nov 03 20:37:02.985814 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:02.985683    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11cd87af-5c07-4e83-80ad-72c4f7e2482b-home\" (UniqueName: \"kubernetes.io/empty-dir/11cd87af-5c07-4e83-80ad-72c4f7e2482b-home\") pod \"11cd87af-5c07-4e83-80ad-72c4f7e2482b\" (UID: \"11cd87af-5c07-4e83-80ad-72c4f7e2482b\") "
Nov 03 20:37:02.985814 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:02.985702    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/11cd87af-5c07-4e83-80ad-72c4f7e2482b-tls-certs\" (UniqueName: \"kubernetes.io/secret/11cd87af-5c07-4e83-80ad-72c4f7e2482b-tls-certs\") pod \"11cd87af-5c07-4e83-80ad-72c4f7e2482b\" (UID: \"11cd87af-5c07-4e83-80ad-72c4f7e2482b\") "
Nov 03 20:37:02.985814 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:02.985755    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11cd87af-5c07-4e83-80ad-72c4f7e2482b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/11cd87af-5c07-4e83-80ad-72c4f7e2482b-tmp-dir\") pod \"11cd87af-5c07-4e83-80ad-72c4f7e2482b\" (UID: \"11cd87af-5c07-4e83-80ad-72c4f7e2482b\") "
Nov 03 20:37:02.986060 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:02.985865    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11cd87af-5c07-4e83-80ad-72c4f7e2482b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/11cd87af-5c07-4e83-80ad-72c4f7e2482b-dshm\") pod \"11cd87af-5c07-4e83-80ad-72c4f7e2482b\" (UID: \"11cd87af-5c07-4e83-80ad-72c4f7e2482b\") "
Nov 03 20:37:02.986060 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:02.985913    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11cd87af-5c07-4e83-80ad-72c4f7e2482b-model-cache" pod "11cd87af-5c07-4e83-80ad-72c4f7e2482b" (UID: "11cd87af-5c07-4e83-80ad-72c4f7e2482b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:37:02.986060 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:02.985948    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11cd87af-5c07-4e83-80ad-72c4f7e2482b-home" pod "11cd87af-5c07-4e83-80ad-72c4f7e2482b" (UID: "11cd87af-5c07-4e83-80ad-72c4f7e2482b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:37:02.986214 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:02.986070    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11cd87af-5c07-4e83-80ad-72c4f7e2482b-tmp-dir" pod "11cd87af-5c07-4e83-80ad-72c4f7e2482b" (UID: "11cd87af-5c07-4e83-80ad-72c4f7e2482b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:37:02.986390 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:02.986368    5570 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/11cd87af-5c07-4e83-80ad-72c4f7e2482b-model-cache\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:37:02.986479 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:02.986392    5570 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/11cd87af-5c07-4e83-80ad-72c4f7e2482b-home\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:37:02.986479 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:02.986427    5570 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/11cd87af-5c07-4e83-80ad-72c4f7e2482b-tmp-dir\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:37:02.988257 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:02.988225    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11cd87af-5c07-4e83-80ad-72c4f7e2482b-kube-api-access-ghmvw" pod "11cd87af-5c07-4e83-80ad-72c4f7e2482b" (UID: "11cd87af-5c07-4e83-80ad-72c4f7e2482b"). InnerVolumeSpecName "kube-api-access-ghmvw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:37:02.988257 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:02.988224    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11cd87af-5c07-4e83-80ad-72c4f7e2482b-dshm" pod "11cd87af-5c07-4e83-80ad-72c4f7e2482b" (UID: "11cd87af-5c07-4e83-80ad-72c4f7e2482b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:37:02.988503 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:02.988425    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11cd87af-5c07-4e83-80ad-72c4f7e2482b-tls-certs" pod "11cd87af-5c07-4e83-80ad-72c4f7e2482b" (UID: "11cd87af-5c07-4e83-80ad-72c4f7e2482b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:37:03.087234 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:03.087199    5570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/11cd87af-5c07-4e83-80ad-72c4f7e2482b-tls-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:37:03.087234 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:03.087229    5570 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/11cd87af-5c07-4e83-80ad-72c4f7e2482b-dshm\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:37:03.087234 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:03.087242    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghmvw\" (UniqueName: \"kubernetes.io/projected/11cd87af-5c07-4e83-80ad-72c4f7e2482b-kube-api-access-ghmvw\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:37:03.170462 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:03.170380    5570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-a9ba83-67b98df76d-24dtv" podUID="6deae9d9-81da-4c77-a15a-d805ff547c51" containerName="payload-processing" containerID="cri-o://bb882318962c16fb856a808f831b224d05ff43f5de4c53413f46980a3b2c636c" gracePeriod=25
Nov 03 20:37:03.202409 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:03.202286    5570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a9ba83-67d4dd56f9-22nhs" podUID="58fc81a1-5df4-4b6c-92bc-6dcf0836f400" containerName="payload-pre-processing" containerID="cri-o://922ea1a73f9216f6a93ff2cd116a56b6f399719095090b582b4199a796ce54f5" gracePeriod=25
Nov 03 20:37:03.706259 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:03.706228    5570 generic.go:356] "Generic (PLEG): container finished" podID="11cd87af-5c07-4e83-80ad-72c4f7e2482b" containerID="c2d718a731911d49f885ce4a3e274c1b444f601ddca4fb57b7cfa54d63d73154" exitCode=0
Nov 03 20:37:03.706735 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:03.706271    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-4b95f3/test-model-4b95f3-kserve-54555ccc49-ncvhw"
Nov 03 20:37:03.706735 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:03.706314    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-4b95f3/test-model-4b95f3-kserve-54555ccc49-ncvhw" event={"ID":"11cd87af-5c07-4e83-80ad-72c4f7e2482b","Type":"ContainerDied","Data":"c2d718a731911d49f885ce4a3e274c1b444f601ddca4fb57b7cfa54d63d73154"}
Nov 03 20:37:03.706735 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:03.706356    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-4b95f3/test-model-4b95f3-kserve-54555ccc49-ncvhw" event={"ID":"11cd87af-5c07-4e83-80ad-72c4f7e2482b","Type":"ContainerDied","Data":"6834c83c13b04814877ab18d3610e4a28602d6ecf5ea20db49d826a6fce7287d"}
Nov 03 20:37:03.706735 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:03.706376    5570 scope.go:122] "RemoveContainer" containerID="c2d718a731911d49f885ce4a3e274c1b444f601ddca4fb57b7cfa54d63d73154"
Nov 03 20:37:03.716185 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:03.716165    5570 scope.go:122] "RemoveContainer" containerID="c2d718a731911d49f885ce4a3e274c1b444f601ddca4fb57b7cfa54d63d73154"
Nov 03 20:37:03.716522 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:37:03.716498    5570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2d718a731911d49f885ce4a3e274c1b444f601ddca4fb57b7cfa54d63d73154\": container with ID starting with c2d718a731911d49f885ce4a3e274c1b444f601ddca4fb57b7cfa54d63d73154 not found: ID does not exist" containerID="c2d718a731911d49f885ce4a3e274c1b444f601ddca4fb57b7cfa54d63d73154"
Nov 03 20:37:03.716580 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:03.716532    5570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2d718a731911d49f885ce4a3e274c1b444f601ddca4fb57b7cfa54d63d73154"} err="failed to get container status \"c2d718a731911d49f885ce4a3e274c1b444f601ddca4fb57b7cfa54d63d73154\": rpc error: code = NotFound desc = could not find container \"c2d718a731911d49f885ce4a3e274c1b444f601ddca4fb57b7cfa54d63d73154\": container with ID starting with c2d718a731911d49f885ce4a3e274c1b444f601ddca4fb57b7cfa54d63d73154 not found: ID does not exist"
Nov 03 20:37:03.759612 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:03.759577    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-4b95f3/test-model-4b95f3-kserve-54555ccc49-ncvhw"]
Nov 03 20:37:03.773079 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:03.773045    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-4b95f3/test-model-4b95f3-kserve-54555ccc49-ncvhw"]
Nov 03 20:37:03.898846 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:03.898811    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a9ba83-754d766d5d-nncw7"]
Nov 03 20:37:03.899083 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:03.899048    5570 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a9ba83-754d766d5d-nncw7" podUID="bf50af56-3e8b-4a3b-b871-5c882235c813" containerName="maas-api" containerID="cri-o://9ea8e68afdde1dbc16ab135a6e73e506bce67315790170fcf2326f8d640c9269" gracePeriod=30
Nov 03 20:37:03.964384 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:03.964304    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11cd87af-5c07-4e83-80ad-72c4f7e2482b" path="/var/lib/kubelet/pods/11cd87af-5c07-4e83-80ad-72c4f7e2482b/volumes"
Nov 03 20:37:03.964812 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:03.964765    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96055efa-780d-46b1-8d17-31587157b508" path="/var/lib/kubelet/pods/96055efa-780d-46b1-8d17-31587157b508/volumes"
Nov 03 20:37:03.975501 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:03.975467    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a9ba83-67d4dd56f9-22nhs" podUID="58fc81a1-5df4-4b6c-92bc-6dcf0836f400" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 20:37:03.975721 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:03.975552    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-a9ba83-67b98df76d-24dtv" podUID="6deae9d9-81da-4c77-a15a-d805ff547c51" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 20:37:04.483586 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:04.483516    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a9ba83-754d766d5d-nncw7"
Nov 03 20:37:04.615542 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:04.615501    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf50af56-3e8b-4a3b-b871-5c882235c813-metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf50af56-3e8b-4a3b-b871-5c882235c813-metrics-certs\") pod \"bf50af56-3e8b-4a3b-b871-5c882235c813\" (UID: \"bf50af56-3e8b-4a3b-b871-5c882235c813\") "
Nov 03 20:37:04.615754 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:04.615554    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf50af56-3e8b-4a3b-b871-5c882235c813-kube-api-access-khfjz\" (UniqueName: \"kubernetes.io/projected/bf50af56-3e8b-4a3b-b871-5c882235c813-kube-api-access-khfjz\") pod \"bf50af56-3e8b-4a3b-b871-5c882235c813\" (UID: \"bf50af56-3e8b-4a3b-b871-5c882235c813\") "
Nov 03 20:37:04.615754 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:04.615664    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf50af56-3e8b-4a3b-b871-5c882235c813-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf50af56-3e8b-4a3b-b871-5c882235c813-openshift-service-ca\") pod \"bf50af56-3e8b-4a3b-b871-5c882235c813\" (UID: \"bf50af56-3e8b-4a3b-b871-5c882235c813\") "
Nov 03 20:37:04.615754 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:04.615692    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf50af56-3e8b-4a3b-b871-5c882235c813-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf50af56-3e8b-4a3b-b871-5c882235c813-maas-api-tls\") pod \"bf50af56-3e8b-4a3b-b871-5c882235c813\" (UID: \"bf50af56-3e8b-4a3b-b871-5c882235c813\") "
Nov 03 20:37:04.616215 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:04.616186    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf50af56-3e8b-4a3b-b871-5c882235c813-openshift-service-ca" pod "bf50af56-3e8b-4a3b-b871-5c882235c813" (UID: "bf50af56-3e8b-4a3b-b871-5c882235c813"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:37:04.616420 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:04.616400    5570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf50af56-3e8b-4a3b-b871-5c882235c813-openshift-service-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:37:04.618202 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:04.618173    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf50af56-3e8b-4a3b-b871-5c882235c813-metrics-certs" pod "bf50af56-3e8b-4a3b-b871-5c882235c813" (UID: "bf50af56-3e8b-4a3b-b871-5c882235c813"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:37:04.618325 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:04.618298    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf50af56-3e8b-4a3b-b871-5c882235c813-maas-api-tls" pod "bf50af56-3e8b-4a3b-b871-5c882235c813" (UID: "bf50af56-3e8b-4a3b-b871-5c882235c813"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:37:04.618394 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:04.618351    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf50af56-3e8b-4a3b-b871-5c882235c813-kube-api-access-khfjz" pod "bf50af56-3e8b-4a3b-b871-5c882235c813" (UID: "bf50af56-3e8b-4a3b-b871-5c882235c813"). InnerVolumeSpecName "kube-api-access-khfjz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:37:04.620682 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:04.620653    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-2x92x"]
Nov 03 20:37:04.620941 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:04.620904    5570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-2x92x" podUID="2ac2fc9c-0a3e-4326-a4f8-8856b1a30e36" containerName="istio-proxy" containerID="cri-o://ba1d33e261ae5b946cd1afd83be4e12a57d46a9a8ace20e5a4394ae6e0f6b53f" gracePeriod=30
Nov 03 20:37:04.625497 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:04.625469    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-vdrfs"]
Nov 03 20:37:04.625723 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:04.625693    5570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-vdrfs" podUID="1b70a87b-76a6-40e6-9ced-037c3ceee5cf" containerName="istio-proxy" containerID="cri-o://cfc34fea0388713e474e0bfd29b896ec0c51595167214dd09881337a3d7882e6" gracePeriod=30
Nov 03 20:37:04.633163 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:04.629007    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-crn7x"]
Nov 03 20:37:04.633163 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:04.629225    5570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-crn7x" podUID="4ae62bda-375f-4252-8606-82bae17d680d" containerName="istio-proxy" containerID="cri-o://9b29a8fc84e904bbb506a3ae0e2651fd977a35f533be777b2ac4126aad45d515" gracePeriod=30
Nov 03 20:37:04.715683 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:04.715649    5570 generic.go:356] "Generic (PLEG): container finished" podID="bf50af56-3e8b-4a3b-b871-5c882235c813" containerID="9ea8e68afdde1dbc16ab135a6e73e506bce67315790170fcf2326f8d640c9269" exitCode=0
Nov 03 20:37:04.716167 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:04.715703    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a9ba83-754d766d5d-nncw7"
Nov 03 20:37:04.716167 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:04.715728    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a9ba83-754d766d5d-nncw7" event={"ID":"bf50af56-3e8b-4a3b-b871-5c882235c813","Type":"ContainerDied","Data":"9ea8e68afdde1dbc16ab135a6e73e506bce67315790170fcf2326f8d640c9269"}
Nov 03 20:37:04.716167 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:04.715796    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a9ba83-754d766d5d-nncw7" event={"ID":"bf50af56-3e8b-4a3b-b871-5c882235c813","Type":"ContainerDied","Data":"a54e45ef93c7166b3de04d9eacd8b4d15234ea159306c56e0b394acc9d1abcc4"}
Nov 03 20:37:04.716167 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:04.715827    5570 scope.go:122] "RemoveContainer" containerID="9ea8e68afdde1dbc16ab135a6e73e506bce67315790170fcf2326f8d640c9269"
Nov 03 20:37:04.717368 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:04.717347    5570 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf50af56-3e8b-4a3b-b871-5c882235c813-metrics-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:37:04.717445 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:04.717375    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khfjz\" (UniqueName: \"kubernetes.io/projected/bf50af56-3e8b-4a3b-b871-5c882235c813-kube-api-access-khfjz\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:37:04.717445 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:04.717393    5570 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf50af56-3e8b-4a3b-b871-5c882235c813-maas-api-tls\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:37:04.727520 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:04.727499    5570 scope.go:122] "RemoveContainer" containerID="9ea8e68afdde1dbc16ab135a6e73e506bce67315790170fcf2326f8d640c9269"
Nov 03 20:37:04.727841 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:37:04.727821    5570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ea8e68afdde1dbc16ab135a6e73e506bce67315790170fcf2326f8d640c9269\": container with ID starting with 9ea8e68afdde1dbc16ab135a6e73e506bce67315790170fcf2326f8d640c9269 not found: ID does not exist" containerID="9ea8e68afdde1dbc16ab135a6e73e506bce67315790170fcf2326f8d640c9269"
Nov 03 20:37:04.727948 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:04.727859    5570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ea8e68afdde1dbc16ab135a6e73e506bce67315790170fcf2326f8d640c9269"} err="failed to get container status \"9ea8e68afdde1dbc16ab135a6e73e506bce67315790170fcf2326f8d640c9269\": rpc error: code = NotFound desc = could not find container \"9ea8e68afdde1dbc16ab135a6e73e506bce67315790170fcf2326f8d640c9269\": container with ID starting with 9ea8e68afdde1dbc16ab135a6e73e506bce67315790170fcf2326f8d640c9269 not found: ID does not exist"
Nov 03 20:37:04.792728 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:04.792635    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a9ba83-754d766d5d-nncw7"]
Nov 03 20:37:04.795494 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:04.795468    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a9ba83-754d766d5d-nncw7"]
Nov 03 20:37:05.851426 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:05.851379    5570 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-2x92x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 20:37:05.851876 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:05.851445    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-2x92x" podUID="2ac2fc9c-0a3e-4326-a4f8-8856b1a30e36" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 20:37:05.982921 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:05.982879    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf50af56-3e8b-4a3b-b871-5c882235c813" path="/var/lib/kubelet/pods/bf50af56-3e8b-4a3b-b871-5c882235c813/volumes"
Nov 03 20:37:05.983300 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:05.983281    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-1d159a-openshift-default-799c8f9695-b8h67"]
Nov 03 20:37:05.983512 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:05.983490    5570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-1d159a-openshift-default-799c8f9695-b8h67" podUID="40e4fc9c-c3c1-4aa0-98ad-eda347be89c5" containerName="istio-proxy" containerID="cri-o://d200d60ddf9d712ccd5b388bb3ab8a745c07816a59a52cc300d0254e93a41a89" gracePeriod=30
Nov 03 20:37:06.553518 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:06.553483    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-503c4e-vjmmp"]
Nov 03 20:37:06.554021 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:06.554007    5570 cpu_manager.go:414] "containerMap: removing container" podUID="bf50af56-3e8b-4a3b-b871-5c882235c813" containerName="maas-api"
Nov 03 20:37:06.554088 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:06.554022    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf50af56-3e8b-4a3b-b871-5c882235c813" containerName="maas-api"
Nov 03 20:37:06.554088 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:06.554039    5570 cpu_manager.go:414] "containerMap: removing container" podUID="b2d9f055-1380-4eb4-84ee-48dc75855852" containerName="main"
Nov 03 20:37:06.554088 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:06.554044    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2d9f055-1380-4eb4-84ee-48dc75855852" containerName="main"
Nov 03 20:37:06.554088 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:06.554057    5570 cpu_manager.go:414] "containerMap: removing container" podUID="effc03b7-0b9e-4239-baa0-5c6cd120f479" containerName="payload-pre-processing"
Nov 03 20:37:06.554088 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:06.554063    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="effc03b7-0b9e-4239-baa0-5c6cd120f479" containerName="payload-pre-processing"
Nov 03 20:37:06.554088 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:06.554076    5570 cpu_manager.go:414] "containerMap: removing container" podUID="c6fb7442-714a-4e4a-a6cf-19326b50a929" containerName="istio-proxy"
Nov 03 20:37:06.554088 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:06.554081    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6fb7442-714a-4e4a-a6cf-19326b50a929" containerName="istio-proxy"
Nov 03 20:37:06.554088 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:06.554090    5570 cpu_manager.go:414] "containerMap: removing container" podUID="11cd87af-5c07-4e83-80ad-72c4f7e2482b" containerName="main"
Nov 03 20:37:06.554339 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:06.554096    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11cd87af-5c07-4e83-80ad-72c4f7e2482b" containerName="main"
Nov 03 20:37:06.554339 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:06.554112    5570 cpu_manager.go:414] "containerMap: removing container" podUID="96055efa-780d-46b1-8d17-31587157b508" containerName="payload-processing"
Nov 03 20:37:06.554339 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:06.554117    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96055efa-780d-46b1-8d17-31587157b508" containerName="payload-processing"
Nov 03 20:37:06.554339 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:06.554124    5570 cpu_manager.go:414] "containerMap: removing container" podUID="5bbecbfa-d55a-4ec2-9db5-77e5d9675955" containerName="revoke-keys"
Nov 03 20:37:06.554339 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:06.554129    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5bbecbfa-d55a-4ec2-9db5-77e5d9675955" containerName="revoke-keys"
Nov 03 20:37:06.554339 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:06.554216    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6fb7442-714a-4e4a-a6cf-19326b50a929" containerName="istio-proxy"
Nov 03 20:37:06.554339 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:06.554224    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="11cd87af-5c07-4e83-80ad-72c4f7e2482b" containerName="main"
Nov 03 20:37:06.554339 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:06.554231    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6fb7442-714a-4e4a-a6cf-19326b50a929" containerName="istio-proxy"
Nov 03 20:37:06.554339 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:06.554239    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="96055efa-780d-46b1-8d17-31587157b508" containerName="payload-processing"
Nov 03 20:37:06.554339 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:06.554247    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="5bbecbfa-d55a-4ec2-9db5-77e5d9675955" containerName="revoke-keys"
Nov 03 20:37:06.554339 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:06.554256    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf50af56-3e8b-4a3b-b871-5c882235c813" containerName="maas-api"
Nov 03 20:37:06.554339 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:06.554263    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2d9f055-1380-4eb4-84ee-48dc75855852" containerName="main"
Nov 03 20:37:06.554339 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:06.554268    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="effc03b7-0b9e-4239-baa0-5c6cd120f479" containerName="payload-pre-processing"
Nov 03 20:37:06.559039 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:06.559015    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-503c4e-vjmmp"
Nov 03 20:37:06.574461 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:06.574424    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-503c4e-vjmmp"]
Nov 03 20:37:06.574686 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:06.574670    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:37:06.574758 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:06.574672    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l7s5w\""
Nov 03 20:37:06.645883 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:06.645847    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttxws\" (UniqueName: \"kubernetes.io/projected/0d5c0fd7-110a-414e-a0cc-acb706ed119b-kube-api-access-ttxws\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-503c4e-vjmmp\" (UID: \"0d5c0fd7-110a-414e-a0cc-acb706ed119b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-503c4e-vjmmp"
Nov 03 20:37:06.646110 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:06.646086    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d5c0fd7-110a-414e-a0cc-acb706ed119b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-503c4e-vjmmp\" (UID: \"0d5c0fd7-110a-414e-a0cc-acb706ed119b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-503c4e-vjmmp"
Nov 03 20:37:06.646179 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:06.646130    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d5c0fd7-110a-414e-a0cc-acb706ed119b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-503c4e-vjmmp\" (UID: \"0d5c0fd7-110a-414e-a0cc-acb706ed119b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-503c4e-vjmmp"
Nov 03 20:37:06.747441 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:06.747396    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d5c0fd7-110a-414e-a0cc-acb706ed119b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-503c4e-vjmmp\" (UID: \"0d5c0fd7-110a-414e-a0cc-acb706ed119b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-503c4e-vjmmp"
Nov 03 20:37:06.747441 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:06.747448    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d5c0fd7-110a-414e-a0cc-acb706ed119b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-503c4e-vjmmp\" (UID: \"0d5c0fd7-110a-414e-a0cc-acb706ed119b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-503c4e-vjmmp"
Nov 03 20:37:06.747748 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:06.747486    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttxws\" (UniqueName: \"kubernetes.io/projected/0d5c0fd7-110a-414e-a0cc-acb706ed119b-kube-api-access-ttxws\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-503c4e-vjmmp\" (UID: \"0d5c0fd7-110a-414e-a0cc-acb706ed119b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-503c4e-vjmmp"
Nov 03 20:37:06.748228 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:06.748198    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d5c0fd7-110a-414e-a0cc-acb706ed119b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-503c4e-vjmmp\" (UID: \"0d5c0fd7-110a-414e-a0cc-acb706ed119b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-503c4e-vjmmp"
Nov 03 20:37:06.750033 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:06.750004    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d5c0fd7-110a-414e-a0cc-acb706ed119b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-503c4e-vjmmp\" (UID: \"0d5c0fd7-110a-414e-a0cc-acb706ed119b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-503c4e-vjmmp"
Nov 03 20:37:06.768108 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:06.768078    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttxws\" (UniqueName: \"kubernetes.io/projected/0d5c0fd7-110a-414e-a0cc-acb706ed119b-kube-api-access-ttxws\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-503c4e-vjmmp\" (UID: \"0d5c0fd7-110a-414e-a0cc-acb706ed119b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-503c4e-vjmmp"
Nov 03 20:37:06.874005 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:06.873902    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-503c4e-vjmmp"
Nov 03 20:37:07.171230 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:07.171192    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-503c4e-vjmmp"]
Nov 03 20:37:07.172602 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:37:07.172575    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d5c0fd7_110a_414e_a0cc_acb706ed119b.slice/crio-c44429429ff4ec12c47ce3395d60ea2f82c5405a3f7c252fe6970c7560bfa265 WatchSource:0}: Error finding container c44429429ff4ec12c47ce3395d60ea2f82c5405a3f7c252fe6970c7560bfa265: Status 404 returned error can't find the container with id c44429429ff4ec12c47ce3395d60ea2f82c5405a3f7c252fe6970c7560bfa265
Nov 03 20:37:07.739051 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:07.738958    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-503c4e-vjmmp" event={"ID":"0d5c0fd7-110a-414e-a0cc-acb706ed119b","Type":"ContainerStarted","Data":"68afaae48b42fbeeb53562b5cbe83c10698a53b81cb7c101633a459964020902"}
Nov 03 20:37:07.739051 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:07.739030    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-503c4e-vjmmp" event={"ID":"0d5c0fd7-110a-414e-a0cc-acb706ed119b","Type":"ContainerStarted","Data":"c44429429ff4ec12c47ce3395d60ea2f82c5405a3f7c252fe6970c7560bfa265"}
Nov 03 20:37:07.780800 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:07.780732    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-503c4e-vjmmp" podStartSLOduration=1.78071694 podStartE2EDuration="1.78071694s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:37:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:37:07.778953236 +0000 UTC m=+1961.931615122" watchObservedRunningTime="2026-11-03 20:37:07.78071694 +0000 UTC m=+1961.933378826"
Nov 03 20:37:07.861124 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:07.861068    5570 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-vdrfs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 20:37:07.861288 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:07.861146    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-vdrfs" podUID="1b70a87b-76a6-40e6-9ced-037c3ceee5cf" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 20:37:08.747309 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:08.747262    5570 generic.go:356] "Generic (PLEG): container finished" podID="0d5c0fd7-110a-414e-a0cc-acb706ed119b" containerID="68afaae48b42fbeeb53562b5cbe83c10698a53b81cb7c101633a459964020902" exitCode=0
Nov 03 20:37:08.747718 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:08.747346    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-503c4e-vjmmp" event={"ID":"0d5c0fd7-110a-414e-a0cc-acb706ed119b","Type":"ContainerDied","Data":"68afaae48b42fbeeb53562b5cbe83c10698a53b81cb7c101633a459964020902"}
Nov 03 20:37:09.162892 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:09.162759    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1d159a-5c484658dc-8qvxq" podUID="1ed630bf-cf60-4e37-9113-a1707b99937f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 20:37:09.757588 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:09.757549    5570 generic.go:356] "Generic (PLEG): container finished" podID="2ac2fc9c-0a3e-4326-a4f8-8856b1a30e36" containerID="ba1d33e261ae5b946cd1afd83be4e12a57d46a9a8ace20e5a4394ae6e0f6b53f" exitCode=0
Nov 03 20:37:09.758107 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:09.757656    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-2x92x" event={"ID":"2ac2fc9c-0a3e-4326-a4f8-8856b1a30e36","Type":"ContainerDied","Data":"ba1d33e261ae5b946cd1afd83be4e12a57d46a9a8ace20e5a4394ae6e0f6b53f"}
Nov 03 20:37:09.759874 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:09.759837    5570 generic.go:356] "Generic (PLEG): container finished" podID="4ae62bda-375f-4252-8606-82bae17d680d" containerID="9b29a8fc84e904bbb506a3ae0e2651fd977a35f533be777b2ac4126aad45d515" exitCode=0
Nov 03 20:37:09.760042 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:09.759932    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-crn7x" event={"ID":"4ae62bda-375f-4252-8606-82bae17d680d","Type":"ContainerDied","Data":"9b29a8fc84e904bbb506a3ae0e2651fd977a35f533be777b2ac4126aad45d515"}
Nov 03 20:37:09.762397 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:09.761996    5570 generic.go:356] "Generic (PLEG): container finished" podID="1b70a87b-76a6-40e6-9ced-037c3ceee5cf" containerID="cfc34fea0388713e474e0bfd29b896ec0c51595167214dd09881337a3d7882e6" exitCode=0
Nov 03 20:37:09.762397 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:09.762329    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-vdrfs" event={"ID":"1b70a87b-76a6-40e6-9ced-037c3ceee5cf","Type":"ContainerDied","Data":"cfc34fea0388713e474e0bfd29b896ec0c51595167214dd09881337a3d7882e6"}
Nov 03 20:37:09.913915 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:09.913836    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-2x92x"
Nov 03 20:37:09.994617 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:09.994584    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ac2fc9c-0a3e-4326-a4f8-8856b1a30e36-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ac2fc9c-0a3e-4326-a4f8-8856b1a30e36-workload-socket\") pod \"2ac2fc9c-0a3e-4326-a4f8-8856b1a30e36\" (UID: \"2ac2fc9c-0a3e-4326-a4f8-8856b1a30e36\") "
Nov 03 20:37:09.994766 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:09.994705    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ac2fc9c-0a3e-4326-a4f8-8856b1a30e36-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ac2fc9c-0a3e-4326-a4f8-8856b1a30e36-istiod-ca-cert\") pod \"2ac2fc9c-0a3e-4326-a4f8-8856b1a30e36\" (UID: \"2ac2fc9c-0a3e-4326-a4f8-8856b1a30e36\") "
Nov 03 20:37:09.994884 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:09.994864    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ac2fc9c-0a3e-4326-a4f8-8856b1a30e36-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ac2fc9c-0a3e-4326-a4f8-8856b1a30e36-credential-socket\") pod \"2ac2fc9c-0a3e-4326-a4f8-8856b1a30e36\" (UID: \"2ac2fc9c-0a3e-4326-a4f8-8856b1a30e36\") "
Nov 03 20:37:09.994968 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:09.994915    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ac2fc9c-0a3e-4326-a4f8-8856b1a30e36-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ac2fc9c-0a3e-4326-a4f8-8856b1a30e36-workload-certs\") pod \"2ac2fc9c-0a3e-4326-a4f8-8856b1a30e36\" (UID: \"2ac2fc9c-0a3e-4326-a4f8-8856b1a30e36\") "
Nov 03 20:37:09.994968 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:09.994949    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2ac2fc9c-0a3e-4326-a4f8-8856b1a30e36-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ac2fc9c-0a3e-4326-a4f8-8856b1a30e36-istio-podinfo\") pod \"2ac2fc9c-0a3e-4326-a4f8-8856b1a30e36\" (UID: \"2ac2fc9c-0a3e-4326-a4f8-8856b1a30e36\") "
Nov 03 20:37:09.994968 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:09.994950    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ac2fc9c-0a3e-4326-a4f8-8856b1a30e36-workload-socket" pod "2ac2fc9c-0a3e-4326-a4f8-8856b1a30e36" (UID: "2ac2fc9c-0a3e-4326-a4f8-8856b1a30e36"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:37:09.995103 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:09.995020    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ac2fc9c-0a3e-4326-a4f8-8856b1a30e36-istio-token\" (UniqueName: \"kubernetes.io/projected/2ac2fc9c-0a3e-4326-a4f8-8856b1a30e36-istio-token\") pod \"2ac2fc9c-0a3e-4326-a4f8-8856b1a30e36\" (UID: \"2ac2fc9c-0a3e-4326-a4f8-8856b1a30e36\") "
Nov 03 20:37:09.995103 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:09.995041    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ac2fc9c-0a3e-4326-a4f8-8856b1a30e36-kube-api-access-xp4s9\" (UniqueName: \"kubernetes.io/projected/2ac2fc9c-0a3e-4326-a4f8-8856b1a30e36-kube-api-access-xp4s9\") pod \"2ac2fc9c-0a3e-4326-a4f8-8856b1a30e36\" (UID: \"2ac2fc9c-0a3e-4326-a4f8-8856b1a30e36\") "
Nov 03 20:37:09.995103 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:09.995061    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ac2fc9c-0a3e-4326-a4f8-8856b1a30e36-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ac2fc9c-0a3e-4326-a4f8-8856b1a30e36-istio-data\") pod \"2ac2fc9c-0a3e-4326-a4f8-8856b1a30e36\" (UID: \"2ac2fc9c-0a3e-4326-a4f8-8856b1a30e36\") "
Nov 03 20:37:09.995214 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:09.995114    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ac2fc9c-0a3e-4326-a4f8-8856b1a30e36-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ac2fc9c-0a3e-4326-a4f8-8856b1a30e36-istio-envoy\") pod \"2ac2fc9c-0a3e-4326-a4f8-8856b1a30e36\" (UID: \"2ac2fc9c-0a3e-4326-a4f8-8856b1a30e36\") "
Nov 03 20:37:09.995569 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:09.995500    5570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ac2fc9c-0a3e-4326-a4f8-8856b1a30e36-workload-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:37:09.997082 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:09.997055    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ac2fc9c-0a3e-4326-a4f8-8856b1a30e36-workload-certs" pod "2ac2fc9c-0a3e-4326-a4f8-8856b1a30e36" (UID: "2ac2fc9c-0a3e-4326-a4f8-8856b1a30e36"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:37:09.997319 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:09.997298    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ac2fc9c-0a3e-4326-a4f8-8856b1a30e36-istiod-ca-cert" pod "2ac2fc9c-0a3e-4326-a4f8-8856b1a30e36" (UID: "2ac2fc9c-0a3e-4326-a4f8-8856b1a30e36"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:37:09.998745 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:09.998715    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ac2fc9c-0a3e-4326-a4f8-8856b1a30e36-credential-socket" pod "2ac2fc9c-0a3e-4326-a4f8-8856b1a30e36" (UID: "2ac2fc9c-0a3e-4326-a4f8-8856b1a30e36"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:37:09.999340 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:09.999310    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ac2fc9c-0a3e-4326-a4f8-8856b1a30e36-istio-data" pod "2ac2fc9c-0a3e-4326-a4f8-8856b1a30e36" (UID: "2ac2fc9c-0a3e-4326-a4f8-8856b1a30e36"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:37:10.000498 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:10.000471    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ac2fc9c-0a3e-4326-a4f8-8856b1a30e36-istio-token" pod "2ac2fc9c-0a3e-4326-a4f8-8856b1a30e36" (UID: "2ac2fc9c-0a3e-4326-a4f8-8856b1a30e36"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:37:10.001433 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:10.001410    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ac2fc9c-0a3e-4326-a4f8-8856b1a30e36-kube-api-access-xp4s9" pod "2ac2fc9c-0a3e-4326-a4f8-8856b1a30e36" (UID: "2ac2fc9c-0a3e-4326-a4f8-8856b1a30e36"). InnerVolumeSpecName "kube-api-access-xp4s9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:37:10.002263 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:10.002244    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ac2fc9c-0a3e-4326-a4f8-8856b1a30e36-istio-envoy" pod "2ac2fc9c-0a3e-4326-a4f8-8856b1a30e36" (UID: "2ac2fc9c-0a3e-4326-a4f8-8856b1a30e36"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:37:10.002549 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:10.002534    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2ac2fc9c-0a3e-4326-a4f8-8856b1a30e36-istio-podinfo" pod "2ac2fc9c-0a3e-4326-a4f8-8856b1a30e36" (UID: "2ac2fc9c-0a3e-4326-a4f8-8856b1a30e36"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:37:10.037351 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:10.037316    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-503c4e-vjmmp"
Nov 03 20:37:10.039395 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:10.039367    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-vdrfs"
Nov 03 20:37:10.043129 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:10.043109    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-crn7x"
Nov 03 20:37:10.103938 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:10.103743    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d5c0fd7-110a-414e-a0cc-acb706ed119b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d5c0fd7-110a-414e-a0cc-acb706ed119b-maas-api-openshift-ca\") pod \"0d5c0fd7-110a-414e-a0cc-acb706ed119b\" (UID: \"0d5c0fd7-110a-414e-a0cc-acb706ed119b\") "
Nov 03 20:37:10.103938 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:10.103828    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b70a87b-76a6-40e6-9ced-037c3ceee5cf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b70a87b-76a6-40e6-9ced-037c3ceee5cf-credential-socket\") pod \"1b70a87b-76a6-40e6-9ced-037c3ceee5cf\" (UID: \"1b70a87b-76a6-40e6-9ced-037c3ceee5cf\") "
Nov 03 20:37:10.103938 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:10.103879    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b70a87b-76a6-40e6-9ced-037c3ceee5cf-istio-token\" (UniqueName: \"kubernetes.io/projected/1b70a87b-76a6-40e6-9ced-037c3ceee5cf-istio-token\") pod \"1b70a87b-76a6-40e6-9ced-037c3ceee5cf\" (UID: \"1b70a87b-76a6-40e6-9ced-037c3ceee5cf\") "
Nov 03 20:37:10.104273 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:10.103977    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ae62bda-375f-4252-8606-82bae17d680d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ae62bda-375f-4252-8606-82bae17d680d-istio-envoy\") pod \"4ae62bda-375f-4252-8606-82bae17d680d\" (UID: \"4ae62bda-375f-4252-8606-82bae17d680d\") "
Nov 03 20:37:10.104273 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:10.104008    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1b70a87b-76a6-40e6-9ced-037c3ceee5cf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b70a87b-76a6-40e6-9ced-037c3ceee5cf-istio-podinfo\") pod \"1b70a87b-76a6-40e6-9ced-037c3ceee5cf\" (UID: \"1b70a87b-76a6-40e6-9ced-037c3ceee5cf\") "
Nov 03 20:37:10.104273 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:10.104048    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ae62bda-375f-4252-8606-82bae17d680d-istio-token\" (UniqueName: \"kubernetes.io/projected/4ae62bda-375f-4252-8606-82bae17d680d-istio-token\") pod \"4ae62bda-375f-4252-8606-82bae17d680d\" (UID: \"4ae62bda-375f-4252-8606-82bae17d680d\") "
Nov 03 20:37:10.104273 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:10.104092    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ae62bda-375f-4252-8606-82bae17d680d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ae62bda-375f-4252-8606-82bae17d680d-workload-socket\") pod \"4ae62bda-375f-4252-8606-82bae17d680d\" (UID: \"4ae62bda-375f-4252-8606-82bae17d680d\") "
Nov 03 20:37:10.104273 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:10.104144    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ae62bda-375f-4252-8606-82bae17d680d-kube-api-access-s2vn4\" (UniqueName: \"kubernetes.io/projected/4ae62bda-375f-4252-8606-82bae17d680d-kube-api-access-s2vn4\") pod \"4ae62bda-375f-4252-8606-82bae17d680d\" (UID: \"4ae62bda-375f-4252-8606-82bae17d680d\") "
Nov 03 20:37:10.104273 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:10.104174    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b70a87b-76a6-40e6-9ced-037c3ceee5cf-kube-api-access-pjs6p\" (UniqueName: \"kubernetes.io/projected/1b70a87b-76a6-40e6-9ced-037c3ceee5cf-kube-api-access-pjs6p\") pod \"1b70a87b-76a6-40e6-9ced-037c3ceee5cf\" (UID: \"1b70a87b-76a6-40e6-9ced-037c3ceee5cf\") "
Nov 03 20:37:10.104273 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:10.104203    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4ae62bda-375f-4252-8606-82bae17d680d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ae62bda-375f-4252-8606-82bae17d680d-istio-podinfo\") pod \"4ae62bda-375f-4252-8606-82bae17d680d\" (UID: \"4ae62bda-375f-4252-8606-82bae17d680d\") "
Nov 03 20:37:10.104273 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:10.104241    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b70a87b-76a6-40e6-9ced-037c3ceee5cf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b70a87b-76a6-40e6-9ced-037c3ceee5cf-istiod-ca-cert\") pod \"1b70a87b-76a6-40e6-9ced-037c3ceee5cf\" (UID: \"1b70a87b-76a6-40e6-9ced-037c3ceee5cf\") "
Nov 03 20:37:10.104686 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:10.104287    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ae62bda-375f-4252-8606-82bae17d680d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ae62bda-375f-4252-8606-82bae17d680d-istio-data\") pod \"4ae62bda-375f-4252-8606-82bae17d680d\" (UID: \"4ae62bda-375f-4252-8606-82bae17d680d\") "
Nov 03 20:37:10.104686 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:10.104329    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d5c0fd7-110a-414e-a0cc-acb706ed119b-maas-api-openshift-ca" pod "0d5c0fd7-110a-414e-a0cc-acb706ed119b" (UID: "0d5c0fd7-110a-414e-a0cc-acb706ed119b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:37:10.104686 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:10.104341    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ae62bda-375f-4252-8606-82bae17d680d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ae62bda-375f-4252-8606-82bae17d680d-credential-socket\") pod \"4ae62bda-375f-4252-8606-82bae17d680d\" (UID: \"4ae62bda-375f-4252-8606-82bae17d680d\") "
Nov 03 20:37:10.104686 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:10.104434    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d5c0fd7-110a-414e-a0cc-acb706ed119b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d5c0fd7-110a-414e-a0cc-acb706ed119b-maas-api-xks-ca\") pod \"0d5c0fd7-110a-414e-a0cc-acb706ed119b\" (UID: \"0d5c0fd7-110a-414e-a0cc-acb706ed119b\") "
Nov 03 20:37:10.104686 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:10.104470    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b70a87b-76a6-40e6-9ced-037c3ceee5cf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b70a87b-76a6-40e6-9ced-037c3ceee5cf-workload-socket\") pod \"1b70a87b-76a6-40e6-9ced-037c3ceee5cf\" (UID: \"1b70a87b-76a6-40e6-9ced-037c3ceee5cf\") "
Nov 03 20:37:10.104686 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:10.104506    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d5c0fd7-110a-414e-a0cc-acb706ed119b-kube-api-access-ttxws\" (UniqueName: \"kubernetes.io/projected/0d5c0fd7-110a-414e-a0cc-acb706ed119b-kube-api-access-ttxws\") pod \"0d5c0fd7-110a-414e-a0cc-acb706ed119b\" (UID: \"0d5c0fd7-110a-414e-a0cc-acb706ed119b\") "
Nov 03 20:37:10.105013 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:10.104736    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b70a87b-76a6-40e6-9ced-037c3ceee5cf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b70a87b-76a6-40e6-9ced-037c3ceee5cf-workload-certs\") pod \"1b70a87b-76a6-40e6-9ced-037c3ceee5cf\" (UID: \"1b70a87b-76a6-40e6-9ced-037c3ceee5cf\") "
Nov 03 20:37:10.105013 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:10.104789    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b70a87b-76a6-40e6-9ced-037c3ceee5cf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b70a87b-76a6-40e6-9ced-037c3ceee5cf-istio-envoy\") pod \"1b70a87b-76a6-40e6-9ced-037c3ceee5cf\" (UID: \"1b70a87b-76a6-40e6-9ced-037c3ceee5cf\") "
Nov 03 20:37:10.105013 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:10.104817    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b70a87b-76a6-40e6-9ced-037c3ceee5cf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b70a87b-76a6-40e6-9ced-037c3ceee5cf-istio-data\") pod \"1b70a87b-76a6-40e6-9ced-037c3ceee5cf\" (UID: \"1b70a87b-76a6-40e6-9ced-037c3ceee5cf\") "
Nov 03 20:37:10.105013 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:10.104855    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ae62bda-375f-4252-8606-82bae17d680d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ae62bda-375f-4252-8606-82bae17d680d-workload-certs\") pod \"4ae62bda-375f-4252-8606-82bae17d680d\" (UID: \"4ae62bda-375f-4252-8606-82bae17d680d\") "
Nov 03 20:37:10.105013 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:10.104935    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ae62bda-375f-4252-8606-82bae17d680d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ae62bda-375f-4252-8606-82bae17d680d-istiod-ca-cert\") pod \"4ae62bda-375f-4252-8606-82bae17d680d\" (UID: \"4ae62bda-375f-4252-8606-82bae17d680d\") "
Nov 03 20:37:10.105304 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:10.105055    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ae62bda-375f-4252-8606-82bae17d680d-credential-socket" pod "4ae62bda-375f-4252-8606-82bae17d680d" (UID: "4ae62bda-375f-4252-8606-82bae17d680d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:37:10.105440 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:10.105419    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b70a87b-76a6-40e6-9ced-037c3ceee5cf-istio-data" pod "1b70a87b-76a6-40e6-9ced-037c3ceee5cf" (UID: "1b70a87b-76a6-40e6-9ced-037c3ceee5cf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:37:10.106675 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:10.105694    5570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ac2fc9c-0a3e-4326-a4f8-8856b1a30e36-credential-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:37:10.106675 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:10.105702    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b70a87b-76a6-40e6-9ced-037c3ceee5cf-workload-certs" pod "1b70a87b-76a6-40e6-9ced-037c3ceee5cf" (UID: "1b70a87b-76a6-40e6-9ced-037c3ceee5cf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:37:10.106675 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:10.105720    5570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ac2fc9c-0a3e-4326-a4f8-8856b1a30e36-workload-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:37:10.106675 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:10.105824    5570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ac2fc9c-0a3e-4326-a4f8-8856b1a30e36-istio-podinfo\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:37:10.106675 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:10.105839    5570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ac2fc9c-0a3e-4326-a4f8-8856b1a30e36-istio-token\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:37:10.106675 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:10.105852    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xp4s9\" (UniqueName: \"kubernetes.io/projected/2ac2fc9c-0a3e-4326-a4f8-8856b1a30e36-kube-api-access-xp4s9\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:37:10.106675 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:10.105867    5570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ac2fc9c-0a3e-4326-a4f8-8856b1a30e36-istio-data\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:37:10.106675 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:10.105881    5570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ae62bda-375f-4252-8606-82bae17d680d-credential-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:37:10.106675 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:10.105892    5570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ac2fc9c-0a3e-4326-a4f8-8856b1a30e36-istio-envoy\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:37:10.106675 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:10.105905    5570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b70a87b-76a6-40e6-9ced-037c3ceee5cf-istio-data\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:37:10.106675 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:10.105917    5570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ac2fc9c-0a3e-4326-a4f8-8856b1a30e36-istiod-ca-cert\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:37:10.106675 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:10.105924    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b70a87b-76a6-40e6-9ced-037c3ceee5cf-credential-socket" pod "1b70a87b-76a6-40e6-9ced-037c3ceee5cf" (UID: "1b70a87b-76a6-40e6-9ced-037c3ceee5cf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:37:10.106675 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:10.105993    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b70a87b-76a6-40e6-9ced-037c3ceee5cf-workload-socket" pod "1b70a87b-76a6-40e6-9ced-037c3ceee5cf" (UID: "1b70a87b-76a6-40e6-9ced-037c3ceee5cf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:37:10.106675 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:10.105930    5570 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d5c0fd7-110a-414e-a0cc-acb706ed119b-maas-api-openshift-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:37:10.107562 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:10.107539    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ae62bda-375f-4252-8606-82bae17d680d-workload-certs" pod "4ae62bda-375f-4252-8606-82bae17d680d" (UID: "4ae62bda-375f-4252-8606-82bae17d680d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:37:10.108210 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:10.108173    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ae62bda-375f-4252-8606-82bae17d680d-workload-socket" pod "4ae62bda-375f-4252-8606-82bae17d680d" (UID: "4ae62bda-375f-4252-8606-82bae17d680d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:37:10.108678 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:10.108655    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ae62bda-375f-4252-8606-82bae17d680d-istio-data" pod "4ae62bda-375f-4252-8606-82bae17d680d" (UID: "4ae62bda-375f-4252-8606-82bae17d680d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:37:10.109414 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:10.109370    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b70a87b-76a6-40e6-9ced-037c3ceee5cf-kube-api-access-pjs6p" pod "1b70a87b-76a6-40e6-9ced-037c3ceee5cf" (UID: "1b70a87b-76a6-40e6-9ced-037c3ceee5cf"). InnerVolumeSpecName "kube-api-access-pjs6p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:37:10.109928 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:10.109694    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ae62bda-375f-4252-8606-82bae17d680d-istiod-ca-cert" pod "4ae62bda-375f-4252-8606-82bae17d680d" (UID: "4ae62bda-375f-4252-8606-82bae17d680d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:37:10.110270 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:10.110230    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b70a87b-76a6-40e6-9ced-037c3ceee5cf-istiod-ca-cert" pod "1b70a87b-76a6-40e6-9ced-037c3ceee5cf" (UID: "1b70a87b-76a6-40e6-9ced-037c3ceee5cf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:37:10.110535 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:10.110492    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4ae62bda-375f-4252-8606-82bae17d680d-istio-podinfo" pod "4ae62bda-375f-4252-8606-82bae17d680d" (UID: "4ae62bda-375f-4252-8606-82bae17d680d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:37:10.110887 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:10.110861    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b70a87b-76a6-40e6-9ced-037c3ceee5cf-istio-token" pod "1b70a87b-76a6-40e6-9ced-037c3ceee5cf" (UID: "1b70a87b-76a6-40e6-9ced-037c3ceee5cf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:37:10.112069 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:10.112041    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ae62bda-375f-4252-8606-82bae17d680d-kube-api-access-s2vn4" pod "4ae62bda-375f-4252-8606-82bae17d680d" (UID: "4ae62bda-375f-4252-8606-82bae17d680d"). InnerVolumeSpecName "kube-api-access-s2vn4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:37:10.116608 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:10.116585    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ae62bda-375f-4252-8606-82bae17d680d-istio-envoy" pod "4ae62bda-375f-4252-8606-82bae17d680d" (UID: "4ae62bda-375f-4252-8606-82bae17d680d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:37:10.116710 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:10.116614    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d5c0fd7-110a-414e-a0cc-acb706ed119b-kube-api-access-ttxws" pod "0d5c0fd7-110a-414e-a0cc-acb706ed119b" (UID: "0d5c0fd7-110a-414e-a0cc-acb706ed119b"). InnerVolumeSpecName "kube-api-access-ttxws". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:37:10.116710 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:10.116614    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1b70a87b-76a6-40e6-9ced-037c3ceee5cf-istio-podinfo" pod "1b70a87b-76a6-40e6-9ced-037c3ceee5cf" (UID: "1b70a87b-76a6-40e6-9ced-037c3ceee5cf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:37:10.116710 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:10.116666    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b70a87b-76a6-40e6-9ced-037c3ceee5cf-istio-envoy" pod "1b70a87b-76a6-40e6-9ced-037c3ceee5cf" (UID: "1b70a87b-76a6-40e6-9ced-037c3ceee5cf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:37:10.116710 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:10.116687    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d5c0fd7-110a-414e-a0cc-acb706ed119b-maas-api-xks-ca" pod "0d5c0fd7-110a-414e-a0cc-acb706ed119b" (UID: "0d5c0fd7-110a-414e-a0cc-acb706ed119b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:37:10.117700 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:10.117680    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ae62bda-375f-4252-8606-82bae17d680d-istio-token" pod "4ae62bda-375f-4252-8606-82bae17d680d" (UID: "4ae62bda-375f-4252-8606-82bae17d680d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:37:10.207970 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:10.207924    5570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b70a87b-76a6-40e6-9ced-037c3ceee5cf-istiod-ca-cert\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:37:10.207970 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:10.207971    5570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ae62bda-375f-4252-8606-82bae17d680d-istio-data\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:37:10.208203 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:10.207984    5570 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d5c0fd7-110a-414e-a0cc-acb706ed119b-maas-api-xks-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:37:10.208203 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:10.207997    5570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b70a87b-76a6-40e6-9ced-037c3ceee5cf-workload-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:37:10.208203 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:10.208013    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ttxws\" (UniqueName: \"kubernetes.io/projected/0d5c0fd7-110a-414e-a0cc-acb706ed119b-kube-api-access-ttxws\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:37:10.208203 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:10.208028    5570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b70a87b-76a6-40e6-9ced-037c3ceee5cf-workload-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:37:10.208203 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:10.208041    5570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b70a87b-76a6-40e6-9ced-037c3ceee5cf-istio-envoy\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:37:10.208203 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:10.208053    5570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ae62bda-375f-4252-8606-82bae17d680d-workload-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:37:10.208203 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:10.208065    5570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ae62bda-375f-4252-8606-82bae17d680d-istiod-ca-cert\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:37:10.208203 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:10.208077    5570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b70a87b-76a6-40e6-9ced-037c3ceee5cf-credential-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:37:10.208203 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:10.208090    5570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b70a87b-76a6-40e6-9ced-037c3ceee5cf-istio-token\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:37:10.208203 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:10.208103    5570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ae62bda-375f-4252-8606-82bae17d680d-istio-envoy\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:37:10.208203 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:10.208115    5570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b70a87b-76a6-40e6-9ced-037c3ceee5cf-istio-podinfo\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:37:10.208203 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:10.208127    5570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ae62bda-375f-4252-8606-82bae17d680d-istio-token\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:37:10.208203 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:10.208139    5570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ae62bda-375f-4252-8606-82bae17d680d-workload-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:37:10.208203 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:10.208151    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2vn4\" (UniqueName: \"kubernetes.io/projected/4ae62bda-375f-4252-8606-82bae17d680d-kube-api-access-s2vn4\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:37:10.208203 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:10.208164    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjs6p\" (UniqueName: \"kubernetes.io/projected/1b70a87b-76a6-40e6-9ced-037c3ceee5cf-kube-api-access-pjs6p\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:37:10.208203 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:10.208177    5570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ae62bda-375f-4252-8606-82bae17d680d-istio-podinfo\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:37:10.770836 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:10.770806    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-crn7x"
Nov 03 20:37:10.771379 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:10.770804    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-crn7x" event={"ID":"4ae62bda-375f-4252-8606-82bae17d680d","Type":"ContainerDied","Data":"3a83f201afce537b6663d5a6804d57aaaa7fd4cde8ab154c3f9e0a65ead2725c"}
Nov 03 20:37:10.771379 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:10.770997    5570 scope.go:122] "RemoveContainer" containerID="9b29a8fc84e904bbb506a3ae0e2651fd977a35f533be777b2ac4126aad45d515"
Nov 03 20:37:10.772615 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:10.772545    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-vdrfs" event={"ID":"1b70a87b-76a6-40e6-9ced-037c3ceee5cf","Type":"ContainerDied","Data":"17b2caebb6f0f8981effc65e9fc788c3d109bad836b965938e22b17da918cf61"}
Nov 03 20:37:10.772703 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:10.772584    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-vdrfs"
Nov 03 20:37:10.774422 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:10.774392    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-503c4e-vjmmp"
Nov 03 20:37:10.774565 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:10.774422    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-503c4e-vjmmp" event={"ID":"0d5c0fd7-110a-414e-a0cc-acb706ed119b","Type":"ContainerDied","Data":"c44429429ff4ec12c47ce3395d60ea2f82c5405a3f7c252fe6970c7560bfa265"}
Nov 03 20:37:10.774565 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:10.774479    5570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c44429429ff4ec12c47ce3395d60ea2f82c5405a3f7c252fe6970c7560bfa265"
Nov 03 20:37:10.776332 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:10.776295    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-2x92x" event={"ID":"2ac2fc9c-0a3e-4326-a4f8-8856b1a30e36","Type":"ContainerDied","Data":"4349d6d8cd982d01734cc03ceb2e0b234d18b47689e6e8c4bdabbbce5bdbd420"}
Nov 03 20:37:10.776439 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:10.776340    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-2x92x"
Nov 03 20:37:10.788274 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:10.788251    5570 scope.go:122] "RemoveContainer" containerID="cfc34fea0388713e474e0bfd29b896ec0c51595167214dd09881337a3d7882e6"
Nov 03 20:37:10.798426 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:10.798404    5570 scope.go:122] "RemoveContainer" containerID="ba1d33e261ae5b946cd1afd83be4e12a57d46a9a8ace20e5a4394ae6e0f6b53f"
Nov 03 20:37:10.826436 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:10.826401    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-crn7x"]
Nov 03 20:37:10.830679 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:10.830646    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-crn7x"]
Nov 03 20:37:10.898579 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:10.898542    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-vdrfs"]
Nov 03 20:37:10.901679 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:10.901650    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-vdrfs"]
Nov 03 20:37:10.954288 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:10.954250    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-2x92x"]
Nov 03 20:37:10.957682 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:10.957608    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-2x92x"]
Nov 03 20:37:11.274455 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:11.274399    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-1d159a-openshift-default-799c8f9695-b8h67"
Nov 03 20:37:11.421184 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:11.420928    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40e4fc9c-c3c1-4aa0-98ad-eda347be89c5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40e4fc9c-c3c1-4aa0-98ad-eda347be89c5-istiod-ca-cert\") pod \"40e4fc9c-c3c1-4aa0-98ad-eda347be89c5\" (UID: \"40e4fc9c-c3c1-4aa0-98ad-eda347be89c5\") "
Nov 03 20:37:11.421533 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:11.421509    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40e4fc9c-c3c1-4aa0-98ad-eda347be89c5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40e4fc9c-c3c1-4aa0-98ad-eda347be89c5-istio-envoy\") pod \"40e4fc9c-c3c1-4aa0-98ad-eda347be89c5\" (UID: \"40e4fc9c-c3c1-4aa0-98ad-eda347be89c5\") "
Nov 03 20:37:11.421705 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:11.421689    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40e4fc9c-c3c1-4aa0-98ad-eda347be89c5-istio-token\" (UniqueName: \"kubernetes.io/projected/40e4fc9c-c3c1-4aa0-98ad-eda347be89c5-istio-token\") pod \"40e4fc9c-c3c1-4aa0-98ad-eda347be89c5\" (UID: \"40e4fc9c-c3c1-4aa0-98ad-eda347be89c5\") "
Nov 03 20:37:11.421920 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:11.421898    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40e4fc9c-c3c1-4aa0-98ad-eda347be89c5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40e4fc9c-c3c1-4aa0-98ad-eda347be89c5-credential-socket\") pod \"40e4fc9c-c3c1-4aa0-98ad-eda347be89c5\" (UID: \"40e4fc9c-c3c1-4aa0-98ad-eda347be89c5\") "
Nov 03 20:37:11.422021 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:11.421966    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40e4fc9c-c3c1-4aa0-98ad-eda347be89c5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40e4fc9c-c3c1-4aa0-98ad-eda347be89c5-istio-data\") pod \"40e4fc9c-c3c1-4aa0-98ad-eda347be89c5\" (UID: \"40e4fc9c-c3c1-4aa0-98ad-eda347be89c5\") "
Nov 03 20:37:11.422021 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:11.421999    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40e4fc9c-c3c1-4aa0-98ad-eda347be89c5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40e4fc9c-c3c1-4aa0-98ad-eda347be89c5-workload-certs\") pod \"40e4fc9c-c3c1-4aa0-98ad-eda347be89c5\" (UID: \"40e4fc9c-c3c1-4aa0-98ad-eda347be89c5\") "
Nov 03 20:37:11.422390 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:11.422031    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40e4fc9c-c3c1-4aa0-98ad-eda347be89c5-kube-api-access-k6jc5\" (UniqueName: \"kubernetes.io/projected/40e4fc9c-c3c1-4aa0-98ad-eda347be89c5-kube-api-access-k6jc5\") pod \"40e4fc9c-c3c1-4aa0-98ad-eda347be89c5\" (UID: \"40e4fc9c-c3c1-4aa0-98ad-eda347be89c5\") "
Nov 03 20:37:11.422390 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:11.422099    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/40e4fc9c-c3c1-4aa0-98ad-eda347be89c5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40e4fc9c-c3c1-4aa0-98ad-eda347be89c5-istio-podinfo\") pod \"40e4fc9c-c3c1-4aa0-98ad-eda347be89c5\" (UID: \"40e4fc9c-c3c1-4aa0-98ad-eda347be89c5\") "
Nov 03 20:37:11.422390 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:11.422227    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40e4fc9c-c3c1-4aa0-98ad-eda347be89c5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40e4fc9c-c3c1-4aa0-98ad-eda347be89c5-workload-socket\") pod \"40e4fc9c-c3c1-4aa0-98ad-eda347be89c5\" (UID: \"40e4fc9c-c3c1-4aa0-98ad-eda347be89c5\") "
Nov 03 20:37:11.423132 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:11.421566    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40e4fc9c-c3c1-4aa0-98ad-eda347be89c5-istiod-ca-cert" pod "40e4fc9c-c3c1-4aa0-98ad-eda347be89c5" (UID: "40e4fc9c-c3c1-4aa0-98ad-eda347be89c5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:37:11.423269 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:11.423107    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40e4fc9c-c3c1-4aa0-98ad-eda347be89c5-workload-socket" pod "40e4fc9c-c3c1-4aa0-98ad-eda347be89c5" (UID: "40e4fc9c-c3c1-4aa0-98ad-eda347be89c5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:37:11.423554 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:11.423535    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40e4fc9c-c3c1-4aa0-98ad-eda347be89c5-credential-socket" pod "40e4fc9c-c3c1-4aa0-98ad-eda347be89c5" (UID: "40e4fc9c-c3c1-4aa0-98ad-eda347be89c5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:37:11.423717 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:11.423698    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40e4fc9c-c3c1-4aa0-98ad-eda347be89c5-istio-data" pod "40e4fc9c-c3c1-4aa0-98ad-eda347be89c5" (UID: "40e4fc9c-c3c1-4aa0-98ad-eda347be89c5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:37:11.423845 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:11.423814    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40e4fc9c-c3c1-4aa0-98ad-eda347be89c5-workload-certs" pod "40e4fc9c-c3c1-4aa0-98ad-eda347be89c5" (UID: "40e4fc9c-c3c1-4aa0-98ad-eda347be89c5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:37:11.428281 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:11.428251    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40e4fc9c-c3c1-4aa0-98ad-eda347be89c5-istio-token" pod "40e4fc9c-c3c1-4aa0-98ad-eda347be89c5" (UID: "40e4fc9c-c3c1-4aa0-98ad-eda347be89c5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:37:11.429817 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:11.429789    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/40e4fc9c-c3c1-4aa0-98ad-eda347be89c5-istio-podinfo" pod "40e4fc9c-c3c1-4aa0-98ad-eda347be89c5" (UID: "40e4fc9c-c3c1-4aa0-98ad-eda347be89c5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:37:11.429926 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:11.429893    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40e4fc9c-c3c1-4aa0-98ad-eda347be89c5-istio-envoy" pod "40e4fc9c-c3c1-4aa0-98ad-eda347be89c5" (UID: "40e4fc9c-c3c1-4aa0-98ad-eda347be89c5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:37:11.431276 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:11.431251    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40e4fc9c-c3c1-4aa0-98ad-eda347be89c5-kube-api-access-k6jc5" pod "40e4fc9c-c3c1-4aa0-98ad-eda347be89c5" (UID: "40e4fc9c-c3c1-4aa0-98ad-eda347be89c5"). InnerVolumeSpecName "kube-api-access-k6jc5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:37:11.444204 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:11.444167    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-a9ba83-openshift-default-8c675fb59-486wc"]
Nov 03 20:37:11.444611 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:11.444435    5570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-a9ba83-openshift-default-8c675fb59-486wc" podUID="f913fe36-7256-4171-ad6f-db25850da8ce" containerName="istio-proxy" containerID="cri-o://8f879707251cdde1af594cce61f974e2b60d0b8a7e3f888f52f804d150f436ac" gracePeriod=30
Nov 03 20:37:11.524466 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:11.524377    5570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40e4fc9c-c3c1-4aa0-98ad-eda347be89c5-istio-token\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:37:11.524466 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:11.524411    5570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40e4fc9c-c3c1-4aa0-98ad-eda347be89c5-credential-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:37:11.524466 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:11.524422    5570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40e4fc9c-c3c1-4aa0-98ad-eda347be89c5-istio-data\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:37:11.524466 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:11.524430    5570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40e4fc9c-c3c1-4aa0-98ad-eda347be89c5-workload-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:37:11.524466 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:11.524438    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6jc5\" (UniqueName: \"kubernetes.io/projected/40e4fc9c-c3c1-4aa0-98ad-eda347be89c5-kube-api-access-k6jc5\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:37:11.524466 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:11.524446    5570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40e4fc9c-c3c1-4aa0-98ad-eda347be89c5-istio-podinfo\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:37:11.524466 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:11.524454    5570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40e4fc9c-c3c1-4aa0-98ad-eda347be89c5-workload-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:37:11.524466 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:11.524464    5570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40e4fc9c-c3c1-4aa0-98ad-eda347be89c5-istiod-ca-cert\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:37:11.524466 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:11.524472    5570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40e4fc9c-c3c1-4aa0-98ad-eda347be89c5-istio-envoy\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:37:11.785016 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:11.784919    5570 generic.go:356] "Generic (PLEG): container finished" podID="40e4fc9c-c3c1-4aa0-98ad-eda347be89c5" containerID="d200d60ddf9d712ccd5b388bb3ab8a745c07816a59a52cc300d0254e93a41a89" exitCode=0
Nov 03 20:37:11.785016 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:11.784982    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-1d159a-openshift-default-799c8f9695-b8h67"
Nov 03 20:37:11.785016 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:11.785003    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-1d159a-openshift-default-799c8f9695-b8h67" event={"ID":"40e4fc9c-c3c1-4aa0-98ad-eda347be89c5","Type":"ContainerDied","Data":"d200d60ddf9d712ccd5b388bb3ab8a745c07816a59a52cc300d0254e93a41a89"}
Nov 03 20:37:11.785627 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:11.785059    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-1d159a-openshift-default-799c8f9695-b8h67" event={"ID":"40e4fc9c-c3c1-4aa0-98ad-eda347be89c5","Type":"ContainerDied","Data":"3861708181af8794b164b11647d15caf660b93d8c3f7983ec586b721c19d01f0"}
Nov 03 20:37:11.785627 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:11.785090    5570 scope.go:122] "RemoveContainer" containerID="d200d60ddf9d712ccd5b388bb3ab8a745c07816a59a52cc300d0254e93a41a89"
Nov 03 20:37:11.797117 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:11.797083    5570 scope.go:122] "RemoveContainer" containerID="e91fa2efdc7c6fd5b80b7258d1bd1d371bd0e89b53a388019b215aa898711527"
Nov 03 20:37:11.806632 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:11.806610    5570 scope.go:122] "RemoveContainer" containerID="d200d60ddf9d712ccd5b388bb3ab8a745c07816a59a52cc300d0254e93a41a89"
Nov 03 20:37:11.806943 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:37:11.806922    5570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d200d60ddf9d712ccd5b388bb3ab8a745c07816a59a52cc300d0254e93a41a89\": container with ID starting with d200d60ddf9d712ccd5b388bb3ab8a745c07816a59a52cc300d0254e93a41a89 not found: ID does not exist" containerID="d200d60ddf9d712ccd5b388bb3ab8a745c07816a59a52cc300d0254e93a41a89"
Nov 03 20:37:11.807007 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:11.806953    5570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d200d60ddf9d712ccd5b388bb3ab8a745c07816a59a52cc300d0254e93a41a89"} err="failed to get container status \"d200d60ddf9d712ccd5b388bb3ab8a745c07816a59a52cc300d0254e93a41a89\": rpc error: code = NotFound desc = could not find container \"d200d60ddf9d712ccd5b388bb3ab8a745c07816a59a52cc300d0254e93a41a89\": container with ID starting with d200d60ddf9d712ccd5b388bb3ab8a745c07816a59a52cc300d0254e93a41a89 not found: ID does not exist"
Nov 03 20:37:11.807007 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:11.806971    5570 scope.go:122] "RemoveContainer" containerID="e91fa2efdc7c6fd5b80b7258d1bd1d371bd0e89b53a388019b215aa898711527"
Nov 03 20:37:11.807216 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:37:11.807192    5570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e91fa2efdc7c6fd5b80b7258d1bd1d371bd0e89b53a388019b215aa898711527\": container with ID starting with e91fa2efdc7c6fd5b80b7258d1bd1d371bd0e89b53a388019b215aa898711527 not found: ID does not exist" containerID="e91fa2efdc7c6fd5b80b7258d1bd1d371bd0e89b53a388019b215aa898711527"
Nov 03 20:37:11.807289 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:11.807225    5570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e91fa2efdc7c6fd5b80b7258d1bd1d371bd0e89b53a388019b215aa898711527"} err="failed to get container status \"e91fa2efdc7c6fd5b80b7258d1bd1d371bd0e89b53a388019b215aa898711527\": rpc error: code = NotFound desc = could not find container \"e91fa2efdc7c6fd5b80b7258d1bd1d371bd0e89b53a388019b215aa898711527\": container with ID starting with e91fa2efdc7c6fd5b80b7258d1bd1d371bd0e89b53a388019b215aa898711527 not found: ID does not exist"
Nov 03 20:37:11.839675 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:11.839640    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-1d159a-openshift-default-799c8f9695-b8h67"]
Nov 03 20:37:11.846798 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:11.846748    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-1d159a-openshift-default-799c8f9695-b8h67"]
Nov 03 20:37:11.981534 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:11.981499    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b70a87b-76a6-40e6-9ced-037c3ceee5cf" path="/var/lib/kubelet/pods/1b70a87b-76a6-40e6-9ced-037c3ceee5cf/volumes"
Nov 03 20:37:11.982532 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:11.982512    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ac2fc9c-0a3e-4326-a4f8-8856b1a30e36" path="/var/lib/kubelet/pods/2ac2fc9c-0a3e-4326-a4f8-8856b1a30e36/volumes"
Nov 03 20:37:11.984400 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:11.984380    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40e4fc9c-c3c1-4aa0-98ad-eda347be89c5" path="/var/lib/kubelet/pods/40e4fc9c-c3c1-4aa0-98ad-eda347be89c5/volumes"
Nov 03 20:37:11.985347 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:11.985332    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ae62bda-375f-4252-8606-82bae17d680d" path="/var/lib/kubelet/pods/4ae62bda-375f-4252-8606-82bae17d680d/volumes"
Nov 03 20:37:12.009643 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:12.009609    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4b95f3-ckdlh"]
Nov 03 20:37:12.010163 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:12.010147    5570 cpu_manager.go:414] "containerMap: removing container" podUID="40e4fc9c-c3c1-4aa0-98ad-eda347be89c5" containerName="istio-proxy"
Nov 03 20:37:12.010228 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:12.010164    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40e4fc9c-c3c1-4aa0-98ad-eda347be89c5" containerName="istio-proxy"
Nov 03 20:37:12.010228 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:12.010182    5570 cpu_manager.go:414] "containerMap: removing container" podUID="2ac2fc9c-0a3e-4326-a4f8-8856b1a30e36" containerName="istio-proxy"
Nov 03 20:37:12.010228 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:12.010187    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ac2fc9c-0a3e-4326-a4f8-8856b1a30e36" containerName="istio-proxy"
Nov 03 20:37:12.010228 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:12.010198    5570 cpu_manager.go:414] "containerMap: removing container" podUID="1b70a87b-76a6-40e6-9ced-037c3ceee5cf" containerName="istio-proxy"
Nov 03 20:37:12.010228 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:12.010203    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b70a87b-76a6-40e6-9ced-037c3ceee5cf" containerName="istio-proxy"
Nov 03 20:37:12.010228 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:12.010217    5570 cpu_manager.go:414] "containerMap: removing container" podUID="4ae62bda-375f-4252-8606-82bae17d680d" containerName="istio-proxy"
Nov 03 20:37:12.010228 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:12.010222    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ae62bda-375f-4252-8606-82bae17d680d" containerName="istio-proxy"
Nov 03 20:37:12.010476 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:12.010238    5570 cpu_manager.go:414] "containerMap: removing container" podUID="0d5c0fd7-110a-414e-a0cc-acb706ed119b" containerName="revoke-keys"
Nov 03 20:37:12.010476 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:12.010243    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d5c0fd7-110a-414e-a0cc-acb706ed119b" containerName="revoke-keys"
Nov 03 20:37:12.010476 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:12.010257    5570 cpu_manager.go:414] "containerMap: removing container" podUID="c6fb7442-714a-4e4a-a6cf-19326b50a929" containerName="istio-proxy"
Nov 03 20:37:12.010476 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:12.010262    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6fb7442-714a-4e4a-a6cf-19326b50a929" containerName="istio-proxy"
Nov 03 20:37:12.010476 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:12.010328    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d5c0fd7-110a-414e-a0cc-acb706ed119b" containerName="revoke-keys"
Nov 03 20:37:12.010476 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:12.010340    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="40e4fc9c-c3c1-4aa0-98ad-eda347be89c5" containerName="istio-proxy"
Nov 03 20:37:12.010476 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:12.010346    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="40e4fc9c-c3c1-4aa0-98ad-eda347be89c5" containerName="istio-proxy"
Nov 03 20:37:12.010476 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:12.010354    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ac2fc9c-0a3e-4326-a4f8-8856b1a30e36" containerName="istio-proxy"
Nov 03 20:37:12.010476 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:12.010360    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b70a87b-76a6-40e6-9ced-037c3ceee5cf" containerName="istio-proxy"
Nov 03 20:37:12.010476 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:12.010365    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ae62bda-375f-4252-8606-82bae17d680d" containerName="istio-proxy"
Nov 03 20:37:12.015460 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:12.015442    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4b95f3-ckdlh"
Nov 03 20:37:12.030101 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:12.030067    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4b95f3-ckdlh"]
Nov 03 20:37:12.124916 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:12.124755    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:37:12.124916 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:12.124858    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l7s5w\""
Nov 03 20:37:12.131422 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:12.131389    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6f7vf\" (UniqueName: \"kubernetes.io/projected/3facaeda-5192-4f41-ab9f-ae752e78dd60-kube-api-access-6f7vf\") pod \"maas-api-revoke-keys-e2e-inf-b-4b95f3-ckdlh\" (UID: \"3facaeda-5192-4f41-ab9f-ae752e78dd60\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4b95f3-ckdlh"
Nov 03 20:37:12.131422 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:12.131434    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3facaeda-5192-4f41-ab9f-ae752e78dd60-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-4b95f3-ckdlh\" (UID: \"3facaeda-5192-4f41-ab9f-ae752e78dd60\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4b95f3-ckdlh"
Nov 03 20:37:12.131675 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:12.131608    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3facaeda-5192-4f41-ab9f-ae752e78dd60-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-4b95f3-ckdlh\" (UID: \"3facaeda-5192-4f41-ab9f-ae752e78dd60\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4b95f3-ckdlh"
Nov 03 20:37:12.233230 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:12.233183    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3facaeda-5192-4f41-ab9f-ae752e78dd60-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-4b95f3-ckdlh\" (UID: \"3facaeda-5192-4f41-ab9f-ae752e78dd60\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4b95f3-ckdlh"
Nov 03 20:37:12.233414 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:12.233366    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6f7vf\" (UniqueName: \"kubernetes.io/projected/3facaeda-5192-4f41-ab9f-ae752e78dd60-kube-api-access-6f7vf\") pod \"maas-api-revoke-keys-e2e-inf-b-4b95f3-ckdlh\" (UID: \"3facaeda-5192-4f41-ab9f-ae752e78dd60\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4b95f3-ckdlh"
Nov 03 20:37:12.233463 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:12.233420    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3facaeda-5192-4f41-ab9f-ae752e78dd60-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-4b95f3-ckdlh\" (UID: \"3facaeda-5192-4f41-ab9f-ae752e78dd60\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4b95f3-ckdlh"
Nov 03 20:37:12.234134 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:12.234107    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3facaeda-5192-4f41-ab9f-ae752e78dd60-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-4b95f3-ckdlh\" (UID: \"3facaeda-5192-4f41-ab9f-ae752e78dd60\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4b95f3-ckdlh"
Nov 03 20:37:12.235683 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:12.235662    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3facaeda-5192-4f41-ab9f-ae752e78dd60-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-4b95f3-ckdlh\" (UID: \"3facaeda-5192-4f41-ab9f-ae752e78dd60\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4b95f3-ckdlh"
Nov 03 20:37:12.255323 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:12.255293    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6f7vf\" (UniqueName: \"kubernetes.io/projected/3facaeda-5192-4f41-ab9f-ae752e78dd60-kube-api-access-6f7vf\") pod \"maas-api-revoke-keys-e2e-inf-b-4b95f3-ckdlh\" (UID: \"3facaeda-5192-4f41-ab9f-ae752e78dd60\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4b95f3-ckdlh"
Nov 03 20:37:12.330404 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:12.330372    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4b95f3-ckdlh"
Nov 03 20:37:12.454457 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:12.454415    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g69f8"]
Nov 03 20:37:12.629178 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:12.629140    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4b95f3-ckdlh"]
Nov 03 20:37:12.629588 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:37:12.629561    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3facaeda_5192_4f41_ab9f_ae752e78dd60.slice/crio-d41d6aad9ae8f8507243137693bf63d1872b09d653a493f96347fcf2988c878f WatchSource:0}: Error finding container d41d6aad9ae8f8507243137693bf63d1872b09d653a493f96347fcf2988c878f: Status 404 returned error can't find the container with id d41d6aad9ae8f8507243137693bf63d1872b09d653a493f96347fcf2988c878f
Nov 03 20:37:12.795444 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:12.795402    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4b95f3-ckdlh" event={"ID":"3facaeda-5192-4f41-ab9f-ae752e78dd60","Type":"ContainerStarted","Data":"d41d6aad9ae8f8507243137693bf63d1872b09d653a493f96347fcf2988c878f"}
Nov 03 20:37:13.804075 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:13.804040    5570 generic.go:356] "Generic (PLEG): container finished" podID="3facaeda-5192-4f41-ab9f-ae752e78dd60" containerID="be059e412f1853a8c5debbfb8af9c7ba18de02b070219f07598fbd3f5bde7924" exitCode=0
Nov 03 20:37:13.804500 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:13.804105    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4b95f3-ckdlh" event={"ID":"3facaeda-5192-4f41-ab9f-ae752e78dd60","Type":"ContainerDied","Data":"be059e412f1853a8c5debbfb8af9c7ba18de02b070219f07598fbd3f5bde7924"}
Nov 03 20:37:13.976471 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:13.976426    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a9ba83-67d4dd56f9-22nhs" podUID="58fc81a1-5df4-4b6c-92bc-6dcf0836f400" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 20:37:13.976471 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:13.976460    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-a9ba83-67b98df76d-24dtv" podUID="6deae9d9-81da-4c77-a15a-d805ff547c51" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 20:37:14.965020 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:14.964993    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4b95f3-ckdlh"
Nov 03 20:37:15.069565 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:15.069467    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3facaeda-5192-4f41-ab9f-ae752e78dd60-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3facaeda-5192-4f41-ab9f-ae752e78dd60-maas-api-xks-ca\") pod \"3facaeda-5192-4f41-ab9f-ae752e78dd60\" (UID: \"3facaeda-5192-4f41-ab9f-ae752e78dd60\") "
Nov 03 20:37:15.069565 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:15.069535    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3facaeda-5192-4f41-ab9f-ae752e78dd60-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3facaeda-5192-4f41-ab9f-ae752e78dd60-maas-api-openshift-ca\") pod \"3facaeda-5192-4f41-ab9f-ae752e78dd60\" (UID: \"3facaeda-5192-4f41-ab9f-ae752e78dd60\") "
Nov 03 20:37:15.069767 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:15.069614    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3facaeda-5192-4f41-ab9f-ae752e78dd60-kube-api-access-6f7vf\" (UniqueName: \"kubernetes.io/projected/3facaeda-5192-4f41-ab9f-ae752e78dd60-kube-api-access-6f7vf\") pod \"3facaeda-5192-4f41-ab9f-ae752e78dd60\" (UID: \"3facaeda-5192-4f41-ab9f-ae752e78dd60\") "
Nov 03 20:37:15.069969 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:15.069941    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3facaeda-5192-4f41-ab9f-ae752e78dd60-maas-api-openshift-ca" pod "3facaeda-5192-4f41-ab9f-ae752e78dd60" (UID: "3facaeda-5192-4f41-ab9f-ae752e78dd60"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:37:15.070264 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:15.070237    5570 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3facaeda-5192-4f41-ab9f-ae752e78dd60-maas-api-openshift-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:37:15.071695 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:15.071674    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3facaeda-5192-4f41-ab9f-ae752e78dd60-maas-api-xks-ca" pod "3facaeda-5192-4f41-ab9f-ae752e78dd60" (UID: "3facaeda-5192-4f41-ab9f-ae752e78dd60"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:37:15.071825 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:15.071804    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3facaeda-5192-4f41-ab9f-ae752e78dd60-kube-api-access-6f7vf" pod "3facaeda-5192-4f41-ab9f-ae752e78dd60" (UID: "3facaeda-5192-4f41-ab9f-ae752e78dd60"). InnerVolumeSpecName "kube-api-access-6f7vf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:37:15.171461 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:15.171423    5570 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3facaeda-5192-4f41-ab9f-ae752e78dd60-maas-api-xks-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:37:15.171461 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:15.171453    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6f7vf\" (UniqueName: \"kubernetes.io/projected/3facaeda-5192-4f41-ab9f-ae752e78dd60-kube-api-access-6f7vf\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:37:15.818765 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:15.818730    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4b95f3-ckdlh"
Nov 03 20:37:15.818961 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:15.818767    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4b95f3-ckdlh" event={"ID":"3facaeda-5192-4f41-ab9f-ae752e78dd60","Type":"ContainerDied","Data":"d41d6aad9ae8f8507243137693bf63d1872b09d653a493f96347fcf2988c878f"}
Nov 03 20:37:15.818961 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:15.818845    5570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d41d6aad9ae8f8507243137693bf63d1872b09d653a493f96347fcf2988c878f"
Nov 03 20:37:16.723006 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:16.722938    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a9ba83-openshift-default-8c675fb59-486wc"
Nov 03 20:37:16.827394 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:16.827357    5570 generic.go:356] "Generic (PLEG): container finished" podID="f913fe36-7256-4171-ad6f-db25850da8ce" containerID="8f879707251cdde1af594cce61f974e2b60d0b8a7e3f888f52f804d150f436ac" exitCode=0
Nov 03 20:37:16.827635 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:16.827408    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a9ba83-openshift-default-8c675fb59-486wc"
Nov 03 20:37:16.827635 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:16.827410    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a9ba83-openshift-default-8c675fb59-486wc" event={"ID":"f913fe36-7256-4171-ad6f-db25850da8ce","Type":"ContainerDied","Data":"8f879707251cdde1af594cce61f974e2b60d0b8a7e3f888f52f804d150f436ac"}
Nov 03 20:37:16.827635 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:16.827532    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a9ba83-openshift-default-8c675fb59-486wc" event={"ID":"f913fe36-7256-4171-ad6f-db25850da8ce","Type":"ContainerDied","Data":"a148340f62f3f9ce63f99870772c1b10dd09ef6a238a2187a61e65d1fa1e026d"}
Nov 03 20:37:16.827635 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:16.827568    5570 scope.go:122] "RemoveContainer" containerID="8f879707251cdde1af594cce61f974e2b60d0b8a7e3f888f52f804d150f436ac"
Nov 03 20:37:16.837512 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:16.837488    5570 scope.go:122] "RemoveContainer" containerID="edeb6af257049566a198a0ed1235fc9c416060dc50a5fd85246d010da5f12ea0"
Nov 03 20:37:16.847579 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:16.847558    5570 scope.go:122] "RemoveContainer" containerID="8f879707251cdde1af594cce61f974e2b60d0b8a7e3f888f52f804d150f436ac"
Nov 03 20:37:16.847997 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:37:16.847970    5570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f879707251cdde1af594cce61f974e2b60d0b8a7e3f888f52f804d150f436ac\": container with ID starting with 8f879707251cdde1af594cce61f974e2b60d0b8a7e3f888f52f804d150f436ac not found: ID does not exist" containerID="8f879707251cdde1af594cce61f974e2b60d0b8a7e3f888f52f804d150f436ac"
Nov 03 20:37:16.848129 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:16.848010    5570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f879707251cdde1af594cce61f974e2b60d0b8a7e3f888f52f804d150f436ac"} err="failed to get container status \"8f879707251cdde1af594cce61f974e2b60d0b8a7e3f888f52f804d150f436ac\": rpc error: code = NotFound desc = could not find container \"8f879707251cdde1af594cce61f974e2b60d0b8a7e3f888f52f804d150f436ac\": container with ID starting with 8f879707251cdde1af594cce61f974e2b60d0b8a7e3f888f52f804d150f436ac not found: ID does not exist"
Nov 03 20:37:16.848129 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:16.848037    5570 scope.go:122] "RemoveContainer" containerID="edeb6af257049566a198a0ed1235fc9c416060dc50a5fd85246d010da5f12ea0"
Nov 03 20:37:16.848341 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:37:16.848322    5570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"edeb6af257049566a198a0ed1235fc9c416060dc50a5fd85246d010da5f12ea0\": container with ID starting with edeb6af257049566a198a0ed1235fc9c416060dc50a5fd85246d010da5f12ea0 not found: ID does not exist" containerID="edeb6af257049566a198a0ed1235fc9c416060dc50a5fd85246d010da5f12ea0"
Nov 03 20:37:16.848378 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:16.848349    5570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edeb6af257049566a198a0ed1235fc9c416060dc50a5fd85246d010da5f12ea0"} err="failed to get container status \"edeb6af257049566a198a0ed1235fc9c416060dc50a5fd85246d010da5f12ea0\": rpc error: code = NotFound desc = could not find container \"edeb6af257049566a198a0ed1235fc9c416060dc50a5fd85246d010da5f12ea0\": container with ID starting with edeb6af257049566a198a0ed1235fc9c416060dc50a5fd85246d010da5f12ea0 not found: ID does not exist"
Nov 03 20:37:16.891608 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:16.891571    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f913fe36-7256-4171-ad6f-db25850da8ce-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f913fe36-7256-4171-ad6f-db25850da8ce-istiod-ca-cert\") pod \"f913fe36-7256-4171-ad6f-db25850da8ce\" (UID: \"f913fe36-7256-4171-ad6f-db25850da8ce\") "
Nov 03 20:37:16.891848 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:16.891629    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f913fe36-7256-4171-ad6f-db25850da8ce-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f913fe36-7256-4171-ad6f-db25850da8ce-workload-socket\") pod \"f913fe36-7256-4171-ad6f-db25850da8ce\" (UID: \"f913fe36-7256-4171-ad6f-db25850da8ce\") "
Nov 03 20:37:16.891848 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:16.891665    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f913fe36-7256-4171-ad6f-db25850da8ce-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f913fe36-7256-4171-ad6f-db25850da8ce-workload-certs\") pod \"f913fe36-7256-4171-ad6f-db25850da8ce\" (UID: \"f913fe36-7256-4171-ad6f-db25850da8ce\") "
Nov 03 20:37:16.891848 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:16.891725    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f913fe36-7256-4171-ad6f-db25850da8ce-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f913fe36-7256-4171-ad6f-db25850da8ce-credential-socket\") pod \"f913fe36-7256-4171-ad6f-db25850da8ce\" (UID: \"f913fe36-7256-4171-ad6f-db25850da8ce\") "
Nov 03 20:37:16.891848 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:16.891816    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f913fe36-7256-4171-ad6f-db25850da8ce-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f913fe36-7256-4171-ad6f-db25850da8ce-istio-data\") pod \"f913fe36-7256-4171-ad6f-db25850da8ce\" (UID: \"f913fe36-7256-4171-ad6f-db25850da8ce\") "
Nov 03 20:37:16.892123 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:16.891847    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f913fe36-7256-4171-ad6f-db25850da8ce-kube-api-access-86clh\" (UniqueName: \"kubernetes.io/projected/f913fe36-7256-4171-ad6f-db25850da8ce-kube-api-access-86clh\") pod \"f913fe36-7256-4171-ad6f-db25850da8ce\" (UID: \"f913fe36-7256-4171-ad6f-db25850da8ce\") "
Nov 03 20:37:16.892123 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:16.891971    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f913fe36-7256-4171-ad6f-db25850da8ce-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f913fe36-7256-4171-ad6f-db25850da8ce-istio-envoy\") pod \"f913fe36-7256-4171-ad6f-db25850da8ce\" (UID: \"f913fe36-7256-4171-ad6f-db25850da8ce\") "
Nov 03 20:37:16.892123 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:16.892033    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f913fe36-7256-4171-ad6f-db25850da8ce-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f913fe36-7256-4171-ad6f-db25850da8ce-istio-podinfo\") pod \"f913fe36-7256-4171-ad6f-db25850da8ce\" (UID: \"f913fe36-7256-4171-ad6f-db25850da8ce\") "
Nov 03 20:37:16.892123 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:16.892061    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f913fe36-7256-4171-ad6f-db25850da8ce-istio-token\" (UniqueName: \"kubernetes.io/projected/f913fe36-7256-4171-ad6f-db25850da8ce-istio-token\") pod \"f913fe36-7256-4171-ad6f-db25850da8ce\" (UID: \"f913fe36-7256-4171-ad6f-db25850da8ce\") "
Nov 03 20:37:16.892123 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:16.892059    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f913fe36-7256-4171-ad6f-db25850da8ce-workload-socket" pod "f913fe36-7256-4171-ad6f-db25850da8ce" (UID: "f913fe36-7256-4171-ad6f-db25850da8ce"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:37:16.892400 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:16.892119    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f913fe36-7256-4171-ad6f-db25850da8ce-workload-certs" pod "f913fe36-7256-4171-ad6f-db25850da8ce" (UID: "f913fe36-7256-4171-ad6f-db25850da8ce"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:37:16.892400 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:16.892206    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f913fe36-7256-4171-ad6f-db25850da8ce-istio-data" pod "f913fe36-7256-4171-ad6f-db25850da8ce" (UID: "f913fe36-7256-4171-ad6f-db25850da8ce"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:37:16.892400 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:16.892330    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f913fe36-7256-4171-ad6f-db25850da8ce-istiod-ca-cert" pod "f913fe36-7256-4171-ad6f-db25850da8ce" (UID: "f913fe36-7256-4171-ad6f-db25850da8ce"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:37:16.892400 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:16.892346    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f913fe36-7256-4171-ad6f-db25850da8ce-credential-socket" pod "f913fe36-7256-4171-ad6f-db25850da8ce" (UID: "f913fe36-7256-4171-ad6f-db25850da8ce"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:37:16.892600 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:16.892557    5570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f913fe36-7256-4171-ad6f-db25850da8ce-credential-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:37:16.892600 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:16.892583    5570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f913fe36-7256-4171-ad6f-db25850da8ce-istio-data\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:37:16.892600 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:16.892599    5570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f913fe36-7256-4171-ad6f-db25850da8ce-istiod-ca-cert\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:37:16.892808 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:16.892611    5570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f913fe36-7256-4171-ad6f-db25850da8ce-workload-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:37:16.892808 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:16.892624    5570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f913fe36-7256-4171-ad6f-db25850da8ce-workload-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:37:16.894483 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:16.894454    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f913fe36-7256-4171-ad6f-db25850da8ce-istio-token" pod "f913fe36-7256-4171-ad6f-db25850da8ce" (UID: "f913fe36-7256-4171-ad6f-db25850da8ce"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:37:16.894483 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:16.894466    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f913fe36-7256-4171-ad6f-db25850da8ce-istio-envoy" pod "f913fe36-7256-4171-ad6f-db25850da8ce" (UID: "f913fe36-7256-4171-ad6f-db25850da8ce"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:37:16.894638 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:16.894554    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f913fe36-7256-4171-ad6f-db25850da8ce-istio-podinfo" pod "f913fe36-7256-4171-ad6f-db25850da8ce" (UID: "f913fe36-7256-4171-ad6f-db25850da8ce"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:37:16.894638 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:16.894567    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f913fe36-7256-4171-ad6f-db25850da8ce-kube-api-access-86clh" pod "f913fe36-7256-4171-ad6f-db25850da8ce" (UID: "f913fe36-7256-4171-ad6f-db25850da8ce"). InnerVolumeSpecName "kube-api-access-86clh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:37:16.994379 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:16.994284    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86clh\" (UniqueName: \"kubernetes.io/projected/f913fe36-7256-4171-ad6f-db25850da8ce-kube-api-access-86clh\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:37:16.994379 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:16.994317    5570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f913fe36-7256-4171-ad6f-db25850da8ce-istio-envoy\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:37:16.994379 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:16.994328    5570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f913fe36-7256-4171-ad6f-db25850da8ce-istio-podinfo\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:37:16.994379 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:16.994336    5570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f913fe36-7256-4171-ad6f-db25850da8ce-istio-token\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:37:17.183403 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:17.183361    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-a9ba83-openshift-default-8c675fb59-486wc"]
Nov 03 20:37:17.196980 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:17.196936    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-a9ba83-openshift-default-8c675fb59-486wc"]
Nov 03 20:37:17.965086 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:17.965035    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f913fe36-7256-4171-ad6f-db25850da8ce" path="/var/lib/kubelet/pods/f913fe36-7256-4171-ad6f-db25850da8ce/volumes"
Nov 03 20:37:18.460650 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:18.460614    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-503c4e-vjmmp"]
Nov 03 20:37:18.474018 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:18.473986    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-503c4e-vjmmp"]
Nov 03 20:37:19.174938 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:19.174888    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1d159a-5c484658dc-8qvxq" podUID="1ed630bf-cf60-4e37-9113-a1707b99937f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 20:37:19.963593 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:19.963549    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d5c0fd7-110a-414e-a0cc-acb706ed119b" path="/var/lib/kubelet/pods/0d5c0fd7-110a-414e-a0cc-acb706ed119b/volumes"
Nov 03 20:37:22.270834 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:22.270759    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1d159a-5c484658dc-8qvxq"
Nov 03 20:37:22.357631 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:22.357591    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ed630bf-cf60-4e37-9113-a1707b99937f-kube-api-access-rp6wq\" (UniqueName: \"kubernetes.io/projected/1ed630bf-cf60-4e37-9113-a1707b99937f-kube-api-access-rp6wq\") pod \"1ed630bf-cf60-4e37-9113-a1707b99937f\" (UID: \"1ed630bf-cf60-4e37-9113-a1707b99937f\") "
Nov 03 20:37:22.357828 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:22.357639    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ed630bf-cf60-4e37-9113-a1707b99937f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ed630bf-cf60-4e37-9113-a1707b99937f-plugins-config-volume\") pod \"1ed630bf-cf60-4e37-9113-a1707b99937f\" (UID: \"1ed630bf-cf60-4e37-9113-a1707b99937f\") "
Nov 03 20:37:22.358087 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:22.358062    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ed630bf-cf60-4e37-9113-a1707b99937f-plugins-config-volume" pod "1ed630bf-cf60-4e37-9113-a1707b99937f" (UID: "1ed630bf-cf60-4e37-9113-a1707b99937f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:37:22.360074 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:22.360045    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ed630bf-cf60-4e37-9113-a1707b99937f-kube-api-access-rp6wq" pod "1ed630bf-cf60-4e37-9113-a1707b99937f" (UID: "1ed630bf-cf60-4e37-9113-a1707b99937f"). InnerVolumeSpecName "kube-api-access-rp6wq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:37:22.418999 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:22.418958    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-4b95f3-9dc7f8fb8-glb8b"]
Nov 03 20:37:22.456630 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:22.456593    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-4b95f3-7544494d7-9j6kp"]
Nov 03 20:37:22.458968 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:22.458935    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rp6wq\" (UniqueName: \"kubernetes.io/projected/1ed630bf-cf60-4e37-9113-a1707b99937f-kube-api-access-rp6wq\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:37:22.458968 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:22.458961    5570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ed630bf-cf60-4e37-9113-a1707b99937f-plugins-config-volume\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:37:22.865446 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:22.865406    5570 generic.go:356] "Generic (PLEG): container finished" podID="1ed630bf-cf60-4e37-9113-a1707b99937f" containerID="dd61f44cdbdbff9a9b8c70ea8c8cc2997d1eae214f81d484aaf387a2325fffee" exitCode=0
Nov 03 20:37:22.865611 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:22.865461    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1d159a-5c484658dc-8qvxq"
Nov 03 20:37:22.865611 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:22.865491    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1d159a-5c484658dc-8qvxq" event={"ID":"1ed630bf-cf60-4e37-9113-a1707b99937f","Type":"ContainerDied","Data":"dd61f44cdbdbff9a9b8c70ea8c8cc2997d1eae214f81d484aaf387a2325fffee"}
Nov 03 20:37:22.865611 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:22.865546    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1d159a-5c484658dc-8qvxq" event={"ID":"1ed630bf-cf60-4e37-9113-a1707b99937f","Type":"ContainerDied","Data":"91798eb4a6b2d791273fd5918f34a359436bcbbf04374c5c7fb7d5abfc34c717"}
Nov 03 20:37:22.865611 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:22.865570    5570 scope.go:122] "RemoveContainer" containerID="dd61f44cdbdbff9a9b8c70ea8c8cc2997d1eae214f81d484aaf387a2325fffee"
Nov 03 20:37:22.876187 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:22.876165    5570 scope.go:122] "RemoveContainer" containerID="dd61f44cdbdbff9a9b8c70ea8c8cc2997d1eae214f81d484aaf387a2325fffee"
Nov 03 20:37:22.876480 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:37:22.876457    5570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd61f44cdbdbff9a9b8c70ea8c8cc2997d1eae214f81d484aaf387a2325fffee\": container with ID starting with dd61f44cdbdbff9a9b8c70ea8c8cc2997d1eae214f81d484aaf387a2325fffee not found: ID does not exist" containerID="dd61f44cdbdbff9a9b8c70ea8c8cc2997d1eae214f81d484aaf387a2325fffee"
Nov 03 20:37:22.876531 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:22.876490    5570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd61f44cdbdbff9a9b8c70ea8c8cc2997d1eae214f81d484aaf387a2325fffee"} err="failed to get container status \"dd61f44cdbdbff9a9b8c70ea8c8cc2997d1eae214f81d484aaf387a2325fffee\": rpc error: code = NotFound desc = could not find container \"dd61f44cdbdbff9a9b8c70ea8c8cc2997d1eae214f81d484aaf387a2325fffee\": container with ID starting with dd61f44cdbdbff9a9b8c70ea8c8cc2997d1eae214f81d484aaf387a2325fffee not found: ID does not exist"
Nov 03 20:37:22.919360 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:22.919327    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-1d159a-5c484658dc-8qvxq"]
Nov 03 20:37:22.922611 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:22.922579    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-1d159a-5c484658dc-8qvxq"]
Nov 03 20:37:23.453787 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:23.453748    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a9ba83-67b98df76d-24dtv"
Nov 03 20:37:23.476927 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:23.476891    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6deae9d9-81da-4c77-a15a-d805ff547c51-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6deae9d9-81da-4c77-a15a-d805ff547c51-plugins-config-volume\") pod \"6deae9d9-81da-4c77-a15a-d805ff547c51\" (UID: \"6deae9d9-81da-4c77-a15a-d805ff547c51\") "
Nov 03 20:37:23.477108 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:23.476983    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6deae9d9-81da-4c77-a15a-d805ff547c51-kube-api-access-9gt8w\" (UniqueName: \"kubernetes.io/projected/6deae9d9-81da-4c77-a15a-d805ff547c51-kube-api-access-9gt8w\") pod \"6deae9d9-81da-4c77-a15a-d805ff547c51\" (UID: \"6deae9d9-81da-4c77-a15a-d805ff547c51\") "
Nov 03 20:37:23.477378 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:23.477350    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6deae9d9-81da-4c77-a15a-d805ff547c51-plugins-config-volume" pod "6deae9d9-81da-4c77-a15a-d805ff547c51" (UID: "6deae9d9-81da-4c77-a15a-d805ff547c51"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:37:23.479198 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:23.479175    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6deae9d9-81da-4c77-a15a-d805ff547c51-kube-api-access-9gt8w" pod "6deae9d9-81da-4c77-a15a-d805ff547c51" (UID: "6deae9d9-81da-4c77-a15a-d805ff547c51"). InnerVolumeSpecName "kube-api-access-9gt8w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:37:23.500504 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:23.500468    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a9ba83-67d4dd56f9-22nhs"
Nov 03 20:37:23.578185 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:23.578148    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58fc81a1-5df4-4b6c-92bc-6dcf0836f400-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58fc81a1-5df4-4b6c-92bc-6dcf0836f400-plugins-config-volume\") pod \"58fc81a1-5df4-4b6c-92bc-6dcf0836f400\" (UID: \"58fc81a1-5df4-4b6c-92bc-6dcf0836f400\") "
Nov 03 20:37:23.578185 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:23.578193    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58fc81a1-5df4-4b6c-92bc-6dcf0836f400-kube-api-access-c7ldl\" (UniqueName: \"kubernetes.io/projected/58fc81a1-5df4-4b6c-92bc-6dcf0836f400-kube-api-access-c7ldl\") pod \"58fc81a1-5df4-4b6c-92bc-6dcf0836f400\" (UID: \"58fc81a1-5df4-4b6c-92bc-6dcf0836f400\") "
Nov 03 20:37:23.578425 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:23.578411    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9gt8w\" (UniqueName: \"kubernetes.io/projected/6deae9d9-81da-4c77-a15a-d805ff547c51-kube-api-access-9gt8w\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:37:23.578487 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:23.578427    5570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6deae9d9-81da-4c77-a15a-d805ff547c51-plugins-config-volume\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:37:23.578548 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:23.578527    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58fc81a1-5df4-4b6c-92bc-6dcf0836f400-plugins-config-volume" pod "58fc81a1-5df4-4b6c-92bc-6dcf0836f400" (UID: "58fc81a1-5df4-4b6c-92bc-6dcf0836f400"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:37:23.580411 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:23.580390    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58fc81a1-5df4-4b6c-92bc-6dcf0836f400-kube-api-access-c7ldl" pod "58fc81a1-5df4-4b6c-92bc-6dcf0836f400" (UID: "58fc81a1-5df4-4b6c-92bc-6dcf0836f400"). InnerVolumeSpecName "kube-api-access-c7ldl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:37:23.679546 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:23.679449    5570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58fc81a1-5df4-4b6c-92bc-6dcf0836f400-plugins-config-volume\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:37:23.679546 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:23.679483    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7ldl\" (UniqueName: \"kubernetes.io/projected/58fc81a1-5df4-4b6c-92bc-6dcf0836f400-kube-api-access-c7ldl\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:37:23.873931 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:23.873897    5570 generic.go:356] "Generic (PLEG): container finished" podID="6deae9d9-81da-4c77-a15a-d805ff547c51" containerID="bb882318962c16fb856a808f831b224d05ff43f5de4c53413f46980a3b2c636c" exitCode=0
Nov 03 20:37:23.874136 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:23.873947    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a9ba83-67b98df76d-24dtv"
Nov 03 20:37:23.874136 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:23.873977    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a9ba83-67b98df76d-24dtv" event={"ID":"6deae9d9-81da-4c77-a15a-d805ff547c51","Type":"ContainerDied","Data":"bb882318962c16fb856a808f831b224d05ff43f5de4c53413f46980a3b2c636c"}
Nov 03 20:37:23.874136 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:23.874030    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a9ba83-67b98df76d-24dtv" event={"ID":"6deae9d9-81da-4c77-a15a-d805ff547c51","Type":"ContainerDied","Data":"5c91b8e9b7e57c1615d4f2f87bda622e1e2a016b0ddbd9ebb2bd02e11f5ac1e6"}
Nov 03 20:37:23.874136 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:23.874059    5570 scope.go:122] "RemoveContainer" containerID="bb882318962c16fb856a808f831b224d05ff43f5de4c53413f46980a3b2c636c"
Nov 03 20:37:23.875492 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:23.875465    5570 generic.go:356] "Generic (PLEG): container finished" podID="58fc81a1-5df4-4b6c-92bc-6dcf0836f400" containerID="922ea1a73f9216f6a93ff2cd116a56b6f399719095090b582b4199a796ce54f5" exitCode=0
Nov 03 20:37:23.875632 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:23.875511    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a9ba83-67d4dd56f9-22nhs"
Nov 03 20:37:23.876416 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:23.876389    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a9ba83-67d4dd56f9-22nhs" event={"ID":"58fc81a1-5df4-4b6c-92bc-6dcf0836f400","Type":"ContainerDied","Data":"922ea1a73f9216f6a93ff2cd116a56b6f399719095090b582b4199a796ce54f5"}
Nov 03 20:37:23.876617 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:23.876589    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a9ba83-67d4dd56f9-22nhs" event={"ID":"58fc81a1-5df4-4b6c-92bc-6dcf0836f400","Type":"ContainerDied","Data":"c410110c2c215ea40e1c1a39095a6cc6def8d7da33eb32acff325a545d3ea3b3"}
Nov 03 20:37:23.884926 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:23.884904    5570 scope.go:122] "RemoveContainer" containerID="bb882318962c16fb856a808f831b224d05ff43f5de4c53413f46980a3b2c636c"
Nov 03 20:37:23.885309 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:37:23.885287    5570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb882318962c16fb856a808f831b224d05ff43f5de4c53413f46980a3b2c636c\": container with ID starting with bb882318962c16fb856a808f831b224d05ff43f5de4c53413f46980a3b2c636c not found: ID does not exist" containerID="bb882318962c16fb856a808f831b224d05ff43f5de4c53413f46980a3b2c636c"
Nov 03 20:37:23.885420 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:23.885319    5570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb882318962c16fb856a808f831b224d05ff43f5de4c53413f46980a3b2c636c"} err="failed to get container status \"bb882318962c16fb856a808f831b224d05ff43f5de4c53413f46980a3b2c636c\": rpc error: code = NotFound desc = could not find container \"bb882318962c16fb856a808f831b224d05ff43f5de4c53413f46980a3b2c636c\": container with ID starting with bb882318962c16fb856a808f831b224d05ff43f5de4c53413f46980a3b2c636c not found: ID does not exist"
Nov 03 20:37:23.885420 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:23.885337    5570 scope.go:122] "RemoveContainer" containerID="922ea1a73f9216f6a93ff2cd116a56b6f399719095090b582b4199a796ce54f5"
Nov 03 20:37:23.895488 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:23.895461    5570 scope.go:122] "RemoveContainer" containerID="922ea1a73f9216f6a93ff2cd116a56b6f399719095090b582b4199a796ce54f5"
Nov 03 20:37:23.895833 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:37:23.895809    5570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"922ea1a73f9216f6a93ff2cd116a56b6f399719095090b582b4199a796ce54f5\": container with ID starting with 922ea1a73f9216f6a93ff2cd116a56b6f399719095090b582b4199a796ce54f5 not found: ID does not exist" containerID="922ea1a73f9216f6a93ff2cd116a56b6f399719095090b582b4199a796ce54f5"
Nov 03 20:37:23.895953 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:23.895846    5570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"922ea1a73f9216f6a93ff2cd116a56b6f399719095090b582b4199a796ce54f5"} err="failed to get container status \"922ea1a73f9216f6a93ff2cd116a56b6f399719095090b582b4199a796ce54f5\": rpc error: code = NotFound desc = could not find container \"922ea1a73f9216f6a93ff2cd116a56b6f399719095090b582b4199a796ce54f5\": container with ID starting with 922ea1a73f9216f6a93ff2cd116a56b6f399719095090b582b4199a796ce54f5 not found: ID does not exist"
Nov 03 20:37:23.931970 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:23.931856    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a9ba83-67d4dd56f9-22nhs"]
Nov 03 20:37:23.934539 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:23.934510    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a9ba83-67d4dd56f9-22nhs"]
Nov 03 20:37:23.964348 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:23.964313    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ed630bf-cf60-4e37-9113-a1707b99937f" path="/var/lib/kubelet/pods/1ed630bf-cf60-4e37-9113-a1707b99937f/volumes"
Nov 03 20:37:23.964725 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:23.964711    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58fc81a1-5df4-4b6c-92bc-6dcf0836f400" path="/var/lib/kubelet/pods/58fc81a1-5df4-4b6c-92bc-6dcf0836f400/volumes"
Nov 03 20:37:23.971009 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:23.970983    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a9ba83-67b98df76d-24dtv"]
Nov 03 20:37:23.976147 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:23.976121    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a9ba83-67b98df76d-24dtv"]
Nov 03 20:37:25.435275 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:25.435239    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g69f8"]
Nov 03 20:37:25.933200 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:25.933162    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-a0866c-4smdj"]
Nov 03 20:37:25.933665 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:25.933651    5570 cpu_manager.go:414] "containerMap: removing container" podUID="40e4fc9c-c3c1-4aa0-98ad-eda347be89c5" containerName="istio-proxy"
Nov 03 20:37:25.933711 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:25.933666    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40e4fc9c-c3c1-4aa0-98ad-eda347be89c5" containerName="istio-proxy"
Nov 03 20:37:25.933711 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:25.933677    5570 cpu_manager.go:414] "containerMap: removing container" podUID="3facaeda-5192-4f41-ab9f-ae752e78dd60" containerName="revoke-keys"
Nov 03 20:37:25.933711 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:25.933685    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3facaeda-5192-4f41-ab9f-ae752e78dd60" containerName="revoke-keys"
Nov 03 20:37:25.933711 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:25.933702    5570 cpu_manager.go:414] "containerMap: removing container" podUID="1ed630bf-cf60-4e37-9113-a1707b99937f" containerName="payload-pre-processing"
Nov 03 20:37:25.933711 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:25.933708    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ed630bf-cf60-4e37-9113-a1707b99937f" containerName="payload-pre-processing"
Nov 03 20:37:25.933883 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:25.933736    5570 cpu_manager.go:414] "containerMap: removing container" podUID="f913fe36-7256-4171-ad6f-db25850da8ce" containerName="istio-proxy"
Nov 03 20:37:25.933883 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:25.933741    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f913fe36-7256-4171-ad6f-db25850da8ce" containerName="istio-proxy"
Nov 03 20:37:25.933883 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:25.933750    5570 cpu_manager.go:414] "containerMap: removing container" podUID="f913fe36-7256-4171-ad6f-db25850da8ce" containerName="istio-proxy"
Nov 03 20:37:25.933883 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:25.933755    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f913fe36-7256-4171-ad6f-db25850da8ce" containerName="istio-proxy"
Nov 03 20:37:25.933883 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:25.933784    5570 cpu_manager.go:414] "containerMap: removing container" podUID="6deae9d9-81da-4c77-a15a-d805ff547c51" containerName="payload-processing"
Nov 03 20:37:25.933883 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:25.933790    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6deae9d9-81da-4c77-a15a-d805ff547c51" containerName="payload-processing"
Nov 03 20:37:25.933883 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:25.933799    5570 cpu_manager.go:414] "containerMap: removing container" podUID="58fc81a1-5df4-4b6c-92bc-6dcf0836f400" containerName="payload-pre-processing"
Nov 03 20:37:25.933883 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:25.933805    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58fc81a1-5df4-4b6c-92bc-6dcf0836f400" containerName="payload-pre-processing"
Nov 03 20:37:25.934110 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:25.933884    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="3facaeda-5192-4f41-ab9f-ae752e78dd60" containerName="revoke-keys"
Nov 03 20:37:25.934110 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:25.933898    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ed630bf-cf60-4e37-9113-a1707b99937f" containerName="payload-pre-processing"
Nov 03 20:37:25.934110 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:25.933910    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="f913fe36-7256-4171-ad6f-db25850da8ce" containerName="istio-proxy"
Nov 03 20:37:25.934110 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:25.933919    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="f913fe36-7256-4171-ad6f-db25850da8ce" containerName="istio-proxy"
Nov 03 20:37:25.934110 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:25.933929    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="6deae9d9-81da-4c77-a15a-d805ff547c51" containerName="payload-processing"
Nov 03 20:37:25.934110 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:25.933936    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="58fc81a1-5df4-4b6c-92bc-6dcf0836f400" containerName="payload-pre-processing"
Nov 03 20:37:25.938593 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:25.938572    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-a0866c-4smdj"
Nov 03 20:37:25.953192 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:25.953161    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-a0866c-4smdj"]
Nov 03 20:37:25.964380 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:25.964353    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l7s5w\""
Nov 03 20:37:25.964380 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:25.964350    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:37:25.965401 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:25.965381    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6deae9d9-81da-4c77-a15a-d805ff547c51" path="/var/lib/kubelet/pods/6deae9d9-81da-4c77-a15a-d805ff547c51/volumes"
Nov 03 20:37:26.005957 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:26.005918    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27f50c73-70d9-4a9e-bbd0-2081d4441bea-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-a0866c-4smdj\" (UID: \"27f50c73-70d9-4a9e-bbd0-2081d4441bea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-a0866c-4smdj"
Nov 03 20:37:26.006132 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:26.005992    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9j5xj\" (UniqueName: \"kubernetes.io/projected/27f50c73-70d9-4a9e-bbd0-2081d4441bea-kube-api-access-9j5xj\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-a0866c-4smdj\" (UID: \"27f50c73-70d9-4a9e-bbd0-2081d4441bea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-a0866c-4smdj"
Nov 03 20:37:26.006132 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:26.006020    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27f50c73-70d9-4a9e-bbd0-2081d4441bea-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-a0866c-4smdj\" (UID: \"27f50c73-70d9-4a9e-bbd0-2081d4441bea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-a0866c-4smdj"
Nov 03 20:37:26.106972 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:26.106931    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27f50c73-70d9-4a9e-bbd0-2081d4441bea-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-a0866c-4smdj\" (UID: \"27f50c73-70d9-4a9e-bbd0-2081d4441bea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-a0866c-4smdj"
Nov 03 20:37:26.107154 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:26.106986    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9j5xj\" (UniqueName: \"kubernetes.io/projected/27f50c73-70d9-4a9e-bbd0-2081d4441bea-kube-api-access-9j5xj\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-a0866c-4smdj\" (UID: \"27f50c73-70d9-4a9e-bbd0-2081d4441bea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-a0866c-4smdj"
Nov 03 20:37:26.107154 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:26.107021    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27f50c73-70d9-4a9e-bbd0-2081d4441bea-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-a0866c-4smdj\" (UID: \"27f50c73-70d9-4a9e-bbd0-2081d4441bea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-a0866c-4smdj"
Nov 03 20:37:26.107618 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:26.107594    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27f50c73-70d9-4a9e-bbd0-2081d4441bea-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-a0866c-4smdj\" (UID: \"27f50c73-70d9-4a9e-bbd0-2081d4441bea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-a0866c-4smdj"
Nov 03 20:37:26.121819 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:26.121788    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:37:26.130066 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:26.130040    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27f50c73-70d9-4a9e-bbd0-2081d4441bea-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-a0866c-4smdj\" (UID: \"27f50c73-70d9-4a9e-bbd0-2081d4441bea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-a0866c-4smdj"
Nov 03 20:37:26.130354 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:26.130333    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9j5xj\" (UniqueName: \"kubernetes.io/projected/27f50c73-70d9-4a9e-bbd0-2081d4441bea-kube-api-access-9j5xj\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-a0866c-4smdj\" (UID: \"27f50c73-70d9-4a9e-bbd0-2081d4441bea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-a0866c-4smdj"
Nov 03 20:37:26.267672 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:26.267635    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l7s5w\""
Nov 03 20:37:26.273637 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:26.273612    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-a0866c-4smdj"
Nov 03 20:37:26.571070 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:26.571041    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-a0866c-4smdj"]
Nov 03 20:37:26.572550 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:37:26.572521    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27f50c73_70d9_4a9e_bbd0_2081d4441bea.slice/crio-82d2dab371b4b28b35d8079083164a003f3eb1ffdca8b065c59fa83a313944c1 WatchSource:0}: Error finding container 82d2dab371b4b28b35d8079083164a003f3eb1ffdca8b065c59fa83a313944c1: Status 404 returned error can't find the container with id 82d2dab371b4b28b35d8079083164a003f3eb1ffdca8b065c59fa83a313944c1
Nov 03 20:37:26.898943 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:26.898889    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-a0866c-4smdj" event={"ID":"27f50c73-70d9-4a9e-bbd0-2081d4441bea","Type":"ContainerStarted","Data":"b525da23b42a2bfa208703a4df2247e35b213257c3859f49c2a4771f1a785802"}
Nov 03 20:37:26.899168 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:26.898962    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-a0866c-4smdj" event={"ID":"27f50c73-70d9-4a9e-bbd0-2081d4441bea","Type":"ContainerStarted","Data":"82d2dab371b4b28b35d8079083164a003f3eb1ffdca8b065c59fa83a313944c1"}
Nov 03 20:37:26.939081 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:26.939021    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-a0866c-4smdj" podStartSLOduration=1.939004599 podStartE2EDuration="1.939004599s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:37:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:37:26.937811764 +0000 UTC m=+1981.090473650" watchObservedRunningTime="2026-11-03 20:37:26.939004599 +0000 UTC m=+1981.091666485"
Nov 03 20:37:27.228837 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:37:27.228796    5570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8bwhz" containerName="extractor"
Nov 03 20:37:27.419650 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:27.419604    5570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-4b95f3-9dc7f8fb8-glb8b" podUID="7a27b292-b8fa-4c2c-baab-8a82994152bf" containerName="payload-processing" containerID="cri-o://c7cfdeab7d60a641bc464aecac52d90e1709f0d8cf8876a563825c911061129f" gracePeriod=25
Nov 03 20:37:27.457499 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:27.457452    5570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4b95f3-7544494d7-9j6kp" podUID="c32d9ae0-0cf8-4f64-ba57-c2197625231d" containerName="payload-pre-processing" containerID="cri-o://02517b38a9d6adf9df62e9aa6ef1ae2b50b5ad31da8c677c3a4899186ee0d32e" gracePeriod=25
Nov 03 20:37:27.689537 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:27.689507    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-4b95f3-9dc7f8fb8-glb8b"
Nov 03 20:37:27.727014 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:27.726975    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a27b292-b8fa-4c2c-baab-8a82994152bf-kube-api-access-n68g5\" (UniqueName: \"kubernetes.io/projected/7a27b292-b8fa-4c2c-baab-8a82994152bf-kube-api-access-n68g5\") pod \"7a27b292-b8fa-4c2c-baab-8a82994152bf\" (UID: \"7a27b292-b8fa-4c2c-baab-8a82994152bf\") "
Nov 03 20:37:27.727183 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:27.727147    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a27b292-b8fa-4c2c-baab-8a82994152bf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a27b292-b8fa-4c2c-baab-8a82994152bf-plugins-config-volume\") pod \"7a27b292-b8fa-4c2c-baab-8a82994152bf\" (UID: \"7a27b292-b8fa-4c2c-baab-8a82994152bf\") "
Nov 03 20:37:27.727584 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:27.727557    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a27b292-b8fa-4c2c-baab-8a82994152bf-plugins-config-volume" pod "7a27b292-b8fa-4c2c-baab-8a82994152bf" (UID: "7a27b292-b8fa-4c2c-baab-8a82994152bf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:37:27.729232 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:27.729197    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a27b292-b8fa-4c2c-baab-8a82994152bf-kube-api-access-n68g5" pod "7a27b292-b8fa-4c2c-baab-8a82994152bf" (UID: "7a27b292-b8fa-4c2c-baab-8a82994152bf"). InnerVolumeSpecName "kube-api-access-n68g5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:37:27.828970 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:27.828877    5570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a27b292-b8fa-4c2c-baab-8a82994152bf-plugins-config-volume\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:37:27.828970 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:27.828908    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n68g5\" (UniqueName: \"kubernetes.io/projected/7a27b292-b8fa-4c2c-baab-8a82994152bf-kube-api-access-n68g5\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:37:27.906171 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:27.906133    5570 generic.go:356] "Generic (PLEG): container finished" podID="27f50c73-70d9-4a9e-bbd0-2081d4441bea" containerID="b525da23b42a2bfa208703a4df2247e35b213257c3859f49c2a4771f1a785802" exitCode=0
Nov 03 20:37:27.906333 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:27.906220    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-a0866c-4smdj" event={"ID":"27f50c73-70d9-4a9e-bbd0-2081d4441bea","Type":"ContainerDied","Data":"b525da23b42a2bfa208703a4df2247e35b213257c3859f49c2a4771f1a785802"}
Nov 03 20:37:27.907584 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:27.907556    5570 generic.go:356] "Generic (PLEG): container finished" podID="7a27b292-b8fa-4c2c-baab-8a82994152bf" containerID="c7cfdeab7d60a641bc464aecac52d90e1709f0d8cf8876a563825c911061129f" exitCode=0
Nov 03 20:37:27.907737 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:27.907602    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-4b95f3-9dc7f8fb8-glb8b"
Nov 03 20:37:27.907737 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:27.907646    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-4b95f3-9dc7f8fb8-glb8b" event={"ID":"7a27b292-b8fa-4c2c-baab-8a82994152bf","Type":"ContainerDied","Data":"c7cfdeab7d60a641bc464aecac52d90e1709f0d8cf8876a563825c911061129f"}
Nov 03 20:37:27.907737 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:27.907705    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-4b95f3-9dc7f8fb8-glb8b" event={"ID":"7a27b292-b8fa-4c2c-baab-8a82994152bf","Type":"ContainerDied","Data":"ec79e53715382869bf5a9918f662c5077bad7bc9e07f5528a5e2914c892c06c3"}
Nov 03 20:37:27.907737 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:27.907733    5570 scope.go:122] "RemoveContainer" containerID="c7cfdeab7d60a641bc464aecac52d90e1709f0d8cf8876a563825c911061129f"
Nov 03 20:37:27.917583 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:27.917562    5570 scope.go:122] "RemoveContainer" containerID="c7cfdeab7d60a641bc464aecac52d90e1709f0d8cf8876a563825c911061129f"
Nov 03 20:37:27.917909 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:37:27.917887    5570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7cfdeab7d60a641bc464aecac52d90e1709f0d8cf8876a563825c911061129f\": container with ID starting with c7cfdeab7d60a641bc464aecac52d90e1709f0d8cf8876a563825c911061129f not found: ID does not exist" containerID="c7cfdeab7d60a641bc464aecac52d90e1709f0d8cf8876a563825c911061129f"
Nov 03 20:37:27.917977 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:27.917915    5570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7cfdeab7d60a641bc464aecac52d90e1709f0d8cf8876a563825c911061129f"} err="failed to get container status \"c7cfdeab7d60a641bc464aecac52d90e1709f0d8cf8876a563825c911061129f\": rpc error: code = NotFound desc = could not find container \"c7cfdeab7d60a641bc464aecac52d90e1709f0d8cf8876a563825c911061129f\": container with ID starting with c7cfdeab7d60a641bc464aecac52d90e1709f0d8cf8876a563825c911061129f not found: ID does not exist"
Nov 03 20:37:27.984134 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:27.983996    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-4b95f3-9dc7f8fb8-glb8b"]
Nov 03 20:37:27.987626 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:27.987595    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-4b95f3-9dc7f8fb8-glb8b"]
Nov 03 20:37:29.062972 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:29.062939    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-a0866c-4smdj"
Nov 03 20:37:29.144629 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:29.144595    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/27f50c73-70d9-4a9e-bbd0-2081d4441bea-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27f50c73-70d9-4a9e-bbd0-2081d4441bea-maas-api-xks-ca\") pod \"27f50c73-70d9-4a9e-bbd0-2081d4441bea\" (UID: \"27f50c73-70d9-4a9e-bbd0-2081d4441bea\") "
Nov 03 20:37:29.144844 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:29.144654    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27f50c73-70d9-4a9e-bbd0-2081d4441bea-kube-api-access-9j5xj\" (UniqueName: \"kubernetes.io/projected/27f50c73-70d9-4a9e-bbd0-2081d4441bea-kube-api-access-9j5xj\") pod \"27f50c73-70d9-4a9e-bbd0-2081d4441bea\" (UID: \"27f50c73-70d9-4a9e-bbd0-2081d4441bea\") "
Nov 03 20:37:29.144844 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:29.144674    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27f50c73-70d9-4a9e-bbd0-2081d4441bea-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27f50c73-70d9-4a9e-bbd0-2081d4441bea-maas-api-openshift-ca\") pod \"27f50c73-70d9-4a9e-bbd0-2081d4441bea\" (UID: \"27f50c73-70d9-4a9e-bbd0-2081d4441bea\") "
Nov 03 20:37:29.145187 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:29.145156    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27f50c73-70d9-4a9e-bbd0-2081d4441bea-maas-api-openshift-ca" pod "27f50c73-70d9-4a9e-bbd0-2081d4441bea" (UID: "27f50c73-70d9-4a9e-bbd0-2081d4441bea"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:37:29.146842 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:29.146812    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27f50c73-70d9-4a9e-bbd0-2081d4441bea-maas-api-xks-ca" pod "27f50c73-70d9-4a9e-bbd0-2081d4441bea" (UID: "27f50c73-70d9-4a9e-bbd0-2081d4441bea"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:37:29.146966 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:29.146907    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27f50c73-70d9-4a9e-bbd0-2081d4441bea-kube-api-access-9j5xj" pod "27f50c73-70d9-4a9e-bbd0-2081d4441bea" (UID: "27f50c73-70d9-4a9e-bbd0-2081d4441bea"). InnerVolumeSpecName "kube-api-access-9j5xj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:37:29.246129 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:29.246091    5570 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27f50c73-70d9-4a9e-bbd0-2081d4441bea-maas-api-xks-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:37:29.246129 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:29.246123    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9j5xj\" (UniqueName: \"kubernetes.io/projected/27f50c73-70d9-4a9e-bbd0-2081d4441bea-kube-api-access-9j5xj\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:37:29.246129 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:29.246132    5570 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27f50c73-70d9-4a9e-bbd0-2081d4441bea-maas-api-openshift-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:37:29.921648 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:29.921616    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-a0866c-4smdj"
Nov 03 20:37:29.921885 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:29.921656    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-a0866c-4smdj" event={"ID":"27f50c73-70d9-4a9e-bbd0-2081d4441bea","Type":"ContainerDied","Data":"82d2dab371b4b28b35d8079083164a003f3eb1ffdca8b065c59fa83a313944c1"}
Nov 03 20:37:29.921885 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:29.921715    5570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="82d2dab371b4b28b35d8079083164a003f3eb1ffdca8b065c59fa83a313944c1"
Nov 03 20:37:29.963803 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:29.963728    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a27b292-b8fa-4c2c-baab-8a82994152bf" path="/var/lib/kubelet/pods/7a27b292-b8fa-4c2c-baab-8a82994152bf/volumes"
Nov 03 20:37:31.492732 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:31.492692    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4b95f3-ckdlh"]
Nov 03 20:37:31.506107 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:31.506070    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4b95f3-ckdlh"]
Nov 03 20:37:31.965432 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:31.965397    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3facaeda-5192-4f41-ab9f-ae752e78dd60" path="/var/lib/kubelet/pods/3facaeda-5192-4f41-ab9f-ae752e78dd60/volumes"
Nov 03 20:37:34.801090 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:34.801055    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4b95f3-7544494d7-9j6kp" podUID="c32d9ae0-0cf8-4f64-ba57-c2197625231d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 20:37:37.283883 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:37.278049    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g69f8"]
Nov 03 20:37:38.647798 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:38.644381    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-76b894-5fbtj"]
Nov 03 20:37:38.665285 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:38.665242    5570 cpu_manager.go:414] "containerMap: removing container" podUID="7a27b292-b8fa-4c2c-baab-8a82994152bf" containerName="payload-processing"
Nov 03 20:37:38.665285 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:38.665275    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a27b292-b8fa-4c2c-baab-8a82994152bf" containerName="payload-processing"
Nov 03 20:37:38.665285 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:38.665290    5570 cpu_manager.go:414] "containerMap: removing container" podUID="27f50c73-70d9-4a9e-bbd0-2081d4441bea" containerName="revoke-keys"
Nov 03 20:37:38.665633 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:38.665298    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27f50c73-70d9-4a9e-bbd0-2081d4441bea" containerName="revoke-keys"
Nov 03 20:37:38.665633 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:38.665449    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a27b292-b8fa-4c2c-baab-8a82994152bf" containerName="payload-processing"
Nov 03 20:37:38.665633 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:38.665464    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="27f50c73-70d9-4a9e-bbd0-2081d4441bea" containerName="revoke-keys"
Nov 03 20:37:38.671396 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:38.671366    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-76b894-5fbtj"]
Nov 03 20:37:38.671604 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:38.671450    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-76b894-5fbtj"
Nov 03 20:37:38.687189 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:38.687160    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:37:38.701170 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:38.701135    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l7s5w\""
Nov 03 20:37:38.797764 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:38.797725    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc7a1e84-39ab-4101-9d88-5d337e8cfe0c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-76b894-5fbtj\" (UID: \"bc7a1e84-39ab-4101-9d88-5d337e8cfe0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-76b894-5fbtj"
Nov 03 20:37:38.797978 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:38.797766    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8hw9\" (UniqueName: \"kubernetes.io/projected/bc7a1e84-39ab-4101-9d88-5d337e8cfe0c-kube-api-access-r8hw9\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-76b894-5fbtj\" (UID: \"bc7a1e84-39ab-4101-9d88-5d337e8cfe0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-76b894-5fbtj"
Nov 03 20:37:38.797978 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:38.797904    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc7a1e84-39ab-4101-9d88-5d337e8cfe0c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-76b894-5fbtj\" (UID: \"bc7a1e84-39ab-4101-9d88-5d337e8cfe0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-76b894-5fbtj"
Nov 03 20:37:38.899142 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:38.899038    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc7a1e84-39ab-4101-9d88-5d337e8cfe0c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-76b894-5fbtj\" (UID: \"bc7a1e84-39ab-4101-9d88-5d337e8cfe0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-76b894-5fbtj"
Nov 03 20:37:38.899142 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:38.899086    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8hw9\" (UniqueName: \"kubernetes.io/projected/bc7a1e84-39ab-4101-9d88-5d337e8cfe0c-kube-api-access-r8hw9\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-76b894-5fbtj\" (UID: \"bc7a1e84-39ab-4101-9d88-5d337e8cfe0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-76b894-5fbtj"
Nov 03 20:37:38.899142 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:38.899124    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc7a1e84-39ab-4101-9d88-5d337e8cfe0c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-76b894-5fbtj\" (UID: \"bc7a1e84-39ab-4101-9d88-5d337e8cfe0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-76b894-5fbtj"
Nov 03 20:37:38.899934 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:38.899886    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc7a1e84-39ab-4101-9d88-5d337e8cfe0c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-76b894-5fbtj\" (UID: \"bc7a1e84-39ab-4101-9d88-5d337e8cfe0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-76b894-5fbtj"
Nov 03 20:37:38.901764 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:38.901742    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc7a1e84-39ab-4101-9d88-5d337e8cfe0c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-76b894-5fbtj\" (UID: \"bc7a1e84-39ab-4101-9d88-5d337e8cfe0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-76b894-5fbtj"
Nov 03 20:37:38.919575 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:38.919543    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8hw9\" (UniqueName: \"kubernetes.io/projected/bc7a1e84-39ab-4101-9d88-5d337e8cfe0c-kube-api-access-r8hw9\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-76b894-5fbtj\" (UID: \"bc7a1e84-39ab-4101-9d88-5d337e8cfe0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-76b894-5fbtj"
Nov 03 20:37:38.986407 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:38.986368    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-76b894-5fbtj"
Nov 03 20:37:39.304077 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:39.304041    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-76b894-5fbtj"]
Nov 03 20:37:39.304460 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:37:39.304434    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc7a1e84_39ab_4101_9d88_5d337e8cfe0c.slice/crio-2dd3f646699f5aef670d80dc69e6294238d89a500c11bc150946649fc341c16e WatchSource:0}: Error finding container 2dd3f646699f5aef670d80dc69e6294238d89a500c11bc150946649fc341c16e: Status 404 returned error can't find the container with id 2dd3f646699f5aef670d80dc69e6294238d89a500c11bc150946649fc341c16e
Nov 03 20:37:40.005293 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:40.005231    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-76b894-5fbtj" event={"ID":"bc7a1e84-39ab-4101-9d88-5d337e8cfe0c","Type":"ContainerStarted","Data":"ff0f070ec195c347623c77fe2142ec5efacfe5e7aab04f2ab730b1a463624952"}
Nov 03 20:37:40.005796 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:40.005311    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-76b894-5fbtj" event={"ID":"bc7a1e84-39ab-4101-9d88-5d337e8cfe0c","Type":"ContainerStarted","Data":"2dd3f646699f5aef670d80dc69e6294238d89a500c11bc150946649fc341c16e"}
Nov 03 20:37:40.060799 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:40.057960    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-76b894-5fbtj" podStartSLOduration=2.057939024 podStartE2EDuration="2.057939024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:37:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:37:40.057242307 +0000 UTC m=+1994.209904194" watchObservedRunningTime="2026-11-03 20:37:40.057939024 +0000 UTC m=+1994.210600912"
Nov 03 20:37:41.012431 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:41.012397    5570 generic.go:356] "Generic (PLEG): container finished" podID="bc7a1e84-39ab-4101-9d88-5d337e8cfe0c" containerID="ff0f070ec195c347623c77fe2142ec5efacfe5e7aab04f2ab730b1a463624952" exitCode=0
Nov 03 20:37:41.012954 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:41.012497    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-76b894-5fbtj" event={"ID":"bc7a1e84-39ab-4101-9d88-5d337e8cfe0c","Type":"ContainerDied","Data":"ff0f070ec195c347623c77fe2142ec5efacfe5e7aab04f2ab730b1a463624952"}
Nov 03 20:37:42.190794 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:42.190756    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-76b894-5fbtj"
Nov 03 20:37:42.231762 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:42.231715    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc7a1e84-39ab-4101-9d88-5d337e8cfe0c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc7a1e84-39ab-4101-9d88-5d337e8cfe0c-maas-api-xks-ca\") pod \"bc7a1e84-39ab-4101-9d88-5d337e8cfe0c\" (UID: \"bc7a1e84-39ab-4101-9d88-5d337e8cfe0c\") "
Nov 03 20:37:42.234374 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:42.234346    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc7a1e84-39ab-4101-9d88-5d337e8cfe0c-maas-api-xks-ca" pod "bc7a1e84-39ab-4101-9d88-5d337e8cfe0c" (UID: "bc7a1e84-39ab-4101-9d88-5d337e8cfe0c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:37:42.332830 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:42.332721    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc7a1e84-39ab-4101-9d88-5d337e8cfe0c-kube-api-access-r8hw9\" (UniqueName: \"kubernetes.io/projected/bc7a1e84-39ab-4101-9d88-5d337e8cfe0c-kube-api-access-r8hw9\") pod \"bc7a1e84-39ab-4101-9d88-5d337e8cfe0c\" (UID: \"bc7a1e84-39ab-4101-9d88-5d337e8cfe0c\") "
Nov 03 20:37:42.333029 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:42.332839    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc7a1e84-39ab-4101-9d88-5d337e8cfe0c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc7a1e84-39ab-4101-9d88-5d337e8cfe0c-maas-api-openshift-ca\") pod \"bc7a1e84-39ab-4101-9d88-5d337e8cfe0c\" (UID: \"bc7a1e84-39ab-4101-9d88-5d337e8cfe0c\") "
Nov 03 20:37:42.333149 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:42.333134    5570 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc7a1e84-39ab-4101-9d88-5d337e8cfe0c-maas-api-xks-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:37:42.333256 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:42.333234    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc7a1e84-39ab-4101-9d88-5d337e8cfe0c-maas-api-openshift-ca" pod "bc7a1e84-39ab-4101-9d88-5d337e8cfe0c" (UID: "bc7a1e84-39ab-4101-9d88-5d337e8cfe0c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:37:42.334871 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:42.334834    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc7a1e84-39ab-4101-9d88-5d337e8cfe0c-kube-api-access-r8hw9" pod "bc7a1e84-39ab-4101-9d88-5d337e8cfe0c" (UID: "bc7a1e84-39ab-4101-9d88-5d337e8cfe0c"). InnerVolumeSpecName "kube-api-access-r8hw9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:37:42.434133 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:42.434097    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8hw9\" (UniqueName: \"kubernetes.io/projected/bc7a1e84-39ab-4101-9d88-5d337e8cfe0c-kube-api-access-r8hw9\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:37:42.434133 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:42.434125    5570 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc7a1e84-39ab-4101-9d88-5d337e8cfe0c-maas-api-openshift-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:37:43.025867 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:43.025833    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-76b894-5fbtj"
Nov 03 20:37:43.026054 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:43.025871    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-76b894-5fbtj" event={"ID":"bc7a1e84-39ab-4101-9d88-5d337e8cfe0c","Type":"ContainerDied","Data":"2dd3f646699f5aef670d80dc69e6294238d89a500c11bc150946649fc341c16e"}
Nov 03 20:37:43.026054 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:43.025916    5570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2dd3f646699f5aef670d80dc69e6294238d89a500c11bc150946649fc341c16e"
Nov 03 20:37:44.759762 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:44.759717    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-a0866c-4smdj"]
Nov 03 20:37:44.764574 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:44.764533    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-a0866c-4smdj"]
Nov 03 20:37:44.801033 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:44.800991    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4b95f3-7544494d7-9j6kp" podUID="c32d9ae0-0cf8-4f64-ba57-c2197625231d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 20:37:45.963509 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:45.963461    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27f50c73-70d9-4a9e-bbd0-2081d4441bea" path="/var/lib/kubelet/pods/27f50c73-70d9-4a9e-bbd0-2081d4441bea/volumes"
Nov 03 20:37:47.050876 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:47.050838    5570 generic.go:356] "Generic (PLEG): container finished" podID="9e1c6800-b892-4846-8686-63dbb9b87690" containerID="a37981af87d8ac58fb4bc1501a5fb409c54c5812026347d5aadec77740acfcac" exitCode=0
Nov 03 20:37:47.051346 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:47.050914    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-a0866c-openshift-default-59c9f599d5-4h8ks" event={"ID":"9e1c6800-b892-4846-8686-63dbb9b87690","Type":"ContainerDied","Data":"a37981af87d8ac58fb4bc1501a5fb409c54c5812026347d5aadec77740acfcac"}
Nov 03 20:37:47.051346 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:47.051337    5570 scope.go:122] "RemoveContainer" containerID="a37981af87d8ac58fb4bc1501a5fb409c54c5812026347d5aadec77740acfcac"
Nov 03 20:37:47.052766 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:47.052741    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:37:47.052856 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:47.052841    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:37:47.052904 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:47.052876    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:37:47.718024 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:47.717956    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4b95f3-7544494d7-9j6kp"
Nov 03 20:37:47.805403 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:47.805361    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c32d9ae0-0cf8-4f64-ba57-c2197625231d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c32d9ae0-0cf8-4f64-ba57-c2197625231d-plugins-config-volume\") pod \"c32d9ae0-0cf8-4f64-ba57-c2197625231d\" (UID: \"c32d9ae0-0cf8-4f64-ba57-c2197625231d\") "
Nov 03 20:37:47.805593 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:47.805514    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c32d9ae0-0cf8-4f64-ba57-c2197625231d-kube-api-access-4s7r7\" (UniqueName: \"kubernetes.io/projected/c32d9ae0-0cf8-4f64-ba57-c2197625231d-kube-api-access-4s7r7\") pod \"c32d9ae0-0cf8-4f64-ba57-c2197625231d\" (UID: \"c32d9ae0-0cf8-4f64-ba57-c2197625231d\") "
Nov 03 20:37:47.805861 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:47.805833    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c32d9ae0-0cf8-4f64-ba57-c2197625231d-plugins-config-volume" pod "c32d9ae0-0cf8-4f64-ba57-c2197625231d" (UID: "c32d9ae0-0cf8-4f64-ba57-c2197625231d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:37:47.805984 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:47.805970    5570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c32d9ae0-0cf8-4f64-ba57-c2197625231d-plugins-config-volume\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:37:47.807604 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:47.807576    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c32d9ae0-0cf8-4f64-ba57-c2197625231d-kube-api-access-4s7r7" pod "c32d9ae0-0cf8-4f64-ba57-c2197625231d" (UID: "c32d9ae0-0cf8-4f64-ba57-c2197625231d"). InnerVolumeSpecName "kube-api-access-4s7r7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:37:47.906768 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:47.906723    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4s7r7\" (UniqueName: \"kubernetes.io/projected/c32d9ae0-0cf8-4f64-ba57-c2197625231d-kube-api-access-4s7r7\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:37:48.067038 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:48.066996    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-a0866c-openshift-default-59c9f599d5-4h8ks" event={"ID":"9e1c6800-b892-4846-8686-63dbb9b87690","Type":"ContainerStarted","Data":"ef5dc7244ed7d95cef32b13b13d9c4ed8a84c5f7aae6df0bb0feedb1f3539e2e"}
Nov 03 20:37:48.068341 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:48.068314    5570 generic.go:356] "Generic (PLEG): container finished" podID="c32d9ae0-0cf8-4f64-ba57-c2197625231d" containerID="02517b38a9d6adf9df62e9aa6ef1ae2b50b5ad31da8c677c3a4899186ee0d32e" exitCode=0
Nov 03 20:37:48.068502 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:48.068352    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4b95f3-7544494d7-9j6kp"
Nov 03 20:37:48.068502 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:48.068393    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4b95f3-7544494d7-9j6kp" event={"ID":"c32d9ae0-0cf8-4f64-ba57-c2197625231d","Type":"ContainerDied","Data":"02517b38a9d6adf9df62e9aa6ef1ae2b50b5ad31da8c677c3a4899186ee0d32e"}
Nov 03 20:37:48.068502 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:48.068442    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4b95f3-7544494d7-9j6kp" event={"ID":"c32d9ae0-0cf8-4f64-ba57-c2197625231d","Type":"ContainerDied","Data":"ca9963b9ae5b8001e7f1f32a7ca965107b250c1bd7bdead358b20fcabc0d5658"}
Nov 03 20:37:48.068502 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:48.068463    5570 scope.go:122] "RemoveContainer" containerID="02517b38a9d6adf9df62e9aa6ef1ae2b50b5ad31da8c677c3a4899186ee0d32e"
Nov 03 20:37:48.078101 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:48.078075    5570 scope.go:122] "RemoveContainer" containerID="02517b38a9d6adf9df62e9aa6ef1ae2b50b5ad31da8c677c3a4899186ee0d32e"
Nov 03 20:37:48.078421 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:37:48.078397    5570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02517b38a9d6adf9df62e9aa6ef1ae2b50b5ad31da8c677c3a4899186ee0d32e\": container with ID starting with 02517b38a9d6adf9df62e9aa6ef1ae2b50b5ad31da8c677c3a4899186ee0d32e not found: ID does not exist" containerID="02517b38a9d6adf9df62e9aa6ef1ae2b50b5ad31da8c677c3a4899186ee0d32e"
Nov 03 20:37:48.078511 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:48.078429    5570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02517b38a9d6adf9df62e9aa6ef1ae2b50b5ad31da8c677c3a4899186ee0d32e"} err="failed to get container status \"02517b38a9d6adf9df62e9aa6ef1ae2b50b5ad31da8c677c3a4899186ee0d32e\": rpc error: code = NotFound desc = could not find container \"02517b38a9d6adf9df62e9aa6ef1ae2b50b5ad31da8c677c3a4899186ee0d32e\": container with ID starting with 02517b38a9d6adf9df62e9aa6ef1ae2b50b5ad31da8c677c3a4899186ee0d32e not found: ID does not exist"
Nov 03 20:37:48.162927 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:48.162872    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-4b95f3-7544494d7-9j6kp"]
Nov 03 20:37:48.163984 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:48.163952    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-4b95f3-7544494d7-9j6kp"]
Nov 03 20:37:48.183087 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:48.183055    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-a0866c-openshift-default-59c9f599d5-4h8ks"
Nov 03 20:37:48.184494 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:48.184463    5570 patch_prober.go:29] interesting pod/e2e-shared-b-w4-a0866c-openshift-default-59c9f599d5-4h8ks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body=
Nov 03 20:37:48.184652 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:48.184522    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-a0866c-openshift-default-59c9f599d5-4h8ks" podUID="9e1c6800-b892-4846-8686-63dbb9b87690" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused"
Nov 03 20:37:48.873799 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:48.872922    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g69f8"]
Nov 03 20:37:49.184383 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:49.184274    5570 patch_prober.go:29] interesting pod/e2e-shared-b-w4-a0866c-openshift-default-59c9f599d5-4h8ks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body=
Nov 03 20:37:49.184383 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:49.184352    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-a0866c-openshift-default-59c9f599d5-4h8ks" podUID="9e1c6800-b892-4846-8686-63dbb9b87690" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused"
Nov 03 20:37:49.969473 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:49.969437    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c32d9ae0-0cf8-4f64-ba57-c2197625231d" path="/var/lib/kubelet/pods/c32d9ae0-0cf8-4f64-ba57-c2197625231d/volumes"
Nov 03 20:37:50.183285 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:50.183250    5570 patch_prober.go:29] interesting pod/e2e-shared-b-w4-a0866c-openshift-default-59c9f599d5-4h8ks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body=
Nov 03 20:37:50.183496 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:50.183310    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-a0866c-openshift-default-59c9f599d5-4h8ks" podUID="9e1c6800-b892-4846-8686-63dbb9b87690" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused"
Nov 03 20:37:50.705360 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:50.705321    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-76b894-5fbtj"]
Nov 03 20:37:50.716549 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:50.716514    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-76b894-5fbtj"]
Nov 03 20:37:51.184319 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:51.184230    5570 patch_prober.go:29] interesting pod/e2e-shared-b-w4-a0866c-openshift-default-59c9f599d5-4h8ks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body=
Nov 03 20:37:51.184319 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:51.184305    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-a0866c-openshift-default-59c9f599d5-4h8ks" podUID="9e1c6800-b892-4846-8686-63dbb9b87690" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused"
Nov 03 20:37:51.777032 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:51.776997    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-a0866c-openshift-default-59c9f599d5-4h8ks"
Nov 03 20:37:51.855937 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:51.855876    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-a0866c-openshift-default-59c9f599d5-4h8ks"]
Nov 03 20:37:51.968243 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:51.968209    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc7a1e84-39ab-4101-9d88-5d337e8cfe0c" path="/var/lib/kubelet/pods/bc7a1e84-39ab-4101-9d88-5d337e8cfe0c/volumes"
Nov 03 20:37:52.096537 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:52.096431    5570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-a0866c-openshift-default-59c9f599d5-4h8ks" podUID="9e1c6800-b892-4846-8686-63dbb9b87690" containerName="istio-proxy" containerID="cri-o://ef5dc7244ed7d95cef32b13b13d9c4ed8a84c5f7aae6df0bb0feedb1f3539e2e" gracePeriod=30
Nov 03 20:37:57.227837 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:37:57.227796    5570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8bwhz" containerName="extractor"
Nov 03 20:37:57.298037 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:57.298006    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-e80203/distinct-2-w4-e80203-kserve-756f9b9bc6-69gxj"]
Nov 03 20:37:57.313452 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:57.313411    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-e80203/premium-w4-e80203-kserve-5f9f445456-bms9d"]
Nov 03 20:37:57.905269 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:57.905230    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-76b894-openshift-default-6669d4d94f-vrq8z"]
Nov 03 20:37:57.905501 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:57.905453    5570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-76b894-openshift-default-6669d4d94f-vrq8z" podUID="3db8ac79-76fc-4d46-ac4f-6536fe192b8f" containerName="istio-proxy" containerID="cri-o://08e6c055f3b06fb5e23c8d8740789d057e79fe1612f7eeefc46ba3e8200254e9" gracePeriod=30
Nov 03 20:37:58.773583 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:58.773548    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-a0866c-openshift-default-59c9f599d5-4h8ks"
Nov 03 20:37:58.859015 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:58.858921    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e1c6800-b892-4846-8686-63dbb9b87690-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e1c6800-b892-4846-8686-63dbb9b87690-workload-socket\") pod \"9e1c6800-b892-4846-8686-63dbb9b87690\" (UID: \"9e1c6800-b892-4846-8686-63dbb9b87690\") "
Nov 03 20:37:58.859015 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:58.858971    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e1c6800-b892-4846-8686-63dbb9b87690-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e1c6800-b892-4846-8686-63dbb9b87690-workload-certs\") pod \"9e1c6800-b892-4846-8686-63dbb9b87690\" (UID: \"9e1c6800-b892-4846-8686-63dbb9b87690\") "
Nov 03 20:37:58.859285 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:58.859075    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e1c6800-b892-4846-8686-63dbb9b87690-istio-token\" (UniqueName: \"kubernetes.io/projected/9e1c6800-b892-4846-8686-63dbb9b87690-istio-token\") pod \"9e1c6800-b892-4846-8686-63dbb9b87690\" (UID: \"9e1c6800-b892-4846-8686-63dbb9b87690\") "
Nov 03 20:37:58.859285 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:58.859124    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e1c6800-b892-4846-8686-63dbb9b87690-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e1c6800-b892-4846-8686-63dbb9b87690-istio-envoy\") pod \"9e1c6800-b892-4846-8686-63dbb9b87690\" (UID: \"9e1c6800-b892-4846-8686-63dbb9b87690\") "
Nov 03 20:37:58.859285 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:58.859171    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e1c6800-b892-4846-8686-63dbb9b87690-kube-api-access-cxpfl\" (UniqueName: \"kubernetes.io/projected/9e1c6800-b892-4846-8686-63dbb9b87690-kube-api-access-cxpfl\") pod \"9e1c6800-b892-4846-8686-63dbb9b87690\" (UID: \"9e1c6800-b892-4846-8686-63dbb9b87690\") "
Nov 03 20:37:58.859285 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:58.859269    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e1c6800-b892-4846-8686-63dbb9b87690-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e1c6800-b892-4846-8686-63dbb9b87690-istiod-ca-cert\") pod \"9e1c6800-b892-4846-8686-63dbb9b87690\" (UID: \"9e1c6800-b892-4846-8686-63dbb9b87690\") "
Nov 03 20:37:58.859502 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:58.859311    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e1c6800-b892-4846-8686-63dbb9b87690-workload-certs" pod "9e1c6800-b892-4846-8686-63dbb9b87690" (UID: "9e1c6800-b892-4846-8686-63dbb9b87690"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:37:58.859502 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:58.859326    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e1c6800-b892-4846-8686-63dbb9b87690-workload-socket" pod "9e1c6800-b892-4846-8686-63dbb9b87690" (UID: "9e1c6800-b892-4846-8686-63dbb9b87690"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:37:58.859502 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:58.859325    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9e1c6800-b892-4846-8686-63dbb9b87690-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e1c6800-b892-4846-8686-63dbb9b87690-istio-podinfo\") pod \"9e1c6800-b892-4846-8686-63dbb9b87690\" (UID: \"9e1c6800-b892-4846-8686-63dbb9b87690\") "
Nov 03 20:37:58.859502 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:58.859402    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e1c6800-b892-4846-8686-63dbb9b87690-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e1c6800-b892-4846-8686-63dbb9b87690-credential-socket\") pod \"9e1c6800-b892-4846-8686-63dbb9b87690\" (UID: \"9e1c6800-b892-4846-8686-63dbb9b87690\") "
Nov 03 20:37:58.859502 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:58.859433    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e1c6800-b892-4846-8686-63dbb9b87690-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e1c6800-b892-4846-8686-63dbb9b87690-istio-data\") pod \"9e1c6800-b892-4846-8686-63dbb9b87690\" (UID: \"9e1c6800-b892-4846-8686-63dbb9b87690\") "
Nov 03 20:37:58.859978 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:58.859684    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e1c6800-b892-4846-8686-63dbb9b87690-credential-socket" pod "9e1c6800-b892-4846-8686-63dbb9b87690" (UID: "9e1c6800-b892-4846-8686-63dbb9b87690"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:37:58.859978 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:58.859710    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e1c6800-b892-4846-8686-63dbb9b87690-istiod-ca-cert" pod "9e1c6800-b892-4846-8686-63dbb9b87690" (UID: "9e1c6800-b892-4846-8686-63dbb9b87690"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:37:58.859978 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:58.859962    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e1c6800-b892-4846-8686-63dbb9b87690-istio-data" pod "9e1c6800-b892-4846-8686-63dbb9b87690" (UID: "9e1c6800-b892-4846-8686-63dbb9b87690"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:37:58.860250 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:58.860229    5570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e1c6800-b892-4846-8686-63dbb9b87690-workload-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:37:58.860250 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:58.860250    5570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e1c6800-b892-4846-8686-63dbb9b87690-workload-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:37:58.860377 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:58.860260    5570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e1c6800-b892-4846-8686-63dbb9b87690-istiod-ca-cert\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:37:58.860377 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:58.860270    5570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e1c6800-b892-4846-8686-63dbb9b87690-credential-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:37:58.860377 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:58.860294    5570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e1c6800-b892-4846-8686-63dbb9b87690-istio-data\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:37:58.861637 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:58.861617    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e1c6800-b892-4846-8686-63dbb9b87690-istio-token" pod "9e1c6800-b892-4846-8686-63dbb9b87690" (UID: "9e1c6800-b892-4846-8686-63dbb9b87690"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:37:58.861943 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:58.861922    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e1c6800-b892-4846-8686-63dbb9b87690-kube-api-access-cxpfl" pod "9e1c6800-b892-4846-8686-63dbb9b87690" (UID: "9e1c6800-b892-4846-8686-63dbb9b87690"). InnerVolumeSpecName "kube-api-access-cxpfl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:37:58.862026 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:58.861987    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e1c6800-b892-4846-8686-63dbb9b87690-istio-envoy" pod "9e1c6800-b892-4846-8686-63dbb9b87690" (UID: "9e1c6800-b892-4846-8686-63dbb9b87690"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:37:58.862123 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:58.862107    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9e1c6800-b892-4846-8686-63dbb9b87690-istio-podinfo" pod "9e1c6800-b892-4846-8686-63dbb9b87690" (UID: "9e1c6800-b892-4846-8686-63dbb9b87690"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:37:58.961699 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:58.961657    5570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e1c6800-b892-4846-8686-63dbb9b87690-istio-envoy\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:37:58.961699 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:58.961690    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxpfl\" (UniqueName: \"kubernetes.io/projected/9e1c6800-b892-4846-8686-63dbb9b87690-kube-api-access-cxpfl\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:37:58.961699 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:58.961699    5570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e1c6800-b892-4846-8686-63dbb9b87690-istio-podinfo\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:37:58.961699 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:58.961708    5570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e1c6800-b892-4846-8686-63dbb9b87690-istio-token\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:37:59.158581 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:59.158488    5570 generic.go:356] "Generic (PLEG): container finished" podID="9e1c6800-b892-4846-8686-63dbb9b87690" containerID="ef5dc7244ed7d95cef32b13b13d9c4ed8a84c5f7aae6df0bb0feedb1f3539e2e" exitCode=0
Nov 03 20:37:59.158581 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:59.158544    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-a0866c-openshift-default-59c9f599d5-4h8ks"
Nov 03 20:37:59.158581 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:59.158570    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-a0866c-openshift-default-59c9f599d5-4h8ks" event={"ID":"9e1c6800-b892-4846-8686-63dbb9b87690","Type":"ContainerDied","Data":"ef5dc7244ed7d95cef32b13b13d9c4ed8a84c5f7aae6df0bb0feedb1f3539e2e"}
Nov 03 20:37:59.158916 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:59.158627    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-a0866c-openshift-default-59c9f599d5-4h8ks" event={"ID":"9e1c6800-b892-4846-8686-63dbb9b87690","Type":"ContainerDied","Data":"d604e752150e2ecdcfbaf7da193696e6c8ead5baeccd6c390a732bb91cf931e9"}
Nov 03 20:37:59.158916 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:59.158648    5570 scope.go:122] "RemoveContainer" containerID="ef5dc7244ed7d95cef32b13b13d9c4ed8a84c5f7aae6df0bb0feedb1f3539e2e"
Nov 03 20:37:59.168888 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:59.168869    5570 scope.go:122] "RemoveContainer" containerID="a37981af87d8ac58fb4bc1501a5fb409c54c5812026347d5aadec77740acfcac"
Nov 03 20:37:59.178739 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:59.178720    5570 scope.go:122] "RemoveContainer" containerID="ef5dc7244ed7d95cef32b13b13d9c4ed8a84c5f7aae6df0bb0feedb1f3539e2e"
Nov 03 20:37:59.179059 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:37:59.179037    5570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef5dc7244ed7d95cef32b13b13d9c4ed8a84c5f7aae6df0bb0feedb1f3539e2e\": container with ID starting with ef5dc7244ed7d95cef32b13b13d9c4ed8a84c5f7aae6df0bb0feedb1f3539e2e not found: ID does not exist" containerID="ef5dc7244ed7d95cef32b13b13d9c4ed8a84c5f7aae6df0bb0feedb1f3539e2e"
Nov 03 20:37:59.179113 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:59.179072    5570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef5dc7244ed7d95cef32b13b13d9c4ed8a84c5f7aae6df0bb0feedb1f3539e2e"} err="failed to get container status \"ef5dc7244ed7d95cef32b13b13d9c4ed8a84c5f7aae6df0bb0feedb1f3539e2e\": rpc error: code = NotFound desc = could not find container \"ef5dc7244ed7d95cef32b13b13d9c4ed8a84c5f7aae6df0bb0feedb1f3539e2e\": container with ID starting with ef5dc7244ed7d95cef32b13b13d9c4ed8a84c5f7aae6df0bb0feedb1f3539e2e not found: ID does not exist"
Nov 03 20:37:59.179113 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:59.179094    5570 scope.go:122] "RemoveContainer" containerID="a37981af87d8ac58fb4bc1501a5fb409c54c5812026347d5aadec77740acfcac"
Nov 03 20:37:59.179380 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:37:59.179361    5570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a37981af87d8ac58fb4bc1501a5fb409c54c5812026347d5aadec77740acfcac\": container with ID starting with a37981af87d8ac58fb4bc1501a5fb409c54c5812026347d5aadec77740acfcac not found: ID does not exist" containerID="a37981af87d8ac58fb4bc1501a5fb409c54c5812026347d5aadec77740acfcac"
Nov 03 20:37:59.179443 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:59.179387    5570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a37981af87d8ac58fb4bc1501a5fb409c54c5812026347d5aadec77740acfcac"} err="failed to get container status \"a37981af87d8ac58fb4bc1501a5fb409c54c5812026347d5aadec77740acfcac\": rpc error: code = NotFound desc = could not find container \"a37981af87d8ac58fb4bc1501a5fb409c54c5812026347d5aadec77740acfcac\": container with ID starting with a37981af87d8ac58fb4bc1501a5fb409c54c5812026347d5aadec77740acfcac not found: ID does not exist"
Nov 03 20:37:59.213944 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:59.213909    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-a0866c-openshift-default-59c9f599d5-4h8ks"]
Nov 03 20:37:59.227062 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:59.227031    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-a0866c-openshift-default-59c9f599d5-4h8ks"]
Nov 03 20:37:59.964308 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:37:59.964273    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e1c6800-b892-4846-8686-63dbb9b87690" path="/var/lib/kubelet/pods/9e1c6800-b892-4846-8686-63dbb9b87690/volumes"
Nov 03 20:38:02.674362 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:02.674323    5570 patch_prober.go:29] interesting pod/e2e-shared-a-w3-76b894-openshift-default-6669d4d94f-vrq8z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 20:38:02.674753 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:02.674389    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-76b894-openshift-default-6669d4d94f-vrq8z" podUID="3db8ac79-76fc-4d46-ac4f-6536fe192b8f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 20:38:03.090543 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:03.090501    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa4571df-5g52m"]
Nov 03 20:38:03.091260 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:03.091231    5570 cpu_manager.go:414] "containerMap: removing container" podUID="9e1c6800-b892-4846-8686-63dbb9b87690" containerName="istio-proxy"
Nov 03 20:38:03.091455 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:03.091436    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e1c6800-b892-4846-8686-63dbb9b87690" containerName="istio-proxy"
Nov 03 20:38:03.093507 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:03.091568    5570 cpu_manager.go:414] "containerMap: removing container" podUID="c32d9ae0-0cf8-4f64-ba57-c2197625231d" containerName="payload-pre-processing"
Nov 03 20:38:03.093507 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:03.091584    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c32d9ae0-0cf8-4f64-ba57-c2197625231d" containerName="payload-pre-processing"
Nov 03 20:38:03.093507 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:03.091596    5570 cpu_manager.go:414] "containerMap: removing container" podUID="bc7a1e84-39ab-4101-9d88-5d337e8cfe0c" containerName="revoke-keys"
Nov 03 20:38:03.093507 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:03.091603    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc7a1e84-39ab-4101-9d88-5d337e8cfe0c" containerName="revoke-keys"
Nov 03 20:38:03.093507 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:03.091668    5570 cpu_manager.go:414] "containerMap: removing container" podUID="9e1c6800-b892-4846-8686-63dbb9b87690" containerName="istio-proxy"
Nov 03 20:38:03.093507 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:03.091676    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e1c6800-b892-4846-8686-63dbb9b87690" containerName="istio-proxy"
Nov 03 20:38:03.093507 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:03.091901    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e1c6800-b892-4846-8686-63dbb9b87690" containerName="istio-proxy"
Nov 03 20:38:03.093507 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:03.091939    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e1c6800-b892-4846-8686-63dbb9b87690" containerName="istio-proxy"
Nov 03 20:38:03.093507 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:03.091949    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="c32d9ae0-0cf8-4f64-ba57-c2197625231d" containerName="payload-pre-processing"
Nov 03 20:38:03.093507 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:03.091960    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc7a1e84-39ab-4101-9d88-5d337e8cfe0c" containerName="revoke-keys"
Nov 03 20:38:03.102224 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:03.102192    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa4571df-5g52m"
Nov 03 20:38:03.112189 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:03.109749    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa4571df-5g52m"]
Nov 03 20:38:03.176940 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:03.176900    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g69f8"]
Nov 03 20:38:03.184149 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:03.184119    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:38:03.184316 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:03.184260    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l7s5w\""
Nov 03 20:38:03.185437 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:03.185414    5570 generic.go:356] "Generic (PLEG): container finished" podID="3db8ac79-76fc-4d46-ac4f-6536fe192b8f" containerID="08e6c055f3b06fb5e23c8d8740789d057e79fe1612f7eeefc46ba3e8200254e9" exitCode=0
Nov 03 20:38:03.185587 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:03.185482    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-76b894-openshift-default-6669d4d94f-vrq8z" event={"ID":"3db8ac79-76fc-4d46-ac4f-6536fe192b8f","Type":"ContainerDied","Data":"08e6c055f3b06fb5e23c8d8740789d057e79fe1612f7eeefc46ba3e8200254e9"}
Nov 03 20:38:03.185587 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:03.185540    5570 scope.go:122] "RemoveContainer" containerID="e71f0ebc0297767f2edd7a26e6152c5a29917591acf45764b21d98a005b7a272"
Nov 03 20:38:03.209544 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:03.209518    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-76b894-openshift-default-6669d4d94f-vrq8z"
Nov 03 20:38:03.214025 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:03.213997    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/746cdf44-bdd0-4040-87a9-43ff9b48cf99-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fa4571df-5g52m\" (UID: \"746cdf44-bdd0-4040-87a9-43ff9b48cf99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa4571df-5g52m"
Nov 03 20:38:03.214171 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:03.214053    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/746cdf44-bdd0-4040-87a9-43ff9b48cf99-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fa4571df-5g52m\" (UID: \"746cdf44-bdd0-4040-87a9-43ff9b48cf99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa4571df-5g52m"
Nov 03 20:38:03.214171 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:03.214143    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ht9dc\" (UniqueName: \"kubernetes.io/projected/746cdf44-bdd0-4040-87a9-43ff9b48cf99-kube-api-access-ht9dc\") pod \"maas-api-revoke-sub-fa4571df-5g52m\" (UID: \"746cdf44-bdd0-4040-87a9-43ff9b48cf99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa4571df-5g52m"
Nov 03 20:38:03.315237 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:03.315202    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3db8ac79-76fc-4d46-ac4f-6536fe192b8f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3db8ac79-76fc-4d46-ac4f-6536fe192b8f-workload-socket\") pod \"3db8ac79-76fc-4d46-ac4f-6536fe192b8f\" (UID: \"3db8ac79-76fc-4d46-ac4f-6536fe192b8f\") "
Nov 03 20:38:03.315237 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:03.315240    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3db8ac79-76fc-4d46-ac4f-6536fe192b8f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3db8ac79-76fc-4d46-ac4f-6536fe192b8f-istio-envoy\") pod \"3db8ac79-76fc-4d46-ac4f-6536fe192b8f\" (UID: \"3db8ac79-76fc-4d46-ac4f-6536fe192b8f\") "
Nov 03 20:38:03.315501 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:03.315266    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3db8ac79-76fc-4d46-ac4f-6536fe192b8f-istio-token\" (UniqueName: \"kubernetes.io/projected/3db8ac79-76fc-4d46-ac4f-6536fe192b8f-istio-token\") pod \"3db8ac79-76fc-4d46-ac4f-6536fe192b8f\" (UID: \"3db8ac79-76fc-4d46-ac4f-6536fe192b8f\") "
Nov 03 20:38:03.315501 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:03.315361    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3db8ac79-76fc-4d46-ac4f-6536fe192b8f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3db8ac79-76fc-4d46-ac4f-6536fe192b8f-istiod-ca-cert\") pod \"3db8ac79-76fc-4d46-ac4f-6536fe192b8f\" (UID: \"3db8ac79-76fc-4d46-ac4f-6536fe192b8f\") "
Nov 03 20:38:03.315501 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:03.315382    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3db8ac79-76fc-4d46-ac4f-6536fe192b8f-kube-api-access-v7nl2\" (UniqueName: \"kubernetes.io/projected/3db8ac79-76fc-4d46-ac4f-6536fe192b8f-kube-api-access-v7nl2\") pod \"3db8ac79-76fc-4d46-ac4f-6536fe192b8f\" (UID: \"3db8ac79-76fc-4d46-ac4f-6536fe192b8f\") "
Nov 03 20:38:03.315501 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:03.315401    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3db8ac79-76fc-4d46-ac4f-6536fe192b8f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3db8ac79-76fc-4d46-ac4f-6536fe192b8f-istio-data\") pod \"3db8ac79-76fc-4d46-ac4f-6536fe192b8f\" (UID: \"3db8ac79-76fc-4d46-ac4f-6536fe192b8f\") "
Nov 03 20:38:03.315501 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:03.315435    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3db8ac79-76fc-4d46-ac4f-6536fe192b8f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3db8ac79-76fc-4d46-ac4f-6536fe192b8f-workload-certs\") pod \"3db8ac79-76fc-4d46-ac4f-6536fe192b8f\" (UID: \"3db8ac79-76fc-4d46-ac4f-6536fe192b8f\") "
Nov 03 20:38:03.315501 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:03.315466    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3db8ac79-76fc-4d46-ac4f-6536fe192b8f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3db8ac79-76fc-4d46-ac4f-6536fe192b8f-istio-podinfo\") pod \"3db8ac79-76fc-4d46-ac4f-6536fe192b8f\" (UID: \"3db8ac79-76fc-4d46-ac4f-6536fe192b8f\") "
Nov 03 20:38:03.315879 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:03.315513    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3db8ac79-76fc-4d46-ac4f-6536fe192b8f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3db8ac79-76fc-4d46-ac4f-6536fe192b8f-credential-socket\") pod \"3db8ac79-76fc-4d46-ac4f-6536fe192b8f\" (UID: \"3db8ac79-76fc-4d46-ac4f-6536fe192b8f\") "
Nov 03 20:38:03.315879 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:03.315763    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3db8ac79-76fc-4d46-ac4f-6536fe192b8f-istio-data" pod "3db8ac79-76fc-4d46-ac4f-6536fe192b8f" (UID: "3db8ac79-76fc-4d46-ac4f-6536fe192b8f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:38:03.315879 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:03.315816    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/746cdf44-bdd0-4040-87a9-43ff9b48cf99-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fa4571df-5g52m\" (UID: \"746cdf44-bdd0-4040-87a9-43ff9b48cf99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa4571df-5g52m"
Nov 03 20:38:03.315879 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:03.315833    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3db8ac79-76fc-4d46-ac4f-6536fe192b8f-workload-certs" pod "3db8ac79-76fc-4d46-ac4f-6536fe192b8f" (UID: "3db8ac79-76fc-4d46-ac4f-6536fe192b8f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:38:03.316125 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:03.315930    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3db8ac79-76fc-4d46-ac4f-6536fe192b8f-istiod-ca-cert" pod "3db8ac79-76fc-4d46-ac4f-6536fe192b8f" (UID: "3db8ac79-76fc-4d46-ac4f-6536fe192b8f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:38:03.316125 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:03.315958    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/746cdf44-bdd0-4040-87a9-43ff9b48cf99-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fa4571df-5g52m\" (UID: \"746cdf44-bdd0-4040-87a9-43ff9b48cf99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa4571df-5g52m"
Nov 03 20:38:03.316125 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:03.316048    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ht9dc\" (UniqueName: \"kubernetes.io/projected/746cdf44-bdd0-4040-87a9-43ff9b48cf99-kube-api-access-ht9dc\") pod \"maas-api-revoke-sub-fa4571df-5g52m\" (UID: \"746cdf44-bdd0-4040-87a9-43ff9b48cf99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa4571df-5g52m"
Nov 03 20:38:03.316125 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:03.316057    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3db8ac79-76fc-4d46-ac4f-6536fe192b8f-credential-socket" pod "3db8ac79-76fc-4d46-ac4f-6536fe192b8f" (UID: "3db8ac79-76fc-4d46-ac4f-6536fe192b8f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:38:03.316367 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:03.316253    5570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3db8ac79-76fc-4d46-ac4f-6536fe192b8f-istiod-ca-cert\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:38:03.316367 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:03.316284    5570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3db8ac79-76fc-4d46-ac4f-6536fe192b8f-istio-data\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:38:03.316367 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:03.316300    5570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3db8ac79-76fc-4d46-ac4f-6536fe192b8f-workload-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:38:03.316367 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:03.316314    5570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3db8ac79-76fc-4d46-ac4f-6536fe192b8f-credential-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:38:03.317166 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:03.317073    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/746cdf44-bdd0-4040-87a9-43ff9b48cf99-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fa4571df-5g52m\" (UID: \"746cdf44-bdd0-4040-87a9-43ff9b48cf99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa4571df-5g52m"
Nov 03 20:38:03.317465 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:03.317437    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3db8ac79-76fc-4d46-ac4f-6536fe192b8f-workload-socket" pod "3db8ac79-76fc-4d46-ac4f-6536fe192b8f" (UID: "3db8ac79-76fc-4d46-ac4f-6536fe192b8f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:38:03.318245 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:03.318220    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3db8ac79-76fc-4d46-ac4f-6536fe192b8f-istio-envoy" pod "3db8ac79-76fc-4d46-ac4f-6536fe192b8f" (UID: "3db8ac79-76fc-4d46-ac4f-6536fe192b8f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:38:03.318245 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:03.318236    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3db8ac79-76fc-4d46-ac4f-6536fe192b8f-istio-token" pod "3db8ac79-76fc-4d46-ac4f-6536fe192b8f" (UID: "3db8ac79-76fc-4d46-ac4f-6536fe192b8f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:38:03.318428 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:03.318380    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3db8ac79-76fc-4d46-ac4f-6536fe192b8f-kube-api-access-v7nl2" pod "3db8ac79-76fc-4d46-ac4f-6536fe192b8f" (UID: "3db8ac79-76fc-4d46-ac4f-6536fe192b8f"). InnerVolumeSpecName "kube-api-access-v7nl2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:38:03.318518 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:03.318497    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/746cdf44-bdd0-4040-87a9-43ff9b48cf99-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fa4571df-5g52m\" (UID: \"746cdf44-bdd0-4040-87a9-43ff9b48cf99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa4571df-5g52m"
Nov 03 20:38:03.318910 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:03.318890    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3db8ac79-76fc-4d46-ac4f-6536fe192b8f-istio-podinfo" pod "3db8ac79-76fc-4d46-ac4f-6536fe192b8f" (UID: "3db8ac79-76fc-4d46-ac4f-6536fe192b8f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:38:03.336439 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:03.336403    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ht9dc\" (UniqueName: \"kubernetes.io/projected/746cdf44-bdd0-4040-87a9-43ff9b48cf99-kube-api-access-ht9dc\") pod \"maas-api-revoke-sub-fa4571df-5g52m\" (UID: \"746cdf44-bdd0-4040-87a9-43ff9b48cf99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa4571df-5g52m"
Nov 03 20:38:03.417680 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:03.417637    5570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3db8ac79-76fc-4d46-ac4f-6536fe192b8f-workload-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:38:03.417680 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:03.417668    5570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3db8ac79-76fc-4d46-ac4f-6536fe192b8f-istio-envoy\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:38:03.417680 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:03.417680    5570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3db8ac79-76fc-4d46-ac4f-6536fe192b8f-istio-token\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:38:03.417680 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:03.417692    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7nl2\" (UniqueName: \"kubernetes.io/projected/3db8ac79-76fc-4d46-ac4f-6536fe192b8f-kube-api-access-v7nl2\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:38:03.418013 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:03.417702    5570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3db8ac79-76fc-4d46-ac4f-6536fe192b8f-istio-podinfo\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:38:03.424602 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:03.424579    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa4571df-5g52m"
Nov 03 20:38:03.719973 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:03.719941    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa4571df-5g52m"]
Nov 03 20:38:03.721418 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:38:03.721394    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod746cdf44_bdd0_4040_87a9_43ff9b48cf99.slice/crio-944ca01d8e9f0cfaaf9b02bf5c415fb24e48c3f68419482d64edaa0c897797c9 WatchSource:0}: Error finding container 944ca01d8e9f0cfaaf9b02bf5c415fb24e48c3f68419482d64edaa0c897797c9: Status 404 returned error can't find the container with id 944ca01d8e9f0cfaaf9b02bf5c415fb24e48c3f68419482d64edaa0c897797c9
Nov 03 20:38:04.192849 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:04.192799    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa4571df-5g52m" event={"ID":"746cdf44-bdd0-4040-87a9-43ff9b48cf99","Type":"ContainerStarted","Data":"692cceb18d53e33b40c86198829996e7618e74182139285f83968bf7b8263a99"}
Nov 03 20:38:04.193062 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:04.192868    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa4571df-5g52m" event={"ID":"746cdf44-bdd0-4040-87a9-43ff9b48cf99","Type":"ContainerStarted","Data":"944ca01d8e9f0cfaaf9b02bf5c415fb24e48c3f68419482d64edaa0c897797c9"}
Nov 03 20:38:04.194332 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:04.194308    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-76b894-openshift-default-6669d4d94f-vrq8z"
Nov 03 20:38:04.194486 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:04.194305    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-76b894-openshift-default-6669d4d94f-vrq8z" event={"ID":"3db8ac79-76fc-4d46-ac4f-6536fe192b8f","Type":"ContainerDied","Data":"f117588d69672e70ba1262cea73f2561329718707205e5e418ca24375cb1f8cd"}
Nov 03 20:38:04.194486 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:04.194433    5570 scope.go:122] "RemoveContainer" containerID="08e6c055f3b06fb5e23c8d8740789d057e79fe1612f7eeefc46ba3e8200254e9"
Nov 03 20:38:04.235091 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:04.234978    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa4571df-5g52m" podStartSLOduration=1.23496265 podStartE2EDuration="1.23496265s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:38:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:38:04.233174822 +0000 UTC m=+2018.385836708" watchObservedRunningTime="2026-11-03 20:38:04.23496265 +0000 UTC m=+2018.387624535"
Nov 03 20:38:04.298051 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:04.298013    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-76b894-openshift-default-6669d4d94f-vrq8z"]
Nov 03 20:38:04.303176 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:04.303146    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-76b894-openshift-default-6669d4d94f-vrq8z"]
Nov 03 20:38:05.202958 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:05.202924    5570 generic.go:356] "Generic (PLEG): container finished" podID="746cdf44-bdd0-4040-87a9-43ff9b48cf99" containerID="692cceb18d53e33b40c86198829996e7618e74182139285f83968bf7b8263a99" exitCode=0
Nov 03 20:38:05.203408 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:05.202982    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa4571df-5g52m" event={"ID":"746cdf44-bdd0-4040-87a9-43ff9b48cf99","Type":"ContainerDied","Data":"692cceb18d53e33b40c86198829996e7618e74182139285f83968bf7b8263a99"}
Nov 03 20:38:05.964073 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:05.964037    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3db8ac79-76fc-4d46-ac4f-6536fe192b8f" path="/var/lib/kubelet/pods/3db8ac79-76fc-4d46-ac4f-6536fe192b8f/volumes"
Nov 03 20:38:06.350104 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:06.350070    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa4571df-5g52m"
Nov 03 20:38:06.447450 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:06.447405    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/746cdf44-bdd0-4040-87a9-43ff9b48cf99-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/746cdf44-bdd0-4040-87a9-43ff9b48cf99-maas-api-xks-ca\") pod \"746cdf44-bdd0-4040-87a9-43ff9b48cf99\" (UID: \"746cdf44-bdd0-4040-87a9-43ff9b48cf99\") "
Nov 03 20:38:06.447649 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:06.447469    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/746cdf44-bdd0-4040-87a9-43ff9b48cf99-kube-api-access-ht9dc\" (UniqueName: \"kubernetes.io/projected/746cdf44-bdd0-4040-87a9-43ff9b48cf99-kube-api-access-ht9dc\") pod \"746cdf44-bdd0-4040-87a9-43ff9b48cf99\" (UID: \"746cdf44-bdd0-4040-87a9-43ff9b48cf99\") "
Nov 03 20:38:06.447649 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:06.447498    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/746cdf44-bdd0-4040-87a9-43ff9b48cf99-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/746cdf44-bdd0-4040-87a9-43ff9b48cf99-maas-api-openshift-ca\") pod \"746cdf44-bdd0-4040-87a9-43ff9b48cf99\" (UID: \"746cdf44-bdd0-4040-87a9-43ff9b48cf99\") "
Nov 03 20:38:06.447974 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:06.447941    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/746cdf44-bdd0-4040-87a9-43ff9b48cf99-maas-api-openshift-ca" pod "746cdf44-bdd0-4040-87a9-43ff9b48cf99" (UID: "746cdf44-bdd0-4040-87a9-43ff9b48cf99"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:38:06.449654 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:06.449628    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/746cdf44-bdd0-4040-87a9-43ff9b48cf99-maas-api-xks-ca" pod "746cdf44-bdd0-4040-87a9-43ff9b48cf99" (UID: "746cdf44-bdd0-4040-87a9-43ff9b48cf99"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:38:06.449745 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:06.449725    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/746cdf44-bdd0-4040-87a9-43ff9b48cf99-kube-api-access-ht9dc" pod "746cdf44-bdd0-4040-87a9-43ff9b48cf99" (UID: "746cdf44-bdd0-4040-87a9-43ff9b48cf99"). InnerVolumeSpecName "kube-api-access-ht9dc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:38:06.549367 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:06.549273    5570 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/746cdf44-bdd0-4040-87a9-43ff9b48cf99-maas-api-xks-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:38:06.549367 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:06.549308    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ht9dc\" (UniqueName: \"kubernetes.io/projected/746cdf44-bdd0-4040-87a9-43ff9b48cf99-kube-api-access-ht9dc\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:38:06.549367 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:06.549323    5570 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/746cdf44-bdd0-4040-87a9-43ff9b48cf99-maas-api-openshift-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:38:07.216690 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:07.216660    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa4571df-5g52m"
Nov 03 20:38:07.216880 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:07.216650    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa4571df-5g52m" event={"ID":"746cdf44-bdd0-4040-87a9-43ff9b48cf99","Type":"ContainerDied","Data":"944ca01d8e9f0cfaaf9b02bf5c415fb24e48c3f68419482d64edaa0c897797c9"}
Nov 03 20:38:07.216880 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:07.216800    5570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="944ca01d8e9f0cfaaf9b02bf5c415fb24e48c3f68419482d64edaa0c897797c9"
Nov 03 20:38:09.649239 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:09.649202    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d1bfb8-7b6cb98c-pkvgg"]
Nov 03 20:38:09.650050 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:09.650012    5570 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d1bfb8-7b6cb98c-pkvgg" podUID="a7c7754a-0ea1-45c7-825e-d527b1ccc1eb" containerName="maas-api" containerID="cri-o://c3e62fd6b5fa8b1021d309d18e1ce52272e7800a4cd8dd9b176ac711d481de7f" gracePeriod=30
Nov 03 20:38:09.933478 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:09.933445    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d1bfb8-7b6cb98c-pkvgg"
Nov 03 20:38:10.125942 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:10.125896    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7c7754a-0ea1-45c7-825e-d527b1ccc1eb-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7c7754a-0ea1-45c7-825e-d527b1ccc1eb-openshift-service-ca\") pod \"a7c7754a-0ea1-45c7-825e-d527b1ccc1eb\" (UID: \"a7c7754a-0ea1-45c7-825e-d527b1ccc1eb\") "
Nov 03 20:38:10.125942 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:10.125947    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7c7754a-0ea1-45c7-825e-d527b1ccc1eb-kube-api-access-jtbml\" (UniqueName: \"kubernetes.io/projected/a7c7754a-0ea1-45c7-825e-d527b1ccc1eb-kube-api-access-jtbml\") pod \"a7c7754a-0ea1-45c7-825e-d527b1ccc1eb\" (UID: \"a7c7754a-0ea1-45c7-825e-d527b1ccc1eb\") "
Nov 03 20:38:10.126198 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:10.126051    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a7c7754a-0ea1-45c7-825e-d527b1ccc1eb-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a7c7754a-0ea1-45c7-825e-d527b1ccc1eb-maas-api-tls\") pod \"a7c7754a-0ea1-45c7-825e-d527b1ccc1eb\" (UID: \"a7c7754a-0ea1-45c7-825e-d527b1ccc1eb\") "
Nov 03 20:38:10.126198 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:10.126142    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a7c7754a-0ea1-45c7-825e-d527b1ccc1eb-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a7c7754a-0ea1-45c7-825e-d527b1ccc1eb-metrics-certs\") pod \"a7c7754a-0ea1-45c7-825e-d527b1ccc1eb\" (UID: \"a7c7754a-0ea1-45c7-825e-d527b1ccc1eb\") "
Nov 03 20:38:10.126378 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:10.126350    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7c7754a-0ea1-45c7-825e-d527b1ccc1eb-openshift-service-ca" pod "a7c7754a-0ea1-45c7-825e-d527b1ccc1eb" (UID: "a7c7754a-0ea1-45c7-825e-d527b1ccc1eb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:38:10.126585 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:10.126561    5570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7c7754a-0ea1-45c7-825e-d527b1ccc1eb-openshift-service-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:38:10.128355 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:10.128315    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7c7754a-0ea1-45c7-825e-d527b1ccc1eb-kube-api-access-jtbml" pod "a7c7754a-0ea1-45c7-825e-d527b1ccc1eb" (UID: "a7c7754a-0ea1-45c7-825e-d527b1ccc1eb"). InnerVolumeSpecName "kube-api-access-jtbml". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:38:10.128355 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:10.128335    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7c7754a-0ea1-45c7-825e-d527b1ccc1eb-metrics-certs" pod "a7c7754a-0ea1-45c7-825e-d527b1ccc1eb" (UID: "a7c7754a-0ea1-45c7-825e-d527b1ccc1eb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:38:10.128486 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:10.128398    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7c7754a-0ea1-45c7-825e-d527b1ccc1eb-maas-api-tls" pod "a7c7754a-0ea1-45c7-825e-d527b1ccc1eb" (UID: "a7c7754a-0ea1-45c7-825e-d527b1ccc1eb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:38:10.227270 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:10.227233    5570 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a7c7754a-0ea1-45c7-825e-d527b1ccc1eb-maas-api-tls\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:38:10.227270 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:10.227266    5570 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a7c7754a-0ea1-45c7-825e-d527b1ccc1eb-metrics-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:38:10.227270 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:10.227281    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jtbml\" (UniqueName: \"kubernetes.io/projected/a7c7754a-0ea1-45c7-825e-d527b1ccc1eb-kube-api-access-jtbml\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:38:10.237511 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:10.237474    5570 generic.go:356] "Generic (PLEG): container finished" podID="a7c7754a-0ea1-45c7-825e-d527b1ccc1eb" containerID="c3e62fd6b5fa8b1021d309d18e1ce52272e7800a4cd8dd9b176ac711d481de7f" exitCode=0
Nov 03 20:38:10.237716 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:10.237523    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d1bfb8-7b6cb98c-pkvgg"
Nov 03 20:38:10.237716 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:10.237548    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d1bfb8-7b6cb98c-pkvgg" event={"ID":"a7c7754a-0ea1-45c7-825e-d527b1ccc1eb","Type":"ContainerDied","Data":"c3e62fd6b5fa8b1021d309d18e1ce52272e7800a4cd8dd9b176ac711d481de7f"}
Nov 03 20:38:10.237716 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:10.237601    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d1bfb8-7b6cb98c-pkvgg" event={"ID":"a7c7754a-0ea1-45c7-825e-d527b1ccc1eb","Type":"ContainerDied","Data":"9da2bd25638efa14c3011e4ac35191e0c7b3e9e8b1d6bdfbddb98bd0361312c6"}
Nov 03 20:38:10.237716 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:10.237629    5570 scope.go:122] "RemoveContainer" containerID="c3e62fd6b5fa8b1021d309d18e1ce52272e7800a4cd8dd9b176ac711d481de7f"
Nov 03 20:38:10.250854 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:10.250834    5570 scope.go:122] "RemoveContainer" containerID="c3e62fd6b5fa8b1021d309d18e1ce52272e7800a4cd8dd9b176ac711d481de7f"
Nov 03 20:38:10.251210 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:38:10.251189    5570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3e62fd6b5fa8b1021d309d18e1ce52272e7800a4cd8dd9b176ac711d481de7f\": container with ID starting with c3e62fd6b5fa8b1021d309d18e1ce52272e7800a4cd8dd9b176ac711d481de7f not found: ID does not exist" containerID="c3e62fd6b5fa8b1021d309d18e1ce52272e7800a4cd8dd9b176ac711d481de7f"
Nov 03 20:38:10.251269 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:10.251224    5570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3e62fd6b5fa8b1021d309d18e1ce52272e7800a4cd8dd9b176ac711d481de7f"} err="failed to get container status \"c3e62fd6b5fa8b1021d309d18e1ce52272e7800a4cd8dd9b176ac711d481de7f\": rpc error: code = NotFound desc = could not find container \"c3e62fd6b5fa8b1021d309d18e1ce52272e7800a4cd8dd9b176ac711d481de7f\": container with ID starting with c3e62fd6b5fa8b1021d309d18e1ce52272e7800a4cd8dd9b176ac711d481de7f not found: ID does not exist"
Nov 03 20:38:10.292115 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:10.292015    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d1bfb8-7b6cb98c-pkvgg"]
Nov 03 20:38:10.297313 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:10.297277    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d1bfb8-7b6cb98c-pkvgg"]
Nov 03 20:38:11.979867 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:11.979825    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7c7754a-0ea1-45c7-825e-d527b1ccc1eb" path="/var/lib/kubelet/pods/a7c7754a-0ea1-45c7-825e-d527b1ccc1eb/volumes"
Nov 03 20:38:12.318920 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:12.318814    5570 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-e80203/distinct-2-w4-e80203-kserve-756f9b9bc6-69gxj" podUID="ccf5dc2f-f116-4917-bb1a-b136d8ed17f6" containerName="main" containerID="cri-o://54522cc77363c1530edc0a1e0af7881d460018cab0c8183826ac56ac16a32f79" gracePeriod=45
Nov 03 20:38:12.335000 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:12.334960    5570 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-e80203/premium-w4-e80203-kserve-5f9f445456-bms9d" podUID="6ad76efc-20c2-4951-b22a-c5a39cd39ed4" containerName="main" containerID="cri-o://c3b73dcf7616cb22723e91b2213cf6b65ff8a87c72b5883c75303046ba91cc20" gracePeriod=45
Nov 03 20:38:12.597043 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:12.597012    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-e80203/distinct-2-w4-e80203-kserve-756f9b9bc6-69gxj"
Nov 03 20:38:12.622360 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:12.622331    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-e80203/premium-w4-e80203-kserve-5f9f445456-bms9d"
Nov 03 20:38:12.648504 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:12.648463    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ccf5dc2f-f116-4917-bb1a-b136d8ed17f6-tls-certs\" (UniqueName: \"kubernetes.io/secret/ccf5dc2f-f116-4917-bb1a-b136d8ed17f6-tls-certs\") pod \"ccf5dc2f-f116-4917-bb1a-b136d8ed17f6\" (UID: \"ccf5dc2f-f116-4917-bb1a-b136d8ed17f6\") "
Nov 03 20:38:12.648711 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:12.648529    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ad76efc-20c2-4951-b22a-c5a39cd39ed4-kube-api-access-dprnk\" (UniqueName: \"kubernetes.io/projected/6ad76efc-20c2-4951-b22a-c5a39cd39ed4-kube-api-access-dprnk\") pod \"6ad76efc-20c2-4951-b22a-c5a39cd39ed4\" (UID: \"6ad76efc-20c2-4951-b22a-c5a39cd39ed4\") "
Nov 03 20:38:12.648711 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:12.648595    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccf5dc2f-f116-4917-bb1a-b136d8ed17f6-kube-api-access-pwf85\" (UniqueName: \"kubernetes.io/projected/ccf5dc2f-f116-4917-bb1a-b136d8ed17f6-kube-api-access-pwf85\") pod \"ccf5dc2f-f116-4917-bb1a-b136d8ed17f6\" (UID: \"ccf5dc2f-f116-4917-bb1a-b136d8ed17f6\") "
Nov 03 20:38:12.648711 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:12.648631    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccf5dc2f-f116-4917-bb1a-b136d8ed17f6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ccf5dc2f-f116-4917-bb1a-b136d8ed17f6-dshm\") pod \"ccf5dc2f-f116-4917-bb1a-b136d8ed17f6\" (UID: \"ccf5dc2f-f116-4917-bb1a-b136d8ed17f6\") "
Nov 03 20:38:12.648711 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:12.648661    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccf5dc2f-f116-4917-bb1a-b136d8ed17f6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ccf5dc2f-f116-4917-bb1a-b136d8ed17f6-model-cache\") pod \"ccf5dc2f-f116-4917-bb1a-b136d8ed17f6\" (UID: \"ccf5dc2f-f116-4917-bb1a-b136d8ed17f6\") "
Nov 03 20:38:12.648711 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:12.648691    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ad76efc-20c2-4951-b22a-c5a39cd39ed4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ad76efc-20c2-4951-b22a-c5a39cd39ed4-dshm\") pod \"6ad76efc-20c2-4951-b22a-c5a39cd39ed4\" (UID: \"6ad76efc-20c2-4951-b22a-c5a39cd39ed4\") "
Nov 03 20:38:12.649034 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:12.648766    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ad76efc-20c2-4951-b22a-c5a39cd39ed4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ad76efc-20c2-4951-b22a-c5a39cd39ed4-model-cache\") pod \"6ad76efc-20c2-4951-b22a-c5a39cd39ed4\" (UID: \"6ad76efc-20c2-4951-b22a-c5a39cd39ed4\") "
Nov 03 20:38:12.649034 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:12.648826    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccf5dc2f-f116-4917-bb1a-b136d8ed17f6-home\" (UniqueName: \"kubernetes.io/empty-dir/ccf5dc2f-f116-4917-bb1a-b136d8ed17f6-home\") pod \"ccf5dc2f-f116-4917-bb1a-b136d8ed17f6\" (UID: \"ccf5dc2f-f116-4917-bb1a-b136d8ed17f6\") "
Nov 03 20:38:12.649034 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:12.648895    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ad76efc-20c2-4951-b22a-c5a39cd39ed4-home\" (UniqueName: \"kubernetes.io/empty-dir/6ad76efc-20c2-4951-b22a-c5a39cd39ed4-home\") pod \"6ad76efc-20c2-4951-b22a-c5a39cd39ed4\" (UID: \"6ad76efc-20c2-4951-b22a-c5a39cd39ed4\") "
Nov 03 20:38:12.649034 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:12.648941    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ad76efc-20c2-4951-b22a-c5a39cd39ed4-tls-certs\" (UniqueName: \"kubernetes.io/secret/6ad76efc-20c2-4951-b22a-c5a39cd39ed4-tls-certs\") pod \"6ad76efc-20c2-4951-b22a-c5a39cd39ed4\" (UID: \"6ad76efc-20c2-4951-b22a-c5a39cd39ed4\") "
Nov 03 20:38:12.649269 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:12.649042    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ad76efc-20c2-4951-b22a-c5a39cd39ed4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ad76efc-20c2-4951-b22a-c5a39cd39ed4-tmp-dir\") pod \"6ad76efc-20c2-4951-b22a-c5a39cd39ed4\" (UID: \"6ad76efc-20c2-4951-b22a-c5a39cd39ed4\") "
Nov 03 20:38:12.649269 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:12.649214    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccf5dc2f-f116-4917-bb1a-b136d8ed17f6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ccf5dc2f-f116-4917-bb1a-b136d8ed17f6-tmp-dir\") pod \"ccf5dc2f-f116-4917-bb1a-b136d8ed17f6\" (UID: \"ccf5dc2f-f116-4917-bb1a-b136d8ed17f6\") "
Nov 03 20:38:12.649982 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:12.649954    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccf5dc2f-f116-4917-bb1a-b136d8ed17f6-tmp-dir" pod "ccf5dc2f-f116-4917-bb1a-b136d8ed17f6" (UID: "ccf5dc2f-f116-4917-bb1a-b136d8ed17f6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:38:12.651146 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:12.650506    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ad76efc-20c2-4951-b22a-c5a39cd39ed4-tmp-dir" pod "6ad76efc-20c2-4951-b22a-c5a39cd39ed4" (UID: "6ad76efc-20c2-4951-b22a-c5a39cd39ed4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:38:12.651146 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:12.650591    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ad76efc-20c2-4951-b22a-c5a39cd39ed4-home" pod "6ad76efc-20c2-4951-b22a-c5a39cd39ed4" (UID: "6ad76efc-20c2-4951-b22a-c5a39cd39ed4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:38:12.651146 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:12.650593    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccf5dc2f-f116-4917-bb1a-b136d8ed17f6-model-cache" pod "ccf5dc2f-f116-4917-bb1a-b136d8ed17f6" (UID: "ccf5dc2f-f116-4917-bb1a-b136d8ed17f6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:38:12.651146 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:12.650941    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccf5dc2f-f116-4917-bb1a-b136d8ed17f6-home" pod "ccf5dc2f-f116-4917-bb1a-b136d8ed17f6" (UID: "ccf5dc2f-f116-4917-bb1a-b136d8ed17f6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:38:12.651146 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:12.651124    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ad76efc-20c2-4951-b22a-c5a39cd39ed4-model-cache" pod "6ad76efc-20c2-4951-b22a-c5a39cd39ed4" (UID: "6ad76efc-20c2-4951-b22a-c5a39cd39ed4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:38:12.651764 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:12.651457    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ccf5dc2f-f116-4917-bb1a-b136d8ed17f6-tls-certs" pod "ccf5dc2f-f116-4917-bb1a-b136d8ed17f6" (UID: "ccf5dc2f-f116-4917-bb1a-b136d8ed17f6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:38:12.653205 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:12.653177    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ad76efc-20c2-4951-b22a-c5a39cd39ed4-tls-certs" pod "6ad76efc-20c2-4951-b22a-c5a39cd39ed4" (UID: "6ad76efc-20c2-4951-b22a-c5a39cd39ed4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:38:12.653562 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:12.653545    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ad76efc-20c2-4951-b22a-c5a39cd39ed4-kube-api-access-dprnk" pod "6ad76efc-20c2-4951-b22a-c5a39cd39ed4" (UID: "6ad76efc-20c2-4951-b22a-c5a39cd39ed4"). InnerVolumeSpecName "kube-api-access-dprnk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:38:12.654712 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:12.654688    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ad76efc-20c2-4951-b22a-c5a39cd39ed4-dshm" pod "6ad76efc-20c2-4951-b22a-c5a39cd39ed4" (UID: "6ad76efc-20c2-4951-b22a-c5a39cd39ed4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:38:12.655096 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:12.655064    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccf5dc2f-f116-4917-bb1a-b136d8ed17f6-kube-api-access-pwf85" pod "ccf5dc2f-f116-4917-bb1a-b136d8ed17f6" (UID: "ccf5dc2f-f116-4917-bb1a-b136d8ed17f6"). InnerVolumeSpecName "kube-api-access-pwf85". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:38:12.655430 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:12.655407    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccf5dc2f-f116-4917-bb1a-b136d8ed17f6-dshm" pod "ccf5dc2f-f116-4917-bb1a-b136d8ed17f6" (UID: "ccf5dc2f-f116-4917-bb1a-b136d8ed17f6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:38:12.750247 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:12.750212    5570 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ccf5dc2f-f116-4917-bb1a-b136d8ed17f6-home\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:38:12.750247 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:12.750243    5570 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ad76efc-20c2-4951-b22a-c5a39cd39ed4-home\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:38:12.750247 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:12.750252    5570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ad76efc-20c2-4951-b22a-c5a39cd39ed4-tls-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:38:12.750507 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:12.750266    5570 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ad76efc-20c2-4951-b22a-c5a39cd39ed4-tmp-dir\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:38:12.750507 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:12.750278    5570 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ccf5dc2f-f116-4917-bb1a-b136d8ed17f6-tmp-dir\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:38:12.750507 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:12.750286    5570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ccf5dc2f-f116-4917-bb1a-b136d8ed17f6-tls-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:38:12.750507 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:12.750294    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dprnk\" (UniqueName: \"kubernetes.io/projected/6ad76efc-20c2-4951-b22a-c5a39cd39ed4-kube-api-access-dprnk\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:38:12.750507 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:12.750306    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwf85\" (UniqueName: \"kubernetes.io/projected/ccf5dc2f-f116-4917-bb1a-b136d8ed17f6-kube-api-access-pwf85\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:38:12.750507 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:12.750314    5570 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ccf5dc2f-f116-4917-bb1a-b136d8ed17f6-dshm\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:38:12.750507 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:12.750322    5570 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ccf5dc2f-f116-4917-bb1a-b136d8ed17f6-model-cache\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:38:12.750507 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:12.750329    5570 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ad76efc-20c2-4951-b22a-c5a39cd39ed4-dshm\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:38:12.750507 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:12.750336    5570 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ad76efc-20c2-4951-b22a-c5a39cd39ed4-model-cache\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:38:13.256347 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:13.256309    5570 generic.go:356] "Generic (PLEG): container finished" podID="ccf5dc2f-f116-4917-bb1a-b136d8ed17f6" containerID="54522cc77363c1530edc0a1e0af7881d460018cab0c8183826ac56ac16a32f79" exitCode=0
Nov 03 20:38:13.256848 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:13.256364    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-e80203/distinct-2-w4-e80203-kserve-756f9b9bc6-69gxj"
Nov 03 20:38:13.256848 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:13.256394    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-e80203/distinct-2-w4-e80203-kserve-756f9b9bc6-69gxj" event={"ID":"ccf5dc2f-f116-4917-bb1a-b136d8ed17f6","Type":"ContainerDied","Data":"54522cc77363c1530edc0a1e0af7881d460018cab0c8183826ac56ac16a32f79"}
Nov 03 20:38:13.256848 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:13.256436    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-e80203/distinct-2-w4-e80203-kserve-756f9b9bc6-69gxj" event={"ID":"ccf5dc2f-f116-4917-bb1a-b136d8ed17f6","Type":"ContainerDied","Data":"50bf087bc34fa4520a8a0bb9461a0bea8c2c409be168d1aa2031ece8fc0a95d3"}
Nov 03 20:38:13.256848 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:13.256453    5570 scope.go:122] "RemoveContainer" containerID="54522cc77363c1530edc0a1e0af7881d460018cab0c8183826ac56ac16a32f79"
Nov 03 20:38:13.258021 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:13.257997    5570 generic.go:356] "Generic (PLEG): container finished" podID="6ad76efc-20c2-4951-b22a-c5a39cd39ed4" containerID="c3b73dcf7616cb22723e91b2213cf6b65ff8a87c72b5883c75303046ba91cc20" exitCode=0
Nov 03 20:38:13.258149 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:13.258044    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-e80203/premium-w4-e80203-kserve-5f9f445456-bms9d"
Nov 03 20:38:13.258149 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:13.258028    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-e80203/premium-w4-e80203-kserve-5f9f445456-bms9d" event={"ID":"6ad76efc-20c2-4951-b22a-c5a39cd39ed4","Type":"ContainerDied","Data":"c3b73dcf7616cb22723e91b2213cf6b65ff8a87c72b5883c75303046ba91cc20"}
Nov 03 20:38:13.258298 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:13.258150    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-e80203/premium-w4-e80203-kserve-5f9f445456-bms9d" event={"ID":"6ad76efc-20c2-4951-b22a-c5a39cd39ed4","Type":"ContainerDied","Data":"80804640db3c0c3ec17bff5c409a25e8034e0fbf6eb3d7bef4ed3c03a4d241f1"}
Nov 03 20:38:13.266619 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:13.266596    5570 scope.go:122] "RemoveContainer" containerID="54522cc77363c1530edc0a1e0af7881d460018cab0c8183826ac56ac16a32f79"
Nov 03 20:38:13.267053 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:38:13.267026    5570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54522cc77363c1530edc0a1e0af7881d460018cab0c8183826ac56ac16a32f79\": container with ID starting with 54522cc77363c1530edc0a1e0af7881d460018cab0c8183826ac56ac16a32f79 not found: ID does not exist" containerID="54522cc77363c1530edc0a1e0af7881d460018cab0c8183826ac56ac16a32f79"
Nov 03 20:38:13.267173 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:13.267065    5570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54522cc77363c1530edc0a1e0af7881d460018cab0c8183826ac56ac16a32f79"} err="failed to get container status \"54522cc77363c1530edc0a1e0af7881d460018cab0c8183826ac56ac16a32f79\": rpc error: code = NotFound desc = could not find container \"54522cc77363c1530edc0a1e0af7881d460018cab0c8183826ac56ac16a32f79\": container with ID starting with 54522cc77363c1530edc0a1e0af7881d460018cab0c8183826ac56ac16a32f79 not found: ID does not exist"
Nov 03 20:38:13.267173 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:13.267091    5570 scope.go:122] "RemoveContainer" containerID="c3b73dcf7616cb22723e91b2213cf6b65ff8a87c72b5883c75303046ba91cc20"
Nov 03 20:38:13.277497 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:13.277459    5570 scope.go:122] "RemoveContainer" containerID="c3b73dcf7616cb22723e91b2213cf6b65ff8a87c72b5883c75303046ba91cc20"
Nov 03 20:38:13.277849 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:38:13.277826    5570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3b73dcf7616cb22723e91b2213cf6b65ff8a87c72b5883c75303046ba91cc20\": container with ID starting with c3b73dcf7616cb22723e91b2213cf6b65ff8a87c72b5883c75303046ba91cc20 not found: ID does not exist" containerID="c3b73dcf7616cb22723e91b2213cf6b65ff8a87c72b5883c75303046ba91cc20"
Nov 03 20:38:13.277927 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:13.277858    5570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3b73dcf7616cb22723e91b2213cf6b65ff8a87c72b5883c75303046ba91cc20"} err="failed to get container status \"c3b73dcf7616cb22723e91b2213cf6b65ff8a87c72b5883c75303046ba91cc20\": rpc error: code = NotFound desc = could not find container \"c3b73dcf7616cb22723e91b2213cf6b65ff8a87c72b5883c75303046ba91cc20\": container with ID starting with c3b73dcf7616cb22723e91b2213cf6b65ff8a87c72b5883c75303046ba91cc20 not found: ID does not exist"
Nov 03 20:38:13.309221 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:13.309190    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-e80203/premium-w4-e80203-kserve-5f9f445456-bms9d"]
Nov 03 20:38:13.322744 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:13.322704    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-e80203/premium-w4-e80203-kserve-5f9f445456-bms9d"]
Nov 03 20:38:13.358520 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:13.358485    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-e80203/distinct-2-w4-e80203-kserve-756f9b9bc6-69gxj"]
Nov 03 20:38:13.362290 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:13.362260    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-e80203/distinct-2-w4-e80203-kserve-756f9b9bc6-69gxj"]
Nov 03 20:38:13.537885 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:13.537766    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e80203/premium-w4-e80203-kserve-5f9f445456-bms9d" podUID="6ad76efc-20c2-4951-b22a-c5a39cd39ed4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:38:13.963742 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:13.963700    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ad76efc-20c2-4951-b22a-c5a39cd39ed4" path="/var/lib/kubelet/pods/6ad76efc-20c2-4951-b22a-c5a39cd39ed4/volumes"
Nov 03 20:38:13.964257 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:13.964234    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ccf5dc2f-f116-4917-bb1a-b136d8ed17f6" path="/var/lib/kubelet/pods/ccf5dc2f-f116-4917-bb1a-b136d8ed17f6/volumes"
Nov 03 20:38:14.732879 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:14.732836    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g69f8"]
Nov 03 20:38:21.445207 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:21.445171    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-482722-lvwgw"]
Nov 03 20:38:21.445600 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:21.445580    5570 cpu_manager.go:414] "containerMap: removing container" podUID="3db8ac79-76fc-4d46-ac4f-6536fe192b8f" containerName="istio-proxy"
Nov 03 20:38:21.445600 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:21.445592    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3db8ac79-76fc-4d46-ac4f-6536fe192b8f" containerName="istio-proxy"
Nov 03 20:38:21.445690 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:21.445619    5570 cpu_manager.go:414] "containerMap: removing container" podUID="a7c7754a-0ea1-45c7-825e-d527b1ccc1eb" containerName="maas-api"
Nov 03 20:38:21.445690 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:21.445624    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7c7754a-0ea1-45c7-825e-d527b1ccc1eb" containerName="maas-api"
Nov 03 20:38:21.445690 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:21.445632    5570 cpu_manager.go:414] "containerMap: removing container" podUID="ccf5dc2f-f116-4917-bb1a-b136d8ed17f6" containerName="main"
Nov 03 20:38:21.445690 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:21.445637    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccf5dc2f-f116-4917-bb1a-b136d8ed17f6" containerName="main"
Nov 03 20:38:21.445690 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:21.445647    5570 cpu_manager.go:414] "containerMap: removing container" podUID="746cdf44-bdd0-4040-87a9-43ff9b48cf99" containerName="revoke-keys"
Nov 03 20:38:21.445690 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:21.445651    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="746cdf44-bdd0-4040-87a9-43ff9b48cf99" containerName="revoke-keys"
Nov 03 20:38:21.445690 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:21.445670    5570 cpu_manager.go:414] "containerMap: removing container" podUID="6ad76efc-20c2-4951-b22a-c5a39cd39ed4" containerName="main"
Nov 03 20:38:21.445690 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:21.445674    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ad76efc-20c2-4951-b22a-c5a39cd39ed4" containerName="main"
Nov 03 20:38:21.445963 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:21.445725    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="3db8ac79-76fc-4d46-ac4f-6536fe192b8f" containerName="istio-proxy"
Nov 03 20:38:21.445963 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:21.445736    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ad76efc-20c2-4951-b22a-c5a39cd39ed4" containerName="main"
Nov 03 20:38:21.445963 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:21.445742    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="3db8ac79-76fc-4d46-ac4f-6536fe192b8f" containerName="istio-proxy"
Nov 03 20:38:21.445963 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:21.445753    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7c7754a-0ea1-45c7-825e-d527b1ccc1eb" containerName="maas-api"
Nov 03 20:38:21.445963 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:21.445759    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccf5dc2f-f116-4917-bb1a-b136d8ed17f6" containerName="main"
Nov 03 20:38:21.445963 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:21.445766    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="746cdf44-bdd0-4040-87a9-43ff9b48cf99" containerName="revoke-keys"
Nov 03 20:38:21.448683 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:21.448664    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-482722-lvwgw"
Nov 03 20:38:21.466406 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:21.466374    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-482722-lvwgw"]
Nov 03 20:38:21.477051 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:21.477018    5570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 20:38:21.477051 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:21.477018    5570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 20:38:21.501269 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:21.501232    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l7s5w\""
Nov 03 20:38:21.501269 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:21.501258    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:38:21.539879 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:21.539826    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnsfl\" (UniqueName: \"kubernetes.io/projected/bfdf5cf0-095b-4ab4-9754-b937e48dedc2-kube-api-access-rnsfl\") pod \"maas-api-revoke-keys-e2e-worker-w3-482722-lvwgw\" (UID: \"bfdf5cf0-095b-4ab4-9754-b937e48dedc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-482722-lvwgw"
Nov 03 20:38:21.540106 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:21.540036    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfdf5cf0-095b-4ab4-9754-b937e48dedc2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-482722-lvwgw\" (UID: \"bfdf5cf0-095b-4ab4-9754-b937e48dedc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-482722-lvwgw"
Nov 03 20:38:21.540106 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:21.540069    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfdf5cf0-095b-4ab4-9754-b937e48dedc2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-482722-lvwgw\" (UID: \"bfdf5cf0-095b-4ab4-9754-b937e48dedc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-482722-lvwgw"
Nov 03 20:38:21.641075 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:21.641037    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnsfl\" (UniqueName: \"kubernetes.io/projected/bfdf5cf0-095b-4ab4-9754-b937e48dedc2-kube-api-access-rnsfl\") pod \"maas-api-revoke-keys-e2e-worker-w3-482722-lvwgw\" (UID: \"bfdf5cf0-095b-4ab4-9754-b937e48dedc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-482722-lvwgw"
Nov 03 20:38:21.641294 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:21.641150    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfdf5cf0-095b-4ab4-9754-b937e48dedc2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-482722-lvwgw\" (UID: \"bfdf5cf0-095b-4ab4-9754-b937e48dedc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-482722-lvwgw"
Nov 03 20:38:21.641294 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:21.641167    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfdf5cf0-095b-4ab4-9754-b937e48dedc2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-482722-lvwgw\" (UID: \"bfdf5cf0-095b-4ab4-9754-b937e48dedc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-482722-lvwgw"
Nov 03 20:38:21.641908 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:21.641884    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfdf5cf0-095b-4ab4-9754-b937e48dedc2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-482722-lvwgw\" (UID: \"bfdf5cf0-095b-4ab4-9754-b937e48dedc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-482722-lvwgw"
Nov 03 20:38:21.643511 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:21.643489    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfdf5cf0-095b-4ab4-9754-b937e48dedc2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-482722-lvwgw\" (UID: \"bfdf5cf0-095b-4ab4-9754-b937e48dedc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-482722-lvwgw"
Nov 03 20:38:21.662261 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:21.662231    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnsfl\" (UniqueName: \"kubernetes.io/projected/bfdf5cf0-095b-4ab4-9754-b937e48dedc2-kube-api-access-rnsfl\") pod \"maas-api-revoke-keys-e2e-worker-w3-482722-lvwgw\" (UID: \"bfdf5cf0-095b-4ab4-9754-b937e48dedc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-482722-lvwgw"
Nov 03 20:38:21.761643 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:21.761547    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-482722-lvwgw"
Nov 03 20:38:22.059579 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:22.059546    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-482722-lvwgw"]
Nov 03 20:38:22.060839 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:38:22.060811    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfdf5cf0_095b_4ab4_9754_b937e48dedc2.slice/crio-49e1b114e28431f513d5ee014539b33c494aca84e0200d27dfc534a2fa711098 WatchSource:0}: Error finding container 49e1b114e28431f513d5ee014539b33c494aca84e0200d27dfc534a2fa711098: Status 404 returned error can't find the container with id 49e1b114e28431f513d5ee014539b33c494aca84e0200d27dfc534a2fa711098
Nov 03 20:38:22.313304 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:22.313210    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-482722-lvwgw" event={"ID":"bfdf5cf0-095b-4ab4-9754-b937e48dedc2","Type":"ContainerStarted","Data":"49e1b114e28431f513d5ee014539b33c494aca84e0200d27dfc534a2fa711098"}
Nov 03 20:38:23.094486 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:23.094447    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmqs6"]
Nov 03 20:38:23.094963 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:23.094718    5570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmqs6" podUID="516f80b2-26e4-44d2-b97a-6e740d0b53e6" containerName="istio-proxy" containerID="cri-o://5ade2867118dee88a4e2a26316583dc95b4296cf59fd952962d9b69fc346409c" gracePeriod=30
Nov 03 20:38:23.319996 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:23.319871    5570 generic.go:356] "Generic (PLEG): container finished" podID="bfdf5cf0-095b-4ab4-9754-b937e48dedc2" containerID="5b66695b9eca6d2eeff4b82a563f532b79d89d3e60e41624cb3a5d6752d28d4b" exitCode=0
Nov 03 20:38:23.319996 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:23.319964    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-482722-lvwgw" event={"ID":"bfdf5cf0-095b-4ab4-9754-b937e48dedc2","Type":"ContainerDied","Data":"5b66695b9eca6d2eeff4b82a563f532b79d89d3e60e41624cb3a5d6752d28d4b"}
Nov 03 20:38:24.463368 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:24.463338    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-482722-lvwgw"
Nov 03 20:38:24.570924 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:24.570879    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfdf5cf0-095b-4ab4-9754-b937e48dedc2-kube-api-access-rnsfl\" (UniqueName: \"kubernetes.io/projected/bfdf5cf0-095b-4ab4-9754-b937e48dedc2-kube-api-access-rnsfl\") pod \"bfdf5cf0-095b-4ab4-9754-b937e48dedc2\" (UID: \"bfdf5cf0-095b-4ab4-9754-b937e48dedc2\") "
Nov 03 20:38:24.571132 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:24.570963    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bfdf5cf0-095b-4ab4-9754-b937e48dedc2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfdf5cf0-095b-4ab4-9754-b937e48dedc2-maas-api-openshift-ca\") pod \"bfdf5cf0-095b-4ab4-9754-b937e48dedc2\" (UID: \"bfdf5cf0-095b-4ab4-9754-b937e48dedc2\") "
Nov 03 20:38:24.571132 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:24.570994    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bfdf5cf0-095b-4ab4-9754-b937e48dedc2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfdf5cf0-095b-4ab4-9754-b937e48dedc2-maas-api-xks-ca\") pod \"bfdf5cf0-095b-4ab4-9754-b937e48dedc2\" (UID: \"bfdf5cf0-095b-4ab4-9754-b937e48dedc2\") "
Nov 03 20:38:24.571307 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:24.571283    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfdf5cf0-095b-4ab4-9754-b937e48dedc2-maas-api-openshift-ca" pod "bfdf5cf0-095b-4ab4-9754-b937e48dedc2" (UID: "bfdf5cf0-095b-4ab4-9754-b937e48dedc2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:38:24.573242 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:24.573207    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bfdf5cf0-095b-4ab4-9754-b937e48dedc2-maas-api-xks-ca" pod "bfdf5cf0-095b-4ab4-9754-b937e48dedc2" (UID: "bfdf5cf0-095b-4ab4-9754-b937e48dedc2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:38:24.573361 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:24.573242    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfdf5cf0-095b-4ab4-9754-b937e48dedc2-kube-api-access-rnsfl" pod "bfdf5cf0-095b-4ab4-9754-b937e48dedc2" (UID: "bfdf5cf0-095b-4ab4-9754-b937e48dedc2"). InnerVolumeSpecName "kube-api-access-rnsfl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:38:24.672373 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:24.672332    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rnsfl\" (UniqueName: \"kubernetes.io/projected/bfdf5cf0-095b-4ab4-9754-b937e48dedc2-kube-api-access-rnsfl\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:38:24.672373 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:24.672368    5570 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfdf5cf0-095b-4ab4-9754-b937e48dedc2-maas-api-openshift-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:38:24.672602 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:24.672382    5570 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfdf5cf0-095b-4ab4-9754-b937e48dedc2-maas-api-xks-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:38:25.332726 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:25.332699    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-482722-lvwgw"
Nov 03 20:38:25.332942 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:25.332693    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-482722-lvwgw" event={"ID":"bfdf5cf0-095b-4ab4-9754-b937e48dedc2","Type":"ContainerDied","Data":"49e1b114e28431f513d5ee014539b33c494aca84e0200d27dfc534a2fa711098"}
Nov 03 20:38:25.332942 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:25.332837    5570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="49e1b114e28431f513d5ee014539b33c494aca84e0200d27dfc534a2fa711098"
Nov 03 20:38:25.934879 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:25.934834    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g69f8"]
Nov 03 20:38:26.201278 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:26.201237    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-482722/distinct-2-w3-482722-kserve-74bf58cf7d-v9cjw"]
Nov 03 20:38:26.214863 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:26.214826    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-482722/distinct-w3-482722-kserve-d876967c-kd488"]
Nov 03 20:38:26.216617 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:26.216591    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-482722/premium-w3-482722-kserve-689dc7bd5d-x8rjp"]
Nov 03 20:38:27.227921 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:38:27.227866    5570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8bwhz" containerName="extractor"
Nov 03 20:38:28.350683 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:28.350653    5570 generic.go:356] "Generic (PLEG): container finished" podID="516f80b2-26e4-44d2-b97a-6e740d0b53e6" containerID="5ade2867118dee88a4e2a26316583dc95b4296cf59fd952962d9b69fc346409c" exitCode=0
Nov 03 20:38:28.351097 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:28.350678    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmqs6" event={"ID":"516f80b2-26e4-44d2-b97a-6e740d0b53e6","Type":"ContainerDied","Data":"5ade2867118dee88a4e2a26316583dc95b4296cf59fd952962d9b69fc346409c"}
Nov 03 20:38:28.351097 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:28.350757    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmqs6" event={"ID":"516f80b2-26e4-44d2-b97a-6e740d0b53e6","Type":"ContainerDied","Data":"2729a9a0f959f575cd063fa1673a90dd9f08b2da502aa55e8876146174afa3ab"}
Nov 03 20:38:28.351097 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:28.350798    5570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2729a9a0f959f575cd063fa1673a90dd9f08b2da502aa55e8876146174afa3ab"
Nov 03 20:38:28.351097 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:28.350808    5570 scope.go:122] "RemoveContainer" containerID="cee1cab22ca2c6810bad67da56bb4911028e06424d912aed8fcdd1742ea8c9ae"
Nov 03 20:38:28.356198 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:28.356177    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmqs6"
Nov 03 20:38:28.409957 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:28.409922    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/516f80b2-26e4-44d2-b97a-6e740d0b53e6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/516f80b2-26e4-44d2-b97a-6e740d0b53e6-workload-certs\") pod \"516f80b2-26e4-44d2-b97a-6e740d0b53e6\" (UID: \"516f80b2-26e4-44d2-b97a-6e740d0b53e6\") "
Nov 03 20:38:28.409957 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:28.409960    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/516f80b2-26e4-44d2-b97a-6e740d0b53e6-istio-token\" (UniqueName: \"kubernetes.io/projected/516f80b2-26e4-44d2-b97a-6e740d0b53e6-istio-token\") pod \"516f80b2-26e4-44d2-b97a-6e740d0b53e6\" (UID: \"516f80b2-26e4-44d2-b97a-6e740d0b53e6\") "
Nov 03 20:38:28.410175 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:28.410010    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/516f80b2-26e4-44d2-b97a-6e740d0b53e6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/516f80b2-26e4-44d2-b97a-6e740d0b53e6-istio-data\") pod \"516f80b2-26e4-44d2-b97a-6e740d0b53e6\" (UID: \"516f80b2-26e4-44d2-b97a-6e740d0b53e6\") "
Nov 03 20:38:28.410175 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:28.410052    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/516f80b2-26e4-44d2-b97a-6e740d0b53e6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/516f80b2-26e4-44d2-b97a-6e740d0b53e6-istio-podinfo\") pod \"516f80b2-26e4-44d2-b97a-6e740d0b53e6\" (UID: \"516f80b2-26e4-44d2-b97a-6e740d0b53e6\") "
Nov 03 20:38:28.410175 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:28.410106    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/516f80b2-26e4-44d2-b97a-6e740d0b53e6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/516f80b2-26e4-44d2-b97a-6e740d0b53e6-credential-socket\") pod \"516f80b2-26e4-44d2-b97a-6e740d0b53e6\" (UID: \"516f80b2-26e4-44d2-b97a-6e740d0b53e6\") "
Nov 03 20:38:28.410175 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:28.410144    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/516f80b2-26e4-44d2-b97a-6e740d0b53e6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/516f80b2-26e4-44d2-b97a-6e740d0b53e6-workload-socket\") pod \"516f80b2-26e4-44d2-b97a-6e740d0b53e6\" (UID: \"516f80b2-26e4-44d2-b97a-6e740d0b53e6\") "
Nov 03 20:38:28.410361 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:28.410236    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/516f80b2-26e4-44d2-b97a-6e740d0b53e6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/516f80b2-26e4-44d2-b97a-6e740d0b53e6-istiod-ca-cert\") pod \"516f80b2-26e4-44d2-b97a-6e740d0b53e6\" (UID: \"516f80b2-26e4-44d2-b97a-6e740d0b53e6\") "
Nov 03 20:38:28.410361 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:28.410264    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/516f80b2-26e4-44d2-b97a-6e740d0b53e6-kube-api-access-nhdbd\" (UniqueName: \"kubernetes.io/projected/516f80b2-26e4-44d2-b97a-6e740d0b53e6-kube-api-access-nhdbd\") pod \"516f80b2-26e4-44d2-b97a-6e740d0b53e6\" (UID: \"516f80b2-26e4-44d2-b97a-6e740d0b53e6\") "
Nov 03 20:38:28.410361 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:28.410254    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/516f80b2-26e4-44d2-b97a-6e740d0b53e6-workload-certs" pod "516f80b2-26e4-44d2-b97a-6e740d0b53e6" (UID: "516f80b2-26e4-44d2-b97a-6e740d0b53e6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:38:28.410361 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:28.410305    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/516f80b2-26e4-44d2-b97a-6e740d0b53e6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/516f80b2-26e4-44d2-b97a-6e740d0b53e6-istio-envoy\") pod \"516f80b2-26e4-44d2-b97a-6e740d0b53e6\" (UID: \"516f80b2-26e4-44d2-b97a-6e740d0b53e6\") "
Nov 03 20:38:28.410361 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:28.410317    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/516f80b2-26e4-44d2-b97a-6e740d0b53e6-istio-data" pod "516f80b2-26e4-44d2-b97a-6e740d0b53e6" (UID: "516f80b2-26e4-44d2-b97a-6e740d0b53e6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:38:28.410709 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:28.410470    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/516f80b2-26e4-44d2-b97a-6e740d0b53e6-credential-socket" pod "516f80b2-26e4-44d2-b97a-6e740d0b53e6" (UID: "516f80b2-26e4-44d2-b97a-6e740d0b53e6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:38:28.410841 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:28.410795    5570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/516f80b2-26e4-44d2-b97a-6e740d0b53e6-workload-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:38:28.410841 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:28.410820    5570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/516f80b2-26e4-44d2-b97a-6e740d0b53e6-istio-data\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:38:28.410841 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:28.410832    5570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/516f80b2-26e4-44d2-b97a-6e740d0b53e6-credential-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:38:28.411032 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:28.410849    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/516f80b2-26e4-44d2-b97a-6e740d0b53e6-workload-socket" pod "516f80b2-26e4-44d2-b97a-6e740d0b53e6" (UID: "516f80b2-26e4-44d2-b97a-6e740d0b53e6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:38:28.411032 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:28.410877    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/516f80b2-26e4-44d2-b97a-6e740d0b53e6-istiod-ca-cert" pod "516f80b2-26e4-44d2-b97a-6e740d0b53e6" (UID: "516f80b2-26e4-44d2-b97a-6e740d0b53e6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:38:28.412946 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:28.412908    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/516f80b2-26e4-44d2-b97a-6e740d0b53e6-istio-envoy" pod "516f80b2-26e4-44d2-b97a-6e740d0b53e6" (UID: "516f80b2-26e4-44d2-b97a-6e740d0b53e6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:38:28.413076 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:28.412973    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/516f80b2-26e4-44d2-b97a-6e740d0b53e6-kube-api-access-nhdbd" pod "516f80b2-26e4-44d2-b97a-6e740d0b53e6" (UID: "516f80b2-26e4-44d2-b97a-6e740d0b53e6"). InnerVolumeSpecName "kube-api-access-nhdbd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:38:28.413166 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:28.413145    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/516f80b2-26e4-44d2-b97a-6e740d0b53e6-istio-podinfo" pod "516f80b2-26e4-44d2-b97a-6e740d0b53e6" (UID: "516f80b2-26e4-44d2-b97a-6e740d0b53e6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:38:28.413313 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:28.413290    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/516f80b2-26e4-44d2-b97a-6e740d0b53e6-istio-token" pod "516f80b2-26e4-44d2-b97a-6e740d0b53e6" (UID: "516f80b2-26e4-44d2-b97a-6e740d0b53e6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:38:28.511905 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:28.511835    5570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/516f80b2-26e4-44d2-b97a-6e740d0b53e6-istiod-ca-cert\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:38:28.511905 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:28.511878    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhdbd\" (UniqueName: \"kubernetes.io/projected/516f80b2-26e4-44d2-b97a-6e740d0b53e6-kube-api-access-nhdbd\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:38:28.511905 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:28.511905    5570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/516f80b2-26e4-44d2-b97a-6e740d0b53e6-istio-envoy\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:38:28.511905 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:28.511913    5570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/516f80b2-26e4-44d2-b97a-6e740d0b53e6-istio-token\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:38:28.511905 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:28.511921    5570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/516f80b2-26e4-44d2-b97a-6e740d0b53e6-istio-podinfo\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:38:28.512243 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:28.511928    5570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/516f80b2-26e4-44d2-b97a-6e740d0b53e6-workload-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:38:29.357520 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:29.357487    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmqs6"
Nov 03 20:38:29.451619 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:29.451582    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmqs6"]
Nov 03 20:38:29.457127 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:29.457082    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-e80203-openshift-default-5b6b7d855c-tmqs6"]
Nov 03 20:38:29.962815 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:29.962765    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="516f80b2-26e4-44d2-b97a-6e740d0b53e6" path="/var/lib/kubelet/pods/516f80b2-26e4-44d2-b97a-6e740d0b53e6/volumes"
Nov 03 20:38:31.748647 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:31.748615    5570 scope.go:122] "RemoveContainer" containerID="5ade2867118dee88a4e2a26316583dc95b4296cf59fd952962d9b69fc346409c"
Nov 03 20:38:33.413935 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:33.413897    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-966ac553-x29lb"]
Nov 03 20:38:33.414361 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:33.414277    5570 cpu_manager.go:414] "containerMap: removing container" podUID="3db8ac79-76fc-4d46-ac4f-6536fe192b8f" containerName="istio-proxy"
Nov 03 20:38:33.414361 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:33.414290    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3db8ac79-76fc-4d46-ac4f-6536fe192b8f" containerName="istio-proxy"
Nov 03 20:38:33.414361 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:33.414308    5570 cpu_manager.go:414] "containerMap: removing container" podUID="bfdf5cf0-095b-4ab4-9754-b937e48dedc2" containerName="revoke-keys"
Nov 03 20:38:33.414361 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:33.414313    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfdf5cf0-095b-4ab4-9754-b937e48dedc2" containerName="revoke-keys"
Nov 03 20:38:33.414361 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:33.414324    5570 cpu_manager.go:414] "containerMap: removing container" podUID="516f80b2-26e4-44d2-b97a-6e740d0b53e6" containerName="istio-proxy"
Nov 03 20:38:33.414361 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:33.414330    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="516f80b2-26e4-44d2-b97a-6e740d0b53e6" containerName="istio-proxy"
Nov 03 20:38:33.414361 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:33.414350    5570 cpu_manager.go:414] "containerMap: removing container" podUID="516f80b2-26e4-44d2-b97a-6e740d0b53e6" containerName="istio-proxy"
Nov 03 20:38:33.414361 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:33.414355    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="516f80b2-26e4-44d2-b97a-6e740d0b53e6" containerName="istio-proxy"
Nov 03 20:38:33.414620 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:33.414414    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="516f80b2-26e4-44d2-b97a-6e740d0b53e6" containerName="istio-proxy"
Nov 03 20:38:33.414620 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:33.414425    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfdf5cf0-095b-4ab4-9754-b937e48dedc2" containerName="revoke-keys"
Nov 03 20:38:33.419644 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:33.419618    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-966ac553-x29lb"
Nov 03 20:38:33.435353 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:33.435322    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-966ac553-x29lb"]
Nov 03 20:38:33.446541 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:33.446508    5570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 20:38:33.446784 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:33.446754    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:38:33.446928 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:33.446825    5570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 20:38:33.446928 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:33.446892    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l7s5w\""
Nov 03 20:38:33.562369 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:33.562329    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsthv\" (UniqueName: \"kubernetes.io/projected/f075a0ee-7327-4a89-bd17-a02edd236f9f-kube-api-access-xsthv\") pod \"maas-api-revoke-sub-966ac553-x29lb\" (UID: \"f075a0ee-7327-4a89-bd17-a02edd236f9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-966ac553-x29lb"
Nov 03 20:38:33.562561 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:33.562391    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f075a0ee-7327-4a89-bd17-a02edd236f9f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-966ac553-x29lb\" (UID: \"f075a0ee-7327-4a89-bd17-a02edd236f9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-966ac553-x29lb"
Nov 03 20:38:33.562561 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:33.562488    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f075a0ee-7327-4a89-bd17-a02edd236f9f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-966ac553-x29lb\" (UID: \"f075a0ee-7327-4a89-bd17-a02edd236f9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-966ac553-x29lb"
Nov 03 20:38:33.663497 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:33.663463    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsthv\" (UniqueName: \"kubernetes.io/projected/f075a0ee-7327-4a89-bd17-a02edd236f9f-kube-api-access-xsthv\") pod \"maas-api-revoke-sub-966ac553-x29lb\" (UID: \"f075a0ee-7327-4a89-bd17-a02edd236f9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-966ac553-x29lb"
Nov 03 20:38:33.663719 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:33.663521    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f075a0ee-7327-4a89-bd17-a02edd236f9f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-966ac553-x29lb\" (UID: \"f075a0ee-7327-4a89-bd17-a02edd236f9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-966ac553-x29lb"
Nov 03 20:38:33.663719 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:33.663552    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f075a0ee-7327-4a89-bd17-a02edd236f9f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-966ac553-x29lb\" (UID: \"f075a0ee-7327-4a89-bd17-a02edd236f9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-966ac553-x29lb"
Nov 03 20:38:33.664413 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:33.664357    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f075a0ee-7327-4a89-bd17-a02edd236f9f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-966ac553-x29lb\" (UID: \"f075a0ee-7327-4a89-bd17-a02edd236f9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-966ac553-x29lb"
Nov 03 20:38:33.665998 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:33.665976    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f075a0ee-7327-4a89-bd17-a02edd236f9f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-966ac553-x29lb\" (UID: \"f075a0ee-7327-4a89-bd17-a02edd236f9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-966ac553-x29lb"
Nov 03 20:38:33.685003 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:33.684974    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsthv\" (UniqueName: \"kubernetes.io/projected/f075a0ee-7327-4a89-bd17-a02edd236f9f-kube-api-access-xsthv\") pod \"maas-api-revoke-sub-966ac553-x29lb\" (UID: \"f075a0ee-7327-4a89-bd17-a02edd236f9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-966ac553-x29lb"
Nov 03 20:38:33.733807 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:33.733759    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-966ac553-x29lb"
Nov 03 20:38:34.032386 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:34.032353    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-966ac553-x29lb"]
Nov 03 20:38:34.033690 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:38:34.033660    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf075a0ee_7327_4a89_bd17_a02edd236f9f.slice/crio-695a7a649970b01f3b06ee06b2fe9bf4999c0ecb2112db1c0ad2b95ff4f25b66 WatchSource:0}: Error finding container 695a7a649970b01f3b06ee06b2fe9bf4999c0ecb2112db1c0ad2b95ff4f25b66: Status 404 returned error can't find the container with id 695a7a649970b01f3b06ee06b2fe9bf4999c0ecb2112db1c0ad2b95ff4f25b66
Nov 03 20:38:34.390745 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:34.390709    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-966ac553-x29lb" event={"ID":"f075a0ee-7327-4a89-bd17-a02edd236f9f","Type":"ContainerStarted","Data":"3f35e269fd588cf53f7e120fc53f6cb2d495d5c1d02046acc4d72e1bd36e3480"}
Nov 03 20:38:34.390944 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:34.390764    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-966ac553-x29lb" event={"ID":"f075a0ee-7327-4a89-bd17-a02edd236f9f","Type":"ContainerStarted","Data":"695a7a649970b01f3b06ee06b2fe9bf4999c0ecb2112db1c0ad2b95ff4f25b66"}
Nov 03 20:38:34.432699 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:34.432646    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-966ac553-x29lb" podStartSLOduration=1.43263144 podStartE2EDuration="1.43263144s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:38:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:38:34.431536164 +0000 UTC m=+2048.584198049" watchObservedRunningTime="2026-11-03 20:38:34.43263144 +0000 UTC m=+2048.585293327"
Nov 03 20:38:35.398532 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:35.398496    5570 generic.go:356] "Generic (PLEG): container finished" podID="f075a0ee-7327-4a89-bd17-a02edd236f9f" containerID="3f35e269fd588cf53f7e120fc53f6cb2d495d5c1d02046acc4d72e1bd36e3480" exitCode=0
Nov 03 20:38:35.398765 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:35.398569    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-966ac553-x29lb" event={"ID":"f075a0ee-7327-4a89-bd17-a02edd236f9f","Type":"ContainerDied","Data":"3f35e269fd588cf53f7e120fc53f6cb2d495d5c1d02046acc4d72e1bd36e3480"}
Nov 03 20:38:36.544439 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:36.544410    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-966ac553-x29lb"
Nov 03 20:38:36.697814 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:36.697750    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f075a0ee-7327-4a89-bd17-a02edd236f9f-kube-api-access-xsthv\" (UniqueName: \"kubernetes.io/projected/f075a0ee-7327-4a89-bd17-a02edd236f9f-kube-api-access-xsthv\") pod \"f075a0ee-7327-4a89-bd17-a02edd236f9f\" (UID: \"f075a0ee-7327-4a89-bd17-a02edd236f9f\") "
Nov 03 20:38:36.698042 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:36.697838    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f075a0ee-7327-4a89-bd17-a02edd236f9f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f075a0ee-7327-4a89-bd17-a02edd236f9f-maas-api-xks-ca\") pod \"f075a0ee-7327-4a89-bd17-a02edd236f9f\" (UID: \"f075a0ee-7327-4a89-bd17-a02edd236f9f\") "
Nov 03 20:38:36.698042 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:36.697948    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f075a0ee-7327-4a89-bd17-a02edd236f9f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f075a0ee-7327-4a89-bd17-a02edd236f9f-maas-api-openshift-ca\") pod \"f075a0ee-7327-4a89-bd17-a02edd236f9f\" (UID: \"f075a0ee-7327-4a89-bd17-a02edd236f9f\") "
Nov 03 20:38:36.698308 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:36.698283    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f075a0ee-7327-4a89-bd17-a02edd236f9f-maas-api-openshift-ca" pod "f075a0ee-7327-4a89-bd17-a02edd236f9f" (UID: "f075a0ee-7327-4a89-bd17-a02edd236f9f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:38:36.700023 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:36.699999    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f075a0ee-7327-4a89-bd17-a02edd236f9f-maas-api-xks-ca" pod "f075a0ee-7327-4a89-bd17-a02edd236f9f" (UID: "f075a0ee-7327-4a89-bd17-a02edd236f9f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:38:36.700023 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:36.700005    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f075a0ee-7327-4a89-bd17-a02edd236f9f-kube-api-access-xsthv" pod "f075a0ee-7327-4a89-bd17-a02edd236f9f" (UID: "f075a0ee-7327-4a89-bd17-a02edd236f9f"). InnerVolumeSpecName "kube-api-access-xsthv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:38:36.799373 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:36.799328    5570 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f075a0ee-7327-4a89-bd17-a02edd236f9f-maas-api-openshift-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:38:36.799373 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:36.799363    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xsthv\" (UniqueName: \"kubernetes.io/projected/f075a0ee-7327-4a89-bd17-a02edd236f9f-kube-api-access-xsthv\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:38:36.799373 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:36.799375    5570 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f075a0ee-7327-4a89-bd17-a02edd236f9f-maas-api-xks-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:38:37.412954 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:37.412921    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-966ac553-x29lb"
Nov 03 20:38:37.412954 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:37.412946    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-966ac553-x29lb" event={"ID":"f075a0ee-7327-4a89-bd17-a02edd236f9f","Type":"ContainerDied","Data":"695a7a649970b01f3b06ee06b2fe9bf4999c0ecb2112db1c0ad2b95ff4f25b66"}
Nov 03 20:38:37.413260 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:37.413006    5570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="695a7a649970b01f3b06ee06b2fe9bf4999c0ecb2112db1c0ad2b95ff4f25b66"
Nov 03 20:38:41.223153 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:41.223084    5570 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-482722/distinct-2-w3-482722-kserve-74bf58cf7d-v9cjw" podUID="9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a" containerName="main" containerID="cri-o://75218134286e7df2bb090b7a834fc8bb8a15f1219e7402cd637a818db00e97e1" gracePeriod=45
Nov 03 20:38:41.236153 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:41.236117    5570 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-482722/distinct-w3-482722-kserve-d876967c-kd488" podUID="9a97f9cc-2261-471d-9053-51ee965850bd" containerName="main" containerID="cri-o://a233a0e6dfafb8dfd9e6fe03c9a65d2eb2db2e0adf497de68e8b2e56a92e53fe" gracePeriod=45
Nov 03 20:38:41.237039 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:41.236732    5570 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-482722/premium-w3-482722-kserve-689dc7bd5d-x8rjp" podUID="369045c3-4d6a-49a7-9bc6-f2939883bee8" containerName="main" containerID="cri-o://ffcbc1a7b9d7c549e41b0ed6f5ea4c2ee7c624ba331bc58642873ce48c298f3b" gracePeriod=45
Nov 03 20:38:41.441649 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:41.441614    5570 generic.go:356] "Generic (PLEG): container finished" podID="9a97f9cc-2261-471d-9053-51ee965850bd" containerID="a233a0e6dfafb8dfd9e6fe03c9a65d2eb2db2e0adf497de68e8b2e56a92e53fe" exitCode=0
Nov 03 20:38:41.441869 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:41.441753    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-482722/distinct-w3-482722-kserve-d876967c-kd488" event={"ID":"9a97f9cc-2261-471d-9053-51ee965850bd","Type":"ContainerDied","Data":"a233a0e6dfafb8dfd9e6fe03c9a65d2eb2db2e0adf497de68e8b2e56a92e53fe"}
Nov 03 20:38:41.447281 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:41.447135    5570 generic.go:356] "Generic (PLEG): container finished" podID="369045c3-4d6a-49a7-9bc6-f2939883bee8" containerID="ffcbc1a7b9d7c549e41b0ed6f5ea4c2ee7c624ba331bc58642873ce48c298f3b" exitCode=0
Nov 03 20:38:41.447281 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:41.447221    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-482722/premium-w3-482722-kserve-689dc7bd5d-x8rjp" event={"ID":"369045c3-4d6a-49a7-9bc6-f2939883bee8","Type":"ContainerDied","Data":"ffcbc1a7b9d7c549e41b0ed6f5ea4c2ee7c624ba331bc58642873ce48c298f3b"}
Nov 03 20:38:41.449172 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:41.449121    5570 generic.go:356] "Generic (PLEG): container finished" podID="9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a" containerID="75218134286e7df2bb090b7a834fc8bb8a15f1219e7402cd637a818db00e97e1" exitCode=0
Nov 03 20:38:41.449348 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:41.449316    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-482722/distinct-2-w3-482722-kserve-74bf58cf7d-v9cjw" event={"ID":"9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a","Type":"ContainerDied","Data":"75218134286e7df2bb090b7a834fc8bb8a15f1219e7402cd637a818db00e97e1"}
Nov 03 20:38:41.528745 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:41.528714    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-482722/distinct-2-w3-482722-kserve-74bf58cf7d-v9cjw"
Nov 03 20:38:41.531445 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:41.531408    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-482722/distinct-w3-482722-kserve-d876967c-kd488"
Nov 03 20:38:41.535435 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:41.535340    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-482722/premium-w3-482722-kserve-689dc7bd5d-x8rjp"
Nov 03 20:38:41.661357 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:41.661314    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/369045c3-4d6a-49a7-9bc6-f2939883bee8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/369045c3-4d6a-49a7-9bc6-f2939883bee8-model-cache\") pod \"369045c3-4d6a-49a7-9bc6-f2939883bee8\" (UID: \"369045c3-4d6a-49a7-9bc6-f2939883bee8\") "
Nov 03 20:38:41.661570 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:41.661386    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a97f9cc-2261-471d-9053-51ee965850bd-kube-api-access-7s8s7\" (UniqueName: \"kubernetes.io/projected/9a97f9cc-2261-471d-9053-51ee965850bd-kube-api-access-7s8s7\") pod \"9a97f9cc-2261-471d-9053-51ee965850bd\" (UID: \"9a97f9cc-2261-471d-9053-51ee965850bd\") "
Nov 03 20:38:41.661570 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:41.661450    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a97f9cc-2261-471d-9053-51ee965850bd-home\" (UniqueName: \"kubernetes.io/empty-dir/9a97f9cc-2261-471d-9053-51ee965850bd-home\") pod \"9a97f9cc-2261-471d-9053-51ee965850bd\" (UID: \"9a97f9cc-2261-471d-9053-51ee965850bd\") "
Nov 03 20:38:41.661570 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:41.661470    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a97f9cc-2261-471d-9053-51ee965850bd-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a97f9cc-2261-471d-9053-51ee965850bd-tmp-dir\") pod \"9a97f9cc-2261-471d-9053-51ee965850bd\" (UID: \"9a97f9cc-2261-471d-9053-51ee965850bd\") "
Nov 03 20:38:41.661570 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:41.661500    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a97f9cc-2261-471d-9053-51ee965850bd-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a97f9cc-2261-471d-9053-51ee965850bd-dshm\") pod \"9a97f9cc-2261-471d-9053-51ee965850bd\" (UID: \"9a97f9cc-2261-471d-9053-51ee965850bd\") "
Nov 03 20:38:41.661570 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:41.661520    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a-kube-api-access-6vlf8\" (UniqueName: \"kubernetes.io/projected/9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a-kube-api-access-6vlf8\") pod \"9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a\" (UID: \"9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a\") "
Nov 03 20:38:41.661570 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:41.661535    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a-dshm\") pod \"9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a\" (UID: \"9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a\") "
Nov 03 20:38:41.661570 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:41.661555    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a97f9cc-2261-471d-9053-51ee965850bd-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a97f9cc-2261-471d-9053-51ee965850bd-model-cache\") pod \"9a97f9cc-2261-471d-9053-51ee965850bd\" (UID: \"9a97f9cc-2261-471d-9053-51ee965850bd\") "
Nov 03 20:38:41.661971 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:41.661584    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a-tls-certs\" (UniqueName: \"kubernetes.io/secret/9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a-tls-certs\") pod \"9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a\" (UID: \"9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a\") "
Nov 03 20:38:41.661971 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:41.661612    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/369045c3-4d6a-49a7-9bc6-f2939883bee8-kube-api-access-6hmzj\" (UniqueName: \"kubernetes.io/projected/369045c3-4d6a-49a7-9bc6-f2939883bee8-kube-api-access-6hmzj\") pod \"369045c3-4d6a-49a7-9bc6-f2939883bee8\" (UID: \"369045c3-4d6a-49a7-9bc6-f2939883bee8\") "
Nov 03 20:38:41.661971 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:41.661632    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a-home\" (UniqueName: \"kubernetes.io/empty-dir/9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a-home\") pod \"9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a\" (UID: \"9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a\") "
Nov 03 20:38:41.661971 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:41.661656    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a-tmp-dir\") pod \"9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a\" (UID: \"9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a\") "
Nov 03 20:38:41.661971 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:41.661685    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/369045c3-4d6a-49a7-9bc6-f2939883bee8-tls-certs\" (UniqueName: \"kubernetes.io/secret/369045c3-4d6a-49a7-9bc6-f2939883bee8-tls-certs\") pod \"369045c3-4d6a-49a7-9bc6-f2939883bee8\" (UID: \"369045c3-4d6a-49a7-9bc6-f2939883bee8\") "
Nov 03 20:38:41.661971 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:41.661719    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9a97f9cc-2261-471d-9053-51ee965850bd-tls-certs\" (UniqueName: \"kubernetes.io/secret/9a97f9cc-2261-471d-9053-51ee965850bd-tls-certs\") pod \"9a97f9cc-2261-471d-9053-51ee965850bd\" (UID: \"9a97f9cc-2261-471d-9053-51ee965850bd\") "
Nov 03 20:38:41.661971 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:41.661747    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a-model-cache\") pod \"9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a\" (UID: \"9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a\") "
Nov 03 20:38:41.661971 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:41.661765    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/369045c3-4d6a-49a7-9bc6-f2939883bee8-home\" (UniqueName: \"kubernetes.io/empty-dir/369045c3-4d6a-49a7-9bc6-f2939883bee8-home\") pod \"369045c3-4d6a-49a7-9bc6-f2939883bee8\" (UID: \"369045c3-4d6a-49a7-9bc6-f2939883bee8\") "
Nov 03 20:38:41.661971 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:41.661827    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/369045c3-4d6a-49a7-9bc6-f2939883bee8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/369045c3-4d6a-49a7-9bc6-f2939883bee8-dshm\") pod \"369045c3-4d6a-49a7-9bc6-f2939883bee8\" (UID: \"369045c3-4d6a-49a7-9bc6-f2939883bee8\") "
Nov 03 20:38:41.661971 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:41.661855    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/369045c3-4d6a-49a7-9bc6-f2939883bee8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/369045c3-4d6a-49a7-9bc6-f2939883bee8-tmp-dir\") pod \"369045c3-4d6a-49a7-9bc6-f2939883bee8\" (UID: \"369045c3-4d6a-49a7-9bc6-f2939883bee8\") "
Nov 03 20:38:41.662487 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:41.662325    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/369045c3-4d6a-49a7-9bc6-f2939883bee8-tmp-dir" pod "369045c3-4d6a-49a7-9bc6-f2939883bee8" (UID: "369045c3-4d6a-49a7-9bc6-f2939883bee8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:38:41.662619 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:41.662595    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a97f9cc-2261-471d-9053-51ee965850bd-tmp-dir" pod "9a97f9cc-2261-471d-9053-51ee965850bd" (UID: "9a97f9cc-2261-471d-9053-51ee965850bd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:38:41.662857 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:41.662833    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/369045c3-4d6a-49a7-9bc6-f2939883bee8-model-cache" pod "369045c3-4d6a-49a7-9bc6-f2939883bee8" (UID: "369045c3-4d6a-49a7-9bc6-f2939883bee8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:38:41.663789 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:41.663641    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/369045c3-4d6a-49a7-9bc6-f2939883bee8-home" pod "369045c3-4d6a-49a7-9bc6-f2939883bee8" (UID: "369045c3-4d6a-49a7-9bc6-f2939883bee8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:38:41.664627 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:41.664109    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a-home" pod "9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a" (UID: "9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:38:41.664627 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:41.664248    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a97f9cc-2261-471d-9053-51ee965850bd-model-cache" pod "9a97f9cc-2261-471d-9053-51ee965850bd" (UID: "9a97f9cc-2261-471d-9053-51ee965850bd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:38:41.664627 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:41.664256    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a-tmp-dir" pod "9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a" (UID: "9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:38:41.664627 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:41.664499    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a-model-cache" pod "9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a" (UID: "9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:38:41.665261 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:41.665222    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a97f9cc-2261-471d-9053-51ee965850bd-home" pod "9a97f9cc-2261-471d-9053-51ee965850bd" (UID: "9a97f9cc-2261-471d-9053-51ee965850bd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:38:41.665367 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:41.665345    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a-tls-certs" pod "9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a" (UID: "9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:38:41.665682 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:41.665654    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a97f9cc-2261-471d-9053-51ee965850bd-tls-certs" pod "9a97f9cc-2261-471d-9053-51ee965850bd" (UID: "9a97f9cc-2261-471d-9053-51ee965850bd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:38:41.667012 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:41.666984    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/369045c3-4d6a-49a7-9bc6-f2939883bee8-tls-certs" pod "369045c3-4d6a-49a7-9bc6-f2939883bee8" (UID: "369045c3-4d6a-49a7-9bc6-f2939883bee8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:38:41.667862 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:41.667824    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a97f9cc-2261-471d-9053-51ee965850bd-dshm" pod "9a97f9cc-2261-471d-9053-51ee965850bd" (UID: "9a97f9cc-2261-471d-9053-51ee965850bd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:38:41.667981 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:41.667925    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/369045c3-4d6a-49a7-9bc6-f2939883bee8-kube-api-access-6hmzj" pod "369045c3-4d6a-49a7-9bc6-f2939883bee8" (UID: "369045c3-4d6a-49a7-9bc6-f2939883bee8"). InnerVolumeSpecName "kube-api-access-6hmzj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:38:41.668034 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:41.667989    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a97f9cc-2261-471d-9053-51ee965850bd-kube-api-access-7s8s7" pod "9a97f9cc-2261-471d-9053-51ee965850bd" (UID: "9a97f9cc-2261-471d-9053-51ee965850bd"). InnerVolumeSpecName "kube-api-access-7s8s7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:38:41.668076 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:41.668058    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a-dshm" pod "9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a" (UID: "9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:38:41.668611 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:41.668580    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/369045c3-4d6a-49a7-9bc6-f2939883bee8-dshm" pod "369045c3-4d6a-49a7-9bc6-f2939883bee8" (UID: "369045c3-4d6a-49a7-9bc6-f2939883bee8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:38:41.669301 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:41.669279    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a-kube-api-access-6vlf8" pod "9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a" (UID: "9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a"). InnerVolumeSpecName "kube-api-access-6vlf8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:38:41.763281 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:41.763185    5570 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/369045c3-4d6a-49a7-9bc6-f2939883bee8-home\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:38:41.763281 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:41.763221    5570 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/369045c3-4d6a-49a7-9bc6-f2939883bee8-dshm\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:38:41.763281 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:41.763237    5570 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/369045c3-4d6a-49a7-9bc6-f2939883bee8-tmp-dir\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:38:41.763281 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:41.763246    5570 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/369045c3-4d6a-49a7-9bc6-f2939883bee8-model-cache\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:38:41.763281 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:41.763257    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7s8s7\" (UniqueName: \"kubernetes.io/projected/9a97f9cc-2261-471d-9053-51ee965850bd-kube-api-access-7s8s7\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:38:41.763281 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:41.763269    5570 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a97f9cc-2261-471d-9053-51ee965850bd-home\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:38:41.763281 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:41.763281    5570 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a97f9cc-2261-471d-9053-51ee965850bd-tmp-dir\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:38:41.763281 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:41.763289    5570 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a97f9cc-2261-471d-9053-51ee965850bd-dshm\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:38:41.763700 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:41.763297    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vlf8\" (UniqueName: \"kubernetes.io/projected/9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a-kube-api-access-6vlf8\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:38:41.763700 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:41.763306    5570 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a-dshm\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:38:41.763700 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:41.763314    5570 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a97f9cc-2261-471d-9053-51ee965850bd-model-cache\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:38:41.763700 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:41.763324    5570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a-tls-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:38:41.763700 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:41.763334    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hmzj\" (UniqueName: \"kubernetes.io/projected/369045c3-4d6a-49a7-9bc6-f2939883bee8-kube-api-access-6hmzj\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:38:41.763700 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:41.763343    5570 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a-home\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:38:41.763700 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:41.763358    5570 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a-tmp-dir\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:38:41.763700 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:41.763366    5570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/369045c3-4d6a-49a7-9bc6-f2939883bee8-tls-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:38:41.763700 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:41.763374    5570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a97f9cc-2261-471d-9053-51ee965850bd-tls-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:38:41.763700 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:41.763380    5570 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a-model-cache\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:38:42.456385 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:42.456354    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-482722/premium-w3-482722-kserve-689dc7bd5d-x8rjp"
Nov 03 20:38:42.456385 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:42.456370    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-482722/premium-w3-482722-kserve-689dc7bd5d-x8rjp" event={"ID":"369045c3-4d6a-49a7-9bc6-f2939883bee8","Type":"ContainerDied","Data":"76c9987e584a0ea357a472e354bbea18668b8836d33e3679c1235c6c19940246"}
Nov 03 20:38:42.456947 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:42.456425    5570 scope.go:122] "RemoveContainer" containerID="ffcbc1a7b9d7c549e41b0ed6f5ea4c2ee7c624ba331bc58642873ce48c298f3b"
Nov 03 20:38:42.457937 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:42.457920    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-482722/distinct-2-w3-482722-kserve-74bf58cf7d-v9cjw"
Nov 03 20:38:42.458032 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:42.457958    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-482722/distinct-2-w3-482722-kserve-74bf58cf7d-v9cjw" event={"ID":"9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a","Type":"ContainerDied","Data":"46282515f5ca787be6ccfb24c4a7baef19ea56ef06b6c9004e7952176a40da75"}
Nov 03 20:38:42.459654 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:42.459631    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-482722/distinct-w3-482722-kserve-d876967c-kd488" event={"ID":"9a97f9cc-2261-471d-9053-51ee965850bd","Type":"ContainerDied","Data":"ad770193bae872b6c4e021f6c8f1c6e4b8ecbf8c12db828b3d4ae5b57950572f"}
Nov 03 20:38:42.459654 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:42.459648    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-482722/distinct-w3-482722-kserve-d876967c-kd488"
Nov 03 20:38:42.467059 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:42.467041    5570 scope.go:122] "RemoveContainer" containerID="75218134286e7df2bb090b7a834fc8bb8a15f1219e7402cd637a818db00e97e1"
Nov 03 20:38:42.475671 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:42.475650    5570 scope.go:122] "RemoveContainer" containerID="a233a0e6dfafb8dfd9e6fe03c9a65d2eb2db2e0adf497de68e8b2e56a92e53fe"
Nov 03 20:38:42.510539 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:42.510442    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-482722/premium-w3-482722-kserve-689dc7bd5d-x8rjp"]
Nov 03 20:38:42.523646 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:42.523612    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-482722/premium-w3-482722-kserve-689dc7bd5d-x8rjp"]
Nov 03 20:38:42.560351 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:42.560317    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-482722/distinct-2-w3-482722-kserve-74bf58cf7d-v9cjw"]
Nov 03 20:38:42.562530 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:42.562503    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-482722/distinct-2-w3-482722-kserve-74bf58cf7d-v9cjw"]
Nov 03 20:38:42.599641 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:42.599604    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-482722/distinct-w3-482722-kserve-d876967c-kd488"]
Nov 03 20:38:42.601698 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:42.601670    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-482722/distinct-w3-482722-kserve-d876967c-kd488"]
Nov 03 20:38:43.964255 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:43.964223    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="369045c3-4d6a-49a7-9bc6-f2939883bee8" path="/var/lib/kubelet/pods/369045c3-4d6a-49a7-9bc6-f2939883bee8/volumes"
Nov 03 20:38:43.964654 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:43.964640    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a97f9cc-2261-471d-9053-51ee965850bd" path="/var/lib/kubelet/pods/9a97f9cc-2261-471d-9053-51ee965850bd/volumes"
Nov 03 20:38:43.965031 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:43.965019    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a" path="/var/lib/kubelet/pods/9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a/volumes"
Nov 03 20:38:45.737122 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:45.737075    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g69f8"]
Nov 03 20:38:49.158133 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:49.158097    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-482722-lvwgw"]
Nov 03 20:38:49.177348 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:49.177306    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-482722-lvwgw"]
Nov 03 20:38:49.963618 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:49.963581    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bfdf5cf0-095b-4ab4-9754-b937e48dedc2" path="/var/lib/kubelet/pods/bfdf5cf0-095b-4ab4-9754-b937e48dedc2/volumes"
Nov 03 20:38:52.664372 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:52.664330    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g69f8"]
Nov 03 20:38:55.581801 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:55.581110    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-c9kvc"]
Nov 03 20:38:55.581801 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:38:55.581395    5570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-c9kvc" podUID="1351662d-6ca4-430b-8bbe-0ca1429df833" containerName="istio-proxy" containerID="cri-o://8977e092d4a9cfa729f07741dd5894f87839bdf7aa38cf0b489c3d40058255c4" gracePeriod=30
Nov 03 20:38:57.231076 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:38:57.231036    5570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8bwhz" containerName="extractor"
Nov 03 20:39:00.880217 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:00.880185    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-c9kvc"
Nov 03 20:39:00.963583 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:00.963547    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1351662d-6ca4-430b-8bbe-0ca1429df833-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1351662d-6ca4-430b-8bbe-0ca1429df833-workload-socket\") pod \"1351662d-6ca4-430b-8bbe-0ca1429df833\" (UID: \"1351662d-6ca4-430b-8bbe-0ca1429df833\") "
Nov 03 20:39:00.963583 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:00.963591    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1351662d-6ca4-430b-8bbe-0ca1429df833-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1351662d-6ca4-430b-8bbe-0ca1429df833-istiod-ca-cert\") pod \"1351662d-6ca4-430b-8bbe-0ca1429df833\" (UID: \"1351662d-6ca4-430b-8bbe-0ca1429df833\") "
Nov 03 20:39:00.963935 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:00.963628    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1351662d-6ca4-430b-8bbe-0ca1429df833-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1351662d-6ca4-430b-8bbe-0ca1429df833-istio-podinfo\") pod \"1351662d-6ca4-430b-8bbe-0ca1429df833\" (UID: \"1351662d-6ca4-430b-8bbe-0ca1429df833\") "
Nov 03 20:39:00.963935 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:00.963699    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1351662d-6ca4-430b-8bbe-0ca1429df833-kube-api-access-qw8fv\" (UniqueName: \"kubernetes.io/projected/1351662d-6ca4-430b-8bbe-0ca1429df833-kube-api-access-qw8fv\") pod \"1351662d-6ca4-430b-8bbe-0ca1429df833\" (UID: \"1351662d-6ca4-430b-8bbe-0ca1429df833\") "
Nov 03 20:39:00.963935 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:00.963740    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1351662d-6ca4-430b-8bbe-0ca1429df833-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1351662d-6ca4-430b-8bbe-0ca1429df833-credential-socket\") pod \"1351662d-6ca4-430b-8bbe-0ca1429df833\" (UID: \"1351662d-6ca4-430b-8bbe-0ca1429df833\") "
Nov 03 20:39:00.963935 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:00.963789    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1351662d-6ca4-430b-8bbe-0ca1429df833-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1351662d-6ca4-430b-8bbe-0ca1429df833-istio-envoy\") pod \"1351662d-6ca4-430b-8bbe-0ca1429df833\" (UID: \"1351662d-6ca4-430b-8bbe-0ca1429df833\") "
Nov 03 20:39:00.963935 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:00.963850    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1351662d-6ca4-430b-8bbe-0ca1429df833-istio-token\" (UniqueName: \"kubernetes.io/projected/1351662d-6ca4-430b-8bbe-0ca1429df833-istio-token\") pod \"1351662d-6ca4-430b-8bbe-0ca1429df833\" (UID: \"1351662d-6ca4-430b-8bbe-0ca1429df833\") "
Nov 03 20:39:00.963935 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:00.963903    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1351662d-6ca4-430b-8bbe-0ca1429df833-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1351662d-6ca4-430b-8bbe-0ca1429df833-istio-data\") pod \"1351662d-6ca4-430b-8bbe-0ca1429df833\" (UID: \"1351662d-6ca4-430b-8bbe-0ca1429df833\") "
Nov 03 20:39:00.964294 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:00.963940    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1351662d-6ca4-430b-8bbe-0ca1429df833-workload-socket" pod "1351662d-6ca4-430b-8bbe-0ca1429df833" (UID: "1351662d-6ca4-430b-8bbe-0ca1429df833"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:39:00.964294 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:00.963974    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1351662d-6ca4-430b-8bbe-0ca1429df833-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1351662d-6ca4-430b-8bbe-0ca1429df833-workload-certs\") pod \"1351662d-6ca4-430b-8bbe-0ca1429df833\" (UID: \"1351662d-6ca4-430b-8bbe-0ca1429df833\") "
Nov 03 20:39:00.964294 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:00.964109    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1351662d-6ca4-430b-8bbe-0ca1429df833-credential-socket" pod "1351662d-6ca4-430b-8bbe-0ca1429df833" (UID: "1351662d-6ca4-430b-8bbe-0ca1429df833"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:39:00.964294 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:00.964129    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1351662d-6ca4-430b-8bbe-0ca1429df833-istiod-ca-cert" pod "1351662d-6ca4-430b-8bbe-0ca1429df833" (UID: "1351662d-6ca4-430b-8bbe-0ca1429df833"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:39:00.964540 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:00.964331    5570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1351662d-6ca4-430b-8bbe-0ca1429df833-workload-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:39:00.964540 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:00.964354    5570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1351662d-6ca4-430b-8bbe-0ca1429df833-istiod-ca-cert\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:39:00.964540 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:00.964363    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1351662d-6ca4-430b-8bbe-0ca1429df833-workload-certs" pod "1351662d-6ca4-430b-8bbe-0ca1429df833" (UID: "1351662d-6ca4-430b-8bbe-0ca1429df833"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:39:00.964540 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:00.964373    5570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1351662d-6ca4-430b-8bbe-0ca1429df833-credential-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:39:00.964807 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:00.964632    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1351662d-6ca4-430b-8bbe-0ca1429df833-istio-data" pod "1351662d-6ca4-430b-8bbe-0ca1429df833" (UID: "1351662d-6ca4-430b-8bbe-0ca1429df833"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:39:00.966249 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:00.966213    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1351662d-6ca4-430b-8bbe-0ca1429df833-istio-envoy" pod "1351662d-6ca4-430b-8bbe-0ca1429df833" (UID: "1351662d-6ca4-430b-8bbe-0ca1429df833"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:39:00.966737 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:00.966707    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1351662d-6ca4-430b-8bbe-0ca1429df833-istio-podinfo" pod "1351662d-6ca4-430b-8bbe-0ca1429df833" (UID: "1351662d-6ca4-430b-8bbe-0ca1429df833"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:39:00.966885 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:00.966762    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1351662d-6ca4-430b-8bbe-0ca1429df833-istio-token" pod "1351662d-6ca4-430b-8bbe-0ca1429df833" (UID: "1351662d-6ca4-430b-8bbe-0ca1429df833"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:39:00.966885 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:00.966844    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1351662d-6ca4-430b-8bbe-0ca1429df833-kube-api-access-qw8fv" pod "1351662d-6ca4-430b-8bbe-0ca1429df833" (UID: "1351662d-6ca4-430b-8bbe-0ca1429df833"). InnerVolumeSpecName "kube-api-access-qw8fv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:39:01.065643 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:01.065598    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qw8fv\" (UniqueName: \"kubernetes.io/projected/1351662d-6ca4-430b-8bbe-0ca1429df833-kube-api-access-qw8fv\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:39:01.065643 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:01.065634    5570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1351662d-6ca4-430b-8bbe-0ca1429df833-istio-envoy\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:39:01.065643 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:01.065645    5570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1351662d-6ca4-430b-8bbe-0ca1429df833-istio-token\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:39:01.065643 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:01.065653    5570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1351662d-6ca4-430b-8bbe-0ca1429df833-istio-data\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:39:01.065643 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:01.065660    5570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1351662d-6ca4-430b-8bbe-0ca1429df833-workload-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:39:01.066106 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:01.065668    5570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1351662d-6ca4-430b-8bbe-0ca1429df833-istio-podinfo\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:39:01.564292 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:01.564253    5570 generic.go:356] "Generic (PLEG): container finished" podID="1351662d-6ca4-430b-8bbe-0ca1429df833" containerID="8977e092d4a9cfa729f07741dd5894f87839bdf7aa38cf0b489c3d40058255c4" exitCode=0
Nov 03 20:39:01.564509 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:01.564302    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-c9kvc"
Nov 03 20:39:01.564509 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:01.564334    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-c9kvc" event={"ID":"1351662d-6ca4-430b-8bbe-0ca1429df833","Type":"ContainerDied","Data":"8977e092d4a9cfa729f07741dd5894f87839bdf7aa38cf0b489c3d40058255c4"}
Nov 03 20:39:01.564509 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:01.564392    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-c9kvc" event={"ID":"1351662d-6ca4-430b-8bbe-0ca1429df833","Type":"ContainerDied","Data":"da42034e32112e0677143e671754162e71f1094f4321bf952e8b0c9de60c2720"}
Nov 03 20:39:01.564509 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:01.564417    5570 scope.go:122] "RemoveContainer" containerID="8977e092d4a9cfa729f07741dd5894f87839bdf7aa38cf0b489c3d40058255c4"
Nov 03 20:39:01.575793 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:01.575744    5570 scope.go:122] "RemoveContainer" containerID="8977e092d4a9cfa729f07741dd5894f87839bdf7aa38cf0b489c3d40058255c4"
Nov 03 20:39:01.576245 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:39:01.576221    5570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8977e092d4a9cfa729f07741dd5894f87839bdf7aa38cf0b489c3d40058255c4\": container with ID starting with 8977e092d4a9cfa729f07741dd5894f87839bdf7aa38cf0b489c3d40058255c4 not found: ID does not exist" containerID="8977e092d4a9cfa729f07741dd5894f87839bdf7aa38cf0b489c3d40058255c4"
Nov 03 20:39:01.576343 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:01.576254    5570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8977e092d4a9cfa729f07741dd5894f87839bdf7aa38cf0b489c3d40058255c4"} err="failed to get container status \"8977e092d4a9cfa729f07741dd5894f87839bdf7aa38cf0b489c3d40058255c4\": rpc error: code = NotFound desc = could not find container \"8977e092d4a9cfa729f07741dd5894f87839bdf7aa38cf0b489c3d40058255c4\": container with ID starting with 8977e092d4a9cfa729f07741dd5894f87839bdf7aa38cf0b489c3d40058255c4 not found: ID does not exist"
Nov 03 20:39:01.632083 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:01.632045    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-c9kvc"]
Nov 03 20:39:01.644629 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:01.644594    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-482722-openshift-default-7645487df-c9kvc"]
Nov 03 20:39:01.906225 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:01.906144    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-66656cd6fd-9xdxz"]
Nov 03 20:39:01.906610 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:01.906564    5570 cpu_manager.go:414] "containerMap: removing container" podUID="9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a" containerName="main"
Nov 03 20:39:01.906610 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:01.906576    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a" containerName="main"
Nov 03 20:39:01.906610 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:01.906596    5570 cpu_manager.go:414] "containerMap: removing container" podUID="9a97f9cc-2261-471d-9053-51ee965850bd" containerName="main"
Nov 03 20:39:01.906610 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:01.906601    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a97f9cc-2261-471d-9053-51ee965850bd" containerName="main"
Nov 03 20:39:01.906750 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:01.906615    5570 cpu_manager.go:414] "containerMap: removing container" podUID="1351662d-6ca4-430b-8bbe-0ca1429df833" containerName="istio-proxy"
Nov 03 20:39:01.906750 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:01.906621    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1351662d-6ca4-430b-8bbe-0ca1429df833" containerName="istio-proxy"
Nov 03 20:39:01.906750 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:01.906639    5570 cpu_manager.go:414] "containerMap: removing container" podUID="369045c3-4d6a-49a7-9bc6-f2939883bee8" containerName="main"
Nov 03 20:39:01.906750 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:01.906643    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="369045c3-4d6a-49a7-9bc6-f2939883bee8" containerName="main"
Nov 03 20:39:01.906750 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:01.906650    5570 cpu_manager.go:414] "containerMap: removing container" podUID="f075a0ee-7327-4a89-bd17-a02edd236f9f" containerName="revoke-keys"
Nov 03 20:39:01.906750 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:01.906655    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f075a0ee-7327-4a89-bd17-a02edd236f9f" containerName="revoke-keys"
Nov 03 20:39:01.906750 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:01.906705    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="369045c3-4d6a-49a7-9bc6-f2939883bee8" containerName="main"
Nov 03 20:39:01.906750 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:01.906713    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="f075a0ee-7327-4a89-bd17-a02edd236f9f" containerName="revoke-keys"
Nov 03 20:39:01.906750 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:01.906720    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d3c8e6e-078d-4b56-be86-b3a5eb4e2d9a" containerName="main"
Nov 03 20:39:01.906750 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:01.906727    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a97f9cc-2261-471d-9053-51ee965850bd" containerName="main"
Nov 03 20:39:01.906750 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:01.906733    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="516f80b2-26e4-44d2-b97a-6e740d0b53e6" containerName="istio-proxy"
Nov 03 20:39:01.906750 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:01.906739    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="1351662d-6ca4-430b-8bbe-0ca1429df833" containerName="istio-proxy"
Nov 03 20:39:01.911758 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:01.911736    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-66656cd6fd-9xdxz"
Nov 03 20:39:01.927863 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:01.927830    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-w4zzp\""
Nov 03 20:39:01.927863 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:01.927850    5570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Nov 03 20:39:01.927863 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:01.927842    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-66656cd6fd-9xdxz"]
Nov 03 20:39:01.928186 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:01.927940    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Nov 03 20:39:01.939146 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:01.939117    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Nov 03 20:39:01.967486 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:01.967439    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1351662d-6ca4-430b-8bbe-0ca1429df833" path="/var/lib/kubelet/pods/1351662d-6ca4-430b-8bbe-0ca1429df833/volumes"
Nov 03 20:39:01.978196 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:01.978157    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/23769922-88e1-4670-a094-22a74aea2e11-webhook-cert\") pod \"maas-controller-66656cd6fd-9xdxz\" (UID: \"23769922-88e1-4670-a094-22a74aea2e11\") " pod="opendatahub/maas-controller-66656cd6fd-9xdxz"
Nov 03 20:39:01.978383 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:01.978267    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrtzc\" (UniqueName: \"kubernetes.io/projected/23769922-88e1-4670-a094-22a74aea2e11-kube-api-access-lrtzc\") pod \"maas-controller-66656cd6fd-9xdxz\" (UID: \"23769922-88e1-4670-a094-22a74aea2e11\") " pod="opendatahub/maas-controller-66656cd6fd-9xdxz"
Nov 03 20:39:01.978476 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:01.978454    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/23769922-88e1-4670-a094-22a74aea2e11-metrics-certs\") pod \"maas-controller-66656cd6fd-9xdxz\" (UID: \"23769922-88e1-4670-a094-22a74aea2e11\") " pod="opendatahub/maas-controller-66656cd6fd-9xdxz"
Nov 03 20:39:02.080140 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:02.079540    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/23769922-88e1-4670-a094-22a74aea2e11-metrics-certs\") pod \"maas-controller-66656cd6fd-9xdxz\" (UID: \"23769922-88e1-4670-a094-22a74aea2e11\") " pod="opendatahub/maas-controller-66656cd6fd-9xdxz"
Nov 03 20:39:02.080140 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:02.079658    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/23769922-88e1-4670-a094-22a74aea2e11-webhook-cert\") pod \"maas-controller-66656cd6fd-9xdxz\" (UID: \"23769922-88e1-4670-a094-22a74aea2e11\") " pod="opendatahub/maas-controller-66656cd6fd-9xdxz"
Nov 03 20:39:02.080140 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:02.079695    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrtzc\" (UniqueName: \"kubernetes.io/projected/23769922-88e1-4670-a094-22a74aea2e11-kube-api-access-lrtzc\") pod \"maas-controller-66656cd6fd-9xdxz\" (UID: \"23769922-88e1-4670-a094-22a74aea2e11\") " pod="opendatahub/maas-controller-66656cd6fd-9xdxz"
Nov 03 20:39:02.082994 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:02.082964    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/23769922-88e1-4670-a094-22a74aea2e11-metrics-certs\") pod \"maas-controller-66656cd6fd-9xdxz\" (UID: \"23769922-88e1-4670-a094-22a74aea2e11\") " pod="opendatahub/maas-controller-66656cd6fd-9xdxz"
Nov 03 20:39:02.083158 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:02.083005    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/23769922-88e1-4670-a094-22a74aea2e11-webhook-cert\") pod \"maas-controller-66656cd6fd-9xdxz\" (UID: \"23769922-88e1-4670-a094-22a74aea2e11\") " pod="opendatahub/maas-controller-66656cd6fd-9xdxz"
Nov 03 20:39:02.100400 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:02.100359    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrtzc\" (UniqueName: \"kubernetes.io/projected/23769922-88e1-4670-a094-22a74aea2e11-kube-api-access-lrtzc\") pod \"maas-controller-66656cd6fd-9xdxz\" (UID: \"23769922-88e1-4670-a094-22a74aea2e11\") " pod="opendatahub/maas-controller-66656cd6fd-9xdxz"
Nov 03 20:39:02.225296 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:02.225260    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-66656cd6fd-9xdxz"
Nov 03 20:39:02.526250 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:02.526217    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-66656cd6fd-9xdxz"]
Nov 03 20:39:02.528280 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:39:02.528249    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23769922_88e1_4670_a094_22a74aea2e11.slice/crio-5b0edff0e1d6d6b1e0d612a3eac743c3b794297deda0ba3e0a721bdaeda20fe4 WatchSource:0}: Error finding container 5b0edff0e1d6d6b1e0d612a3eac743c3b794297deda0ba3e0a721bdaeda20fe4: Status 404 returned error can't find the container with id 5b0edff0e1d6d6b1e0d612a3eac743c3b794297deda0ba3e0a721bdaeda20fe4
Nov 03 20:39:02.529559 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:02.529542    5570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 20:39:02.571861 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:02.571811    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-66656cd6fd-9xdxz" event={"ID":"23769922-88e1-4670-a094-22a74aea2e11","Type":"ContainerStarted","Data":"5b0edff0e1d6d6b1e0d612a3eac743c3b794297deda0ba3e0a721bdaeda20fe4"}
Nov 03 20:39:04.586618 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:04.586578    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-66656cd6fd-9xdxz" event={"ID":"23769922-88e1-4670-a094-22a74aea2e11","Type":"ContainerStarted","Data":"780fe02a475fc627441ef23ada5d025c8cd83bb7f7283833365edf3b35a88129"}
Nov 03 20:39:04.587062 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:04.586742    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-66656cd6fd-9xdxz"
Nov 03 20:39:04.630371 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:04.630319    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-66656cd6fd-9xdxz" podStartSLOduration=1.979712665 podStartE2EDuration="3.630304311s" totalImagesPullingTime="1.650591646s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:39:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:39:04.630106151 +0000 UTC m=+2078.782768037" watchObservedRunningTime="2026-11-03 20:39:04.630304311 +0000 UTC m=+2078.782966192"
Nov 03 20:39:10.026902 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:10.026859    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1cc51cc3-k5g2m"]
Nov 03 20:39:10.040233 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:10.040194    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1cc51cc3-k5g2m"]
Nov 03 20:39:11.962815 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:11.962762    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89266b89-4697-45b3-af99-c2dddf7034bc" path="/var/lib/kubelet/pods/89266b89-4697-45b3-af99-c2dddf7034bc/volumes"
Nov 03 20:39:15.594458 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:15.594428    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-66656cd6fd-9xdxz"
Nov 03 20:39:21.412756 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:21.412719    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-62krd"]
Nov 03 20:39:21.416119 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:21.416098    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-62krd"
Nov 03 20:39:21.436632 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:21.436600    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-62krd"]
Nov 03 20:39:21.447803 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:21.447735    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:39:21.447803 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:21.447735    5570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 20:39:21.448113 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:21.447946    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l7s5w\""
Nov 03 20:39:21.448163 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:21.448117    5570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 20:39:21.572032 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:21.571997    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kktsr\" (UniqueName: \"kubernetes.io/projected/721644ce-8cbd-4514-ae21-586c296a6773-kube-api-access-kktsr\") pod \"maas-api-revoke-sub-85237e45-62krd\" (UID: \"721644ce-8cbd-4514-ae21-586c296a6773\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-62krd"
Nov 03 20:39:21.572243 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:21.572112    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/721644ce-8cbd-4514-ae21-586c296a6773-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-62krd\" (UID: \"721644ce-8cbd-4514-ae21-586c296a6773\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-62krd"
Nov 03 20:39:21.572243 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:21.572163    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/721644ce-8cbd-4514-ae21-586c296a6773-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-62krd\" (UID: \"721644ce-8cbd-4514-ae21-586c296a6773\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-62krd"
Nov 03 20:39:21.672866 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:21.672753    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/721644ce-8cbd-4514-ae21-586c296a6773-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-62krd\" (UID: \"721644ce-8cbd-4514-ae21-586c296a6773\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-62krd"
Nov 03 20:39:21.672866 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:21.672806    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/721644ce-8cbd-4514-ae21-586c296a6773-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-62krd\" (UID: \"721644ce-8cbd-4514-ae21-586c296a6773\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-62krd"
Nov 03 20:39:21.673136 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:21.672873    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kktsr\" (UniqueName: \"kubernetes.io/projected/721644ce-8cbd-4514-ae21-586c296a6773-kube-api-access-kktsr\") pod \"maas-api-revoke-sub-85237e45-62krd\" (UID: \"721644ce-8cbd-4514-ae21-586c296a6773\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-62krd"
Nov 03 20:39:21.673456 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:21.673434    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/721644ce-8cbd-4514-ae21-586c296a6773-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-62krd\" (UID: \"721644ce-8cbd-4514-ae21-586c296a6773\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-62krd"
Nov 03 20:39:21.675369 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:21.675336    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/721644ce-8cbd-4514-ae21-586c296a6773-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-62krd\" (UID: \"721644ce-8cbd-4514-ae21-586c296a6773\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-62krd"
Nov 03 20:39:21.693832 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:21.693765    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kktsr\" (UniqueName: \"kubernetes.io/projected/721644ce-8cbd-4514-ae21-586c296a6773-kube-api-access-kktsr\") pod \"maas-api-revoke-sub-85237e45-62krd\" (UID: \"721644ce-8cbd-4514-ae21-586c296a6773\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-62krd"
Nov 03 20:39:21.730113 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:21.730074    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-62krd"
Nov 03 20:39:22.029632 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:22.029603    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-62krd"]
Nov 03 20:39:22.031838 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:39:22.031807    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod721644ce_8cbd_4514_ae21_586c296a6773.slice/crio-337d678be0372285d02a738ec83794e95146d2d4b68fc21cf8b374202e1440a9 WatchSource:0}: Error finding container 337d678be0372285d02a738ec83794e95146d2d4b68fc21cf8b374202e1440a9: Status 404 returned error can't find the container with id 337d678be0372285d02a738ec83794e95146d2d4b68fc21cf8b374202e1440a9
Nov 03 20:39:22.679292 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:22.679172    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-62krd" event={"ID":"721644ce-8cbd-4514-ae21-586c296a6773","Type":"ContainerStarted","Data":"ad57d654eb4ed4dcb230c25d2971137825f128848dba78ebe27ed2618b8ad233"}
Nov 03 20:39:22.679292 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:22.679244    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-62krd" event={"ID":"721644ce-8cbd-4514-ae21-586c296a6773","Type":"ContainerStarted","Data":"337d678be0372285d02a738ec83794e95146d2d4b68fc21cf8b374202e1440a9"}
Nov 03 20:39:22.721608 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:22.721542    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-62krd" podStartSLOduration=1.721527124 podStartE2EDuration="1.721527124s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:39:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:39:22.720013457 +0000 UTC m=+2096.872675570" watchObservedRunningTime="2026-11-03 20:39:22.721527124 +0000 UTC m=+2096.874189009"
Nov 03 20:39:23.693868 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:23.693833    5570 generic.go:356] "Generic (PLEG): container finished" podID="721644ce-8cbd-4514-ae21-586c296a6773" containerID="ad57d654eb4ed4dcb230c25d2971137825f128848dba78ebe27ed2618b8ad233" exitCode=0
Nov 03 20:39:23.694295 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:23.693923    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-62krd" event={"ID":"721644ce-8cbd-4514-ae21-586c296a6773","Type":"ContainerDied","Data":"ad57d654eb4ed4dcb230c25d2971137825f128848dba78ebe27ed2618b8ad233"}
Nov 03 20:39:24.824623 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:24.824597    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-62krd"
Nov 03 20:39:25.005685 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:25.005590    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/721644ce-8cbd-4514-ae21-586c296a6773-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/721644ce-8cbd-4514-ae21-586c296a6773-maas-api-xks-ca\") pod \"721644ce-8cbd-4514-ae21-586c296a6773\" (UID: \"721644ce-8cbd-4514-ae21-586c296a6773\") "
Nov 03 20:39:25.005685 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:25.005649    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/721644ce-8cbd-4514-ae21-586c296a6773-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/721644ce-8cbd-4514-ae21-586c296a6773-maas-api-openshift-ca\") pod \"721644ce-8cbd-4514-ae21-586c296a6773\" (UID: \"721644ce-8cbd-4514-ae21-586c296a6773\") "
Nov 03 20:39:25.005981 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:25.005803    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/721644ce-8cbd-4514-ae21-586c296a6773-kube-api-access-kktsr\" (UniqueName: \"kubernetes.io/projected/721644ce-8cbd-4514-ae21-586c296a6773-kube-api-access-kktsr\") pod \"721644ce-8cbd-4514-ae21-586c296a6773\" (UID: \"721644ce-8cbd-4514-ae21-586c296a6773\") "
Nov 03 20:39:25.006088 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:25.006061    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/721644ce-8cbd-4514-ae21-586c296a6773-maas-api-openshift-ca" pod "721644ce-8cbd-4514-ae21-586c296a6773" (UID: "721644ce-8cbd-4514-ae21-586c296a6773"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:39:25.007922 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:25.007874    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/721644ce-8cbd-4514-ae21-586c296a6773-kube-api-access-kktsr" pod "721644ce-8cbd-4514-ae21-586c296a6773" (UID: "721644ce-8cbd-4514-ae21-586c296a6773"). InnerVolumeSpecName "kube-api-access-kktsr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:39:25.008027 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:25.007897    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/721644ce-8cbd-4514-ae21-586c296a6773-maas-api-xks-ca" pod "721644ce-8cbd-4514-ae21-586c296a6773" (UID: "721644ce-8cbd-4514-ae21-586c296a6773"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:39:25.107340 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:25.107296    5570 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/721644ce-8cbd-4514-ae21-586c296a6773-maas-api-xks-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:39:25.107340 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:25.107332    5570 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/721644ce-8cbd-4514-ae21-586c296a6773-maas-api-openshift-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:39:25.107340 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:25.107345    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kktsr\" (UniqueName: \"kubernetes.io/projected/721644ce-8cbd-4514-ae21-586c296a6773-kube-api-access-kktsr\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:39:25.705263 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:25.705229    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-62krd"
Nov 03 20:39:25.705263 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:25.705234    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-62krd" event={"ID":"721644ce-8cbd-4514-ae21-586c296a6773","Type":"ContainerDied","Data":"337d678be0372285d02a738ec83794e95146d2d4b68fc21cf8b374202e1440a9"}
Nov 03 20:39:25.705485 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:25.705283    5570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="337d678be0372285d02a738ec83794e95146d2d4b68fc21cf8b374202e1440a9"
Nov 03 20:39:27.228182 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:39:27.228143    5570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8bwhz" containerName="extractor"
Nov 03 20:39:33.151060 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:33.151021    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-66656cd6fd-9xdxz"]
Nov 03 20:39:33.151634 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:33.151220    5570 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-66656cd6fd-9xdxz" podUID="23769922-88e1-4670-a094-22a74aea2e11" containerName="manager" containerID="cri-o://780fe02a475fc627441ef23ada5d025c8cd83bb7f7283833365edf3b35a88129" gracePeriod=30
Nov 03 20:39:33.395556 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:33.395527    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-66656cd6fd-9xdxz"
Nov 03 20:39:33.473074 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:33.473038    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/23769922-88e1-4670-a094-22a74aea2e11-metrics-certs\" (UniqueName: \"kubernetes.io/secret/23769922-88e1-4670-a094-22a74aea2e11-metrics-certs\") pod \"23769922-88e1-4670-a094-22a74aea2e11\" (UID: \"23769922-88e1-4670-a094-22a74aea2e11\") "
Nov 03 20:39:33.473303 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:33.473104    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/23769922-88e1-4670-a094-22a74aea2e11-webhook-cert\" (UniqueName: \"kubernetes.io/secret/23769922-88e1-4670-a094-22a74aea2e11-webhook-cert\") pod \"23769922-88e1-4670-a094-22a74aea2e11\" (UID: \"23769922-88e1-4670-a094-22a74aea2e11\") "
Nov 03 20:39:33.473303 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:33.473145    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23769922-88e1-4670-a094-22a74aea2e11-kube-api-access-lrtzc\" (UniqueName: \"kubernetes.io/projected/23769922-88e1-4670-a094-22a74aea2e11-kube-api-access-lrtzc\") pod \"23769922-88e1-4670-a094-22a74aea2e11\" (UID: \"23769922-88e1-4670-a094-22a74aea2e11\") "
Nov 03 20:39:33.475475 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:33.475441    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23769922-88e1-4670-a094-22a74aea2e11-kube-api-access-lrtzc" pod "23769922-88e1-4670-a094-22a74aea2e11" (UID: "23769922-88e1-4670-a094-22a74aea2e11"). InnerVolumeSpecName "kube-api-access-lrtzc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:39:33.475646 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:33.475519    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23769922-88e1-4670-a094-22a74aea2e11-webhook-cert" pod "23769922-88e1-4670-a094-22a74aea2e11" (UID: "23769922-88e1-4670-a094-22a74aea2e11"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:39:33.475646 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:33.475587    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23769922-88e1-4670-a094-22a74aea2e11-metrics-certs" pod "23769922-88e1-4670-a094-22a74aea2e11" (UID: "23769922-88e1-4670-a094-22a74aea2e11"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:39:33.573973 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:33.573911    5570 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/23769922-88e1-4670-a094-22a74aea2e11-metrics-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:39:33.573973 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:33.573960    5570 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/23769922-88e1-4670-a094-22a74aea2e11-webhook-cert\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:39:33.573973 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:33.573972    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrtzc\" (UniqueName: \"kubernetes.io/projected/23769922-88e1-4670-a094-22a74aea2e11-kube-api-access-lrtzc\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:39:33.749805 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:33.749693    5570 generic.go:356] "Generic (PLEG): container finished" podID="23769922-88e1-4670-a094-22a74aea2e11" containerID="780fe02a475fc627441ef23ada5d025c8cd83bb7f7283833365edf3b35a88129" exitCode=0
Nov 03 20:39:33.749805 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:33.749749    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-66656cd6fd-9xdxz"
Nov 03 20:39:33.749805 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:33.749791    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-66656cd6fd-9xdxz" event={"ID":"23769922-88e1-4670-a094-22a74aea2e11","Type":"ContainerDied","Data":"780fe02a475fc627441ef23ada5d025c8cd83bb7f7283833365edf3b35a88129"}
Nov 03 20:39:33.750060 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:33.749850    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-66656cd6fd-9xdxz" event={"ID":"23769922-88e1-4670-a094-22a74aea2e11","Type":"ContainerDied","Data":"5b0edff0e1d6d6b1e0d612a3eac743c3b794297deda0ba3e0a721bdaeda20fe4"}
Nov 03 20:39:33.750060 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:33.749873    5570 scope.go:122] "RemoveContainer" containerID="780fe02a475fc627441ef23ada5d025c8cd83bb7f7283833365edf3b35a88129"
Nov 03 20:39:33.759301 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:33.759279    5570 scope.go:122] "RemoveContainer" containerID="780fe02a475fc627441ef23ada5d025c8cd83bb7f7283833365edf3b35a88129"
Nov 03 20:39:33.759585 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:39:33.759562    5570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"780fe02a475fc627441ef23ada5d025c8cd83bb7f7283833365edf3b35a88129\": container with ID starting with 780fe02a475fc627441ef23ada5d025c8cd83bb7f7283833365edf3b35a88129 not found: ID does not exist" containerID="780fe02a475fc627441ef23ada5d025c8cd83bb7f7283833365edf3b35a88129"
Nov 03 20:39:33.759641 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:33.759594    5570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"780fe02a475fc627441ef23ada5d025c8cd83bb7f7283833365edf3b35a88129"} err="failed to get container status \"780fe02a475fc627441ef23ada5d025c8cd83bb7f7283833365edf3b35a88129\": rpc error: code = NotFound desc = could not find container \"780fe02a475fc627441ef23ada5d025c8cd83bb7f7283833365edf3b35a88129\": container with ID starting with 780fe02a475fc627441ef23ada5d025c8cd83bb7f7283833365edf3b35a88129 not found: ID does not exist"
Nov 03 20:39:33.803664 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:33.803629    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-66656cd6fd-9xdxz"]
Nov 03 20:39:33.816974 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:33.816939    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-66656cd6fd-9xdxz"]
Nov 03 20:39:33.963011 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:33.962976    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23769922-88e1-4670-a094-22a74aea2e11" path="/var/lib/kubelet/pods/23769922-88e1-4670-a094-22a74aea2e11/volumes"
Nov 03 20:39:34.932510 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:34.932479    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-66656cd6fd-qq5n6"]
Nov 03 20:39:34.932954 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:34.932882    5570 cpu_manager.go:414] "containerMap: removing container" podUID="721644ce-8cbd-4514-ae21-586c296a6773" containerName="revoke-keys"
Nov 03 20:39:34.932954 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:34.932895    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="721644ce-8cbd-4514-ae21-586c296a6773" containerName="revoke-keys"
Nov 03 20:39:34.932954 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:34.932907    5570 cpu_manager.go:414] "containerMap: removing container" podUID="23769922-88e1-4670-a094-22a74aea2e11" containerName="manager"
Nov 03 20:39:34.932954 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:34.932912    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23769922-88e1-4670-a094-22a74aea2e11" containerName="manager"
Nov 03 20:39:34.933111 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:34.932977    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="721644ce-8cbd-4514-ae21-586c296a6773" containerName="revoke-keys"
Nov 03 20:39:34.933111 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:34.932986    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="23769922-88e1-4670-a094-22a74aea2e11" containerName="manager"
Nov 03 20:39:34.937670 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:34.937644    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-66656cd6fd-qq5n6"
Nov 03 20:39:34.954047 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:34.954015    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-66656cd6fd-qq5n6"]
Nov 03 20:39:34.965969 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:34.965899    5570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Nov 03 20:39:34.966164 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:34.966007    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Nov 03 20:39:34.966967 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:34.966947    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-w4zzp\""
Nov 03 20:39:34.967089 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:34.967073    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Nov 03 20:39:34.986822 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:34.986750    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b2accca8-9328-4135-9957-d2e947b104a7-webhook-cert\") pod \"maas-controller-66656cd6fd-qq5n6\" (UID: \"b2accca8-9328-4135-9957-d2e947b104a7\") " pod="opendatahub/maas-controller-66656cd6fd-qq5n6"
Nov 03 20:39:34.987045 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:34.986907    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bvsm\" (UniqueName: \"kubernetes.io/projected/b2accca8-9328-4135-9957-d2e947b104a7-kube-api-access-6bvsm\") pod \"maas-controller-66656cd6fd-qq5n6\" (UID: \"b2accca8-9328-4135-9957-d2e947b104a7\") " pod="opendatahub/maas-controller-66656cd6fd-qq5n6"
Nov 03 20:39:34.987045 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:34.986953    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2accca8-9328-4135-9957-d2e947b104a7-metrics-certs\") pod \"maas-controller-66656cd6fd-qq5n6\" (UID: \"b2accca8-9328-4135-9957-d2e947b104a7\") " pod="opendatahub/maas-controller-66656cd6fd-qq5n6"
Nov 03 20:39:35.041383 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:35.041345    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-78txb"]
Nov 03 20:39:35.046576 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:35.046549    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-78txb"
Nov 03 20:39:35.054861 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:35.054832    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-msmxq"]
Nov 03 20:39:35.059089 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:35.059064    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-wpxcz"]
Nov 03 20:39:35.059228 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:35.059178    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-msmxq"
Nov 03 20:39:35.064726 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:35.064673    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-wpxcz"
Nov 03 20:39:35.065432 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:35.065405    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-78txb"]
Nov 03 20:39:35.083415 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:35.083382    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-wpxcz"]
Nov 03 20:39:35.085364 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:35.085335    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-msmxq"]
Nov 03 20:39:35.088067 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:35.088040    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b65b5a79-c181-45ec-9cae-ebb5e5bbd865-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-78txb\" (UID: \"b65b5a79-c181-45ec-9cae-ebb5e5bbd865\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-78txb"
Nov 03 20:39:35.088216 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:35.088071    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b65b5a79-c181-45ec-9cae-ebb5e5bbd865-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-78txb\" (UID: \"b65b5a79-c181-45ec-9cae-ebb5e5bbd865\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-78txb"
Nov 03 20:39:35.088216 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:35.088099    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ebdd5ef-b321-4dfa-8c3e-1ad7c249a0ff-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-msmxq\" (UID: \"9ebdd5ef-b321-4dfa-8c3e-1ad7c249a0ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-msmxq"
Nov 03 20:39:35.088216 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:35.088119    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bvsm\" (UniqueName: \"kubernetes.io/projected/b2accca8-9328-4135-9957-d2e947b104a7-kube-api-access-6bvsm\") pod \"maas-controller-66656cd6fd-qq5n6\" (UID: \"b2accca8-9328-4135-9957-d2e947b104a7\") " pod="opendatahub/maas-controller-66656cd6fd-qq5n6"
Nov 03 20:39:35.088376 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:35.088211    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b65b5a79-c181-45ec-9cae-ebb5e5bbd865-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-78txb\" (UID: \"b65b5a79-c181-45ec-9cae-ebb5e5bbd865\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-78txb"
Nov 03 20:39:35.088458 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:35.088402    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2accca8-9328-4135-9957-d2e947b104a7-metrics-certs\") pod \"maas-controller-66656cd6fd-qq5n6\" (UID: \"b2accca8-9328-4135-9957-d2e947b104a7\") " pod="opendatahub/maas-controller-66656cd6fd-qq5n6"
Nov 03 20:39:35.088522 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:35.088466    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36477cba-9827-41ba-997d-d3d646ef4904-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-wpxcz\" (UID: \"36477cba-9827-41ba-997d-d3d646ef4904\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-wpxcz"
Nov 03 20:39:35.088522 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:35.088499    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b65b5a79-c181-45ec-9cae-ebb5e5bbd865-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-78txb\" (UID: \"b65b5a79-c181-45ec-9cae-ebb5e5bbd865\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-78txb"
Nov 03 20:39:35.088659 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:35.088549    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36477cba-9827-41ba-997d-d3d646ef4904-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-wpxcz\" (UID: \"36477cba-9827-41ba-997d-d3d646ef4904\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-wpxcz"
Nov 03 20:39:35.088659 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:35.088628    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b65b5a79-c181-45ec-9cae-ebb5e5bbd865-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-78txb\" (UID: \"b65b5a79-c181-45ec-9cae-ebb5e5bbd865\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-78txb"
Nov 03 20:39:35.088748 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:35.088659    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ebdd5ef-b321-4dfa-8c3e-1ad7c249a0ff-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-msmxq\" (UID: \"9ebdd5ef-b321-4dfa-8c3e-1ad7c249a0ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-msmxq"
Nov 03 20:39:35.088748 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:35.088687    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b65b5a79-c181-45ec-9cae-ebb5e5bbd865-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-78txb\" (UID: \"b65b5a79-c181-45ec-9cae-ebb5e5bbd865\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-78txb"
Nov 03 20:39:35.088884 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:35.088792    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b65b5a79-c181-45ec-9cae-ebb5e5bbd865-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-78txb\" (UID: \"b65b5a79-c181-45ec-9cae-ebb5e5bbd865\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-78txb"
Nov 03 20:39:35.088884 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:35.088822    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36477cba-9827-41ba-997d-d3d646ef4904-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-wpxcz\" (UID: \"36477cba-9827-41ba-997d-d3d646ef4904\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-wpxcz"
Nov 03 20:39:35.088884 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:35.088845    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36477cba-9827-41ba-997d-d3d646ef4904-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-wpxcz\" (UID: \"36477cba-9827-41ba-997d-d3d646ef4904\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-wpxcz"
Nov 03 20:39:35.089123 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:35.088907    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36477cba-9827-41ba-997d-d3d646ef4904-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-wpxcz\" (UID: \"36477cba-9827-41ba-997d-d3d646ef4904\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-wpxcz"
Nov 03 20:39:35.089123 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:35.088951    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ebdd5ef-b321-4dfa-8c3e-1ad7c249a0ff-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-msmxq\" (UID: \"9ebdd5ef-b321-4dfa-8c3e-1ad7c249a0ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-msmxq"
Nov 03 20:39:35.089123 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:35.088994    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36477cba-9827-41ba-997d-d3d646ef4904-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-wpxcz\" (UID: \"36477cba-9827-41ba-997d-d3d646ef4904\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-wpxcz"
Nov 03 20:39:35.089123 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:35.089082    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrp6v\" (UniqueName: \"kubernetes.io/projected/b65b5a79-c181-45ec-9cae-ebb5e5bbd865-kube-api-access-lrp6v\") pod \"maas-default-gateway-openshift-default-694db55c87-78txb\" (UID: \"b65b5a79-c181-45ec-9cae-ebb5e5bbd865\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-78txb"
Nov 03 20:39:35.089260 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:35.089125    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ebdd5ef-b321-4dfa-8c3e-1ad7c249a0ff-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-msmxq\" (UID: \"9ebdd5ef-b321-4dfa-8c3e-1ad7c249a0ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-msmxq"
Nov 03 20:39:35.089260 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:35.089179    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rd7zt\" (UniqueName: \"kubernetes.io/projected/36477cba-9827-41ba-997d-d3d646ef4904-kube-api-access-rd7zt\") pod \"maas-default-gateway-openshift-default-694db55c87-wpxcz\" (UID: \"36477cba-9827-41ba-997d-d3d646ef4904\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-wpxcz"
Nov 03 20:39:35.089326 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:35.089262    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36477cba-9827-41ba-997d-d3d646ef4904-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-wpxcz\" (UID: \"36477cba-9827-41ba-997d-d3d646ef4904\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-wpxcz"
Nov 03 20:39:35.089326 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:35.089299    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ebdd5ef-b321-4dfa-8c3e-1ad7c249a0ff-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-msmxq\" (UID: \"9ebdd5ef-b321-4dfa-8c3e-1ad7c249a0ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-msmxq"
Nov 03 20:39:35.089405 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:35.089381    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ebdd5ef-b321-4dfa-8c3e-1ad7c249a0ff-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-msmxq\" (UID: \"9ebdd5ef-b321-4dfa-8c3e-1ad7c249a0ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-msmxq"
Nov 03 20:39:35.089460 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:35.089423    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b65b5a79-c181-45ec-9cae-ebb5e5bbd865-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-78txb\" (UID: \"b65b5a79-c181-45ec-9cae-ebb5e5bbd865\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-78txb"
Nov 03 20:39:35.089605 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:35.089584    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ebdd5ef-b321-4dfa-8c3e-1ad7c249a0ff-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-msmxq\" (UID: \"9ebdd5ef-b321-4dfa-8c3e-1ad7c249a0ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-msmxq"
Nov 03 20:39:35.089709 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:35.089631    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b2accca8-9328-4135-9957-d2e947b104a7-webhook-cert\") pod \"maas-controller-66656cd6fd-qq5n6\" (UID: \"b2accca8-9328-4135-9957-d2e947b104a7\") " pod="opendatahub/maas-controller-66656cd6fd-qq5n6"
Nov 03 20:39:35.089709 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:35.089659    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4wfp\" (UniqueName: \"kubernetes.io/projected/9ebdd5ef-b321-4dfa-8c3e-1ad7c249a0ff-kube-api-access-m4wfp\") pod \"maas-default-gateway-openshift-default-694db55c87-msmxq\" (UID: \"9ebdd5ef-b321-4dfa-8c3e-1ad7c249a0ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-msmxq"
Nov 03 20:39:35.089709 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:35.089693    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ebdd5ef-b321-4dfa-8c3e-1ad7c249a0ff-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-msmxq\" (UID: \"9ebdd5ef-b321-4dfa-8c3e-1ad7c249a0ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-msmxq"
Nov 03 20:39:35.089918 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:35.089725    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36477cba-9827-41ba-997d-d3d646ef4904-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-wpxcz\" (UID: \"36477cba-9827-41ba-997d-d3d646ef4904\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-wpxcz"
Nov 03 20:39:35.091634 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:35.091263    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2accca8-9328-4135-9957-d2e947b104a7-metrics-certs\") pod \"maas-controller-66656cd6fd-qq5n6\" (UID: \"b2accca8-9328-4135-9957-d2e947b104a7\") " pod="opendatahub/maas-controller-66656cd6fd-qq5n6"
Nov 03 20:39:35.093615 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:35.093590    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b2accca8-9328-4135-9957-d2e947b104a7-webhook-cert\") pod \"maas-controller-66656cd6fd-qq5n6\" (UID: \"b2accca8-9328-4135-9957-d2e947b104a7\") " pod="opendatahub/maas-controller-66656cd6fd-qq5n6"
Nov 03 20:39:35.110193 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:35.110152    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bvsm\" (UniqueName: \"kubernetes.io/projected/b2accca8-9328-4135-9957-d2e947b104a7-kube-api-access-6bvsm\") pod \"maas-controller-66656cd6fd-qq5n6\" (UID: \"b2accca8-9328-4135-9957-d2e947b104a7\") " pod="opendatahub/maas-controller-66656cd6fd-qq5n6"
Nov 03 20:39:35.191078 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:35.191042    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b65b5a79-c181-45ec-9cae-ebb5e5bbd865-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-78txb\" (UID: \"b65b5a79-c181-45ec-9cae-ebb5e5bbd865\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-78txb"
Nov 03 20:39:35.191078 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:35.191080    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b65b5a79-c181-45ec-9cae-ebb5e5bbd865-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-78txb\" (UID: \"b65b5a79-c181-45ec-9cae-ebb5e5bbd865\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-78txb"
Nov 03 20:39:35.191348 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:35.191104    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ebdd5ef-b321-4dfa-8c3e-1ad7c249a0ff-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-msmxq\" (UID: \"9ebdd5ef-b321-4dfa-8c3e-1ad7c249a0ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-msmxq"
Nov 03 20:39:35.191348 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:35.191123    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b65b5a79-c181-45ec-9cae-ebb5e5bbd865-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-78txb\" (UID: \"b65b5a79-c181-45ec-9cae-ebb5e5bbd865\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-78txb"
Nov 03 20:39:35.191348 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:35.191148    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36477cba-9827-41ba-997d-d3d646ef4904-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-wpxcz\" (UID: \"36477cba-9827-41ba-997d-d3d646ef4904\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-wpxcz"
Nov 03 20:39:35.191348 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:35.191163    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b65b5a79-c181-45ec-9cae-ebb5e5bbd865-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-78txb\" (UID: \"b65b5a79-c181-45ec-9cae-ebb5e5bbd865\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-78txb"
Nov 03 20:39:35.191348 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:35.191184    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36477cba-9827-41ba-997d-d3d646ef4904-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-wpxcz\" (UID: \"36477cba-9827-41ba-997d-d3d646ef4904\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-wpxcz"
Nov 03 20:39:35.191348 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:35.191211    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b65b5a79-c181-45ec-9cae-ebb5e5bbd865-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-78txb\" (UID: \"b65b5a79-c181-45ec-9cae-ebb5e5bbd865\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-78txb"
Nov 03 20:39:35.191348 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:35.191234    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ebdd5ef-b321-4dfa-8c3e-1ad7c249a0ff-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-msmxq\" (UID: \"9ebdd5ef-b321-4dfa-8c3e-1ad7c249a0ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-msmxq"
Nov 03 20:39:35.191348 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:35.191259    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b65b5a79-c181-45ec-9cae-ebb5e5bbd865-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-78txb\" (UID: \"b65b5a79-c181-45ec-9cae-ebb5e5bbd865\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-78txb"
Nov 03 20:39:35.191348 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:35.191298    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b65b5a79-c181-45ec-9cae-ebb5e5bbd865-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-78txb\" (UID: \"b65b5a79-c181-45ec-9cae-ebb5e5bbd865\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-78txb"
Nov 03 20:39:35.191348 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:35.191322    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36477cba-9827-41ba-997d-d3d646ef4904-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-wpxcz\" (UID: \"36477cba-9827-41ba-997d-d3d646ef4904\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-wpxcz"
Nov 03 20:39:35.191348 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:35.191345    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36477cba-9827-41ba-997d-d3d646ef4904-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-wpxcz\" (UID: \"36477cba-9827-41ba-997d-d3d646ef4904\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-wpxcz"
Nov 03 20:39:35.192038 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:35.191368    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36477cba-9827-41ba-997d-d3d646ef4904-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-wpxcz\" (UID: \"36477cba-9827-41ba-997d-d3d646ef4904\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-wpxcz"
Nov 03 20:39:35.192038 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:35.191394    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ebdd5ef-b321-4dfa-8c3e-1ad7c249a0ff-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-msmxq\" (UID: \"9ebdd5ef-b321-4dfa-8c3e-1ad7c249a0ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-msmxq"
Nov 03 20:39:35.192038 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:35.191433    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36477cba-9827-41ba-997d-d3d646ef4904-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-wpxcz\" (UID: \"36477cba-9827-41ba-997d-d3d646ef4904\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-wpxcz"
Nov 03 20:39:35.192038 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:35.191462    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrp6v\" (UniqueName: \"kubernetes.io/projected/b65b5a79-c181-45ec-9cae-ebb5e5bbd865-kube-api-access-lrp6v\") pod \"maas-default-gateway-openshift-default-694db55c87-78txb\" (UID: \"b65b5a79-c181-45ec-9cae-ebb5e5bbd865\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-78txb"
Nov 03 20:39:35.192038 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:35.191485    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ebdd5ef-b321-4dfa-8c3e-1ad7c249a0ff-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-msmxq\" (UID: \"9ebdd5ef-b321-4dfa-8c3e-1ad7c249a0ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-msmxq"
Nov 03 20:39:35.192038 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:35.191507    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rd7zt\" (UniqueName: \"kubernetes.io/projected/36477cba-9827-41ba-997d-d3d646ef4904-kube-api-access-rd7zt\") pod \"maas-default-gateway-openshift-default-694db55c87-wpxcz\" (UID: \"36477cba-9827-41ba-997d-d3d646ef4904\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-wpxcz"
Nov 03 20:39:35.192038 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:35.191542    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36477cba-9827-41ba-997d-d3d646ef4904-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-wpxcz\" (UID: \"36477cba-9827-41ba-997d-d3d646ef4904\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-wpxcz"
Nov 03 20:39:35.192038 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:35.191556    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ebdd5ef-b321-4dfa-8c3e-1ad7c249a0ff-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-msmxq\" (UID: \"9ebdd5ef-b321-4dfa-8c3e-1ad7c249a0ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-msmxq"
Nov 03 20:39:35.192038 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:35.191570    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ebdd5ef-b321-4dfa-8c3e-1ad7c249a0ff-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-msmxq\" (UID: \"9ebdd5ef-b321-4dfa-8c3e-1ad7c249a0ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-msmxq"
Nov 03 20:39:35.192038 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:35.191611    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b65b5a79-c181-45ec-9cae-ebb5e5bbd865-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-78txb\" (UID: \"b65b5a79-c181-45ec-9cae-ebb5e5bbd865\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-78txb"
Nov 03 20:39:35.192038 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:35.191635    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ebdd5ef-b321-4dfa-8c3e-1ad7c249a0ff-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-msmxq\" (UID: \"9ebdd5ef-b321-4dfa-8c3e-1ad7c249a0ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-msmxq"
Nov 03 20:39:35.192038 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:35.191665    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b65b5a79-c181-45ec-9cae-ebb5e5bbd865-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-78txb\" (UID: \"b65b5a79-c181-45ec-9cae-ebb5e5bbd865\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-78txb"
Nov 03 20:39:35.192038 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:35.191691    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36477cba-9827-41ba-997d-d3d646ef4904-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-wpxcz\" (UID: \"36477cba-9827-41ba-997d-d3d646ef4904\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-wpxcz"
Nov 03 20:39:35.192038 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:35.191720    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ebdd5ef-b321-4dfa-8c3e-1ad7c249a0ff-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-msmxq\" (UID: \"9ebdd5ef-b321-4dfa-8c3e-1ad7c249a0ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-msmxq"
Nov 03 20:39:35.192038 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:35.191758    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4wfp\" (UniqueName: \"kubernetes.io/projected/9ebdd5ef-b321-4dfa-8c3e-1ad7c249a0ff-kube-api-access-m4wfp\") pod \"maas-default-gateway-openshift-default-694db55c87-msmxq\" (UID: \"9ebdd5ef-b321-4dfa-8c3e-1ad7c249a0ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-msmxq"
Nov 03 20:39:35.192038 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:35.191769    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b65b5a79-c181-45ec-9cae-ebb5e5bbd865-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-78txb\" (UID: \"b65b5a79-c181-45ec-9cae-ebb5e5bbd865\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-78txb"
Nov 03 20:39:35.192038 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:35.191810    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ebdd5ef-b321-4dfa-8c3e-1ad7c249a0ff-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-msmxq\" (UID: \"9ebdd5ef-b321-4dfa-8c3e-1ad7c249a0ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-msmxq"
Nov 03 20:39:35.192038 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:35.191943    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36477cba-9827-41ba-997d-d3d646ef4904-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-wpxcz\" (UID: \"36477cba-9827-41ba-997d-d3d646ef4904\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-wpxcz"
Nov 03 20:39:35.192038 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:35.192048    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36477cba-9827-41ba-997d-d3d646ef4904-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-wpxcz\" (UID: \"36477cba-9827-41ba-997d-d3d646ef4904\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-wpxcz"
Nov 03 20:39:35.193168 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:35.192067    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ebdd5ef-b321-4dfa-8c3e-1ad7c249a0ff-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-msmxq\" (UID: \"9ebdd5ef-b321-4dfa-8c3e-1ad7c249a0ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-msmxq"
Nov 03 20:39:35.193168 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:35.192135    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b65b5a79-c181-45ec-9cae-ebb5e5bbd865-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-78txb\" (UID: \"b65b5a79-c181-45ec-9cae-ebb5e5bbd865\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-78txb"
Nov 03 20:39:35.193168 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:35.192248    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b65b5a79-c181-45ec-9cae-ebb5e5bbd865-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-78txb\" (UID: \"b65b5a79-c181-45ec-9cae-ebb5e5bbd865\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-78txb"
Nov 03 20:39:35.193168 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:35.192418    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36477cba-9827-41ba-997d-d3d646ef4904-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-wpxcz\" (UID: \"36477cba-9827-41ba-997d-d3d646ef4904\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-wpxcz"
Nov 03 20:39:35.193168 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:35.192680    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36477cba-9827-41ba-997d-d3d646ef4904-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-wpxcz\" (UID: \"36477cba-9827-41ba-997d-d3d646ef4904\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-wpxcz"
Nov 03 20:39:35.193168 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:35.192801    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36477cba-9827-41ba-997d-d3d646ef4904-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-wpxcz\" (UID: \"36477cba-9827-41ba-997d-d3d646ef4904\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-wpxcz"
Nov 03 20:39:35.193168 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:35.193017    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ebdd5ef-b321-4dfa-8c3e-1ad7c249a0ff-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-msmxq\" (UID: \"9ebdd5ef-b321-4dfa-8c3e-1ad7c249a0ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-msmxq"
Nov 03 20:39:35.193168 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:35.193058    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ebdd5ef-b321-4dfa-8c3e-1ad7c249a0ff-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-msmxq\" (UID: \"9ebdd5ef-b321-4dfa-8c3e-1ad7c249a0ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-msmxq"
Nov 03 20:39:35.193599 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:35.193237    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ebdd5ef-b321-4dfa-8c3e-1ad7c249a0ff-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-msmxq\" (UID: \"9ebdd5ef-b321-4dfa-8c3e-1ad7c249a0ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-msmxq"
Nov 03 20:39:35.193599 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:35.193332    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b65b5a79-c181-45ec-9cae-ebb5e5bbd865-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-78txb\" (UID: \"b65b5a79-c181-45ec-9cae-ebb5e5bbd865\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-78txb"
Nov 03 20:39:35.194310 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:35.194285    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ebdd5ef-b321-4dfa-8c3e-1ad7c249a0ff-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-msmxq\" (UID: \"9ebdd5ef-b321-4dfa-8c3e-1ad7c249a0ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-msmxq"
Nov 03 20:39:35.194424 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:35.194285    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b65b5a79-c181-45ec-9cae-ebb5e5bbd865-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-78txb\" (UID: \"b65b5a79-c181-45ec-9cae-ebb5e5bbd865\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-78txb"
Nov 03 20:39:35.194791 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:35.194761    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b65b5a79-c181-45ec-9cae-ebb5e5bbd865-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-78txb\" (UID: \"b65b5a79-c181-45ec-9cae-ebb5e5bbd865\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-78txb"
Nov 03 20:39:35.195119 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:35.195097    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36477cba-9827-41ba-997d-d3d646ef4904-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-wpxcz\" (UID: \"36477cba-9827-41ba-997d-d3d646ef4904\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-wpxcz"
Nov 03 20:39:35.195755 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:35.195727    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36477cba-9827-41ba-997d-d3d646ef4904-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-wpxcz\" (UID: \"36477cba-9827-41ba-997d-d3d646ef4904\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-wpxcz"
Nov 03 20:39:35.196109 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:35.196091    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ebdd5ef-b321-4dfa-8c3e-1ad7c249a0ff-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-msmxq\" (UID: \"9ebdd5ef-b321-4dfa-8c3e-1ad7c249a0ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-msmxq"
Nov 03 20:39:35.212452 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:35.212119    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b65b5a79-c181-45ec-9cae-ebb5e5bbd865-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-78txb\" (UID: \"b65b5a79-c181-45ec-9cae-ebb5e5bbd865\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-78txb"
Nov 03 20:39:35.212452 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:35.212222    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36477cba-9827-41ba-997d-d3d646ef4904-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-wpxcz\" (UID: \"36477cba-9827-41ba-997d-d3d646ef4904\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-wpxcz"
Nov 03 20:39:35.213626 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:35.213603    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ebdd5ef-b321-4dfa-8c3e-1ad7c249a0ff-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-msmxq\" (UID: \"9ebdd5ef-b321-4dfa-8c3e-1ad7c249a0ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-msmxq"
Nov 03 20:39:35.214426 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:35.214393    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4wfp\" (UniqueName: \"kubernetes.io/projected/9ebdd5ef-b321-4dfa-8c3e-1ad7c249a0ff-kube-api-access-m4wfp\") pod \"maas-default-gateway-openshift-default-694db55c87-msmxq\" (UID: \"9ebdd5ef-b321-4dfa-8c3e-1ad7c249a0ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-msmxq"
Nov 03 20:39:35.214627 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:35.214607    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rd7zt\" (UniqueName: \"kubernetes.io/projected/36477cba-9827-41ba-997d-d3d646ef4904-kube-api-access-rd7zt\") pod \"maas-default-gateway-openshift-default-694db55c87-wpxcz\" (UID: \"36477cba-9827-41ba-997d-d3d646ef4904\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-wpxcz"
Nov 03 20:39:35.214871 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:35.214853    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrp6v\" (UniqueName: \"kubernetes.io/projected/b65b5a79-c181-45ec-9cae-ebb5e5bbd865-kube-api-access-lrp6v\") pod \"maas-default-gateway-openshift-default-694db55c87-78txb\" (UID: \"b65b5a79-c181-45ec-9cae-ebb5e5bbd865\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-78txb"
Nov 03 20:39:35.250248 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:35.250210    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-66656cd6fd-qq5n6"
Nov 03 20:39:35.362032 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:35.361997    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-78txb"
Nov 03 20:39:35.377552 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:35.377528    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-msmxq"
Nov 03 20:39:35.384681 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:35.384654    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-wpxcz"
Nov 03 20:39:35.548002 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:35.547973    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-66656cd6fd-qq5n6"]
Nov 03 20:39:35.550425 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:39:35.550395    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2accca8_9328_4135_9957_d2e947b104a7.slice/crio-757cefab9ef5b67d8819336d0d9d198103a3658717251f2afc564fa62ed818f2 WatchSource:0}: Error finding container 757cefab9ef5b67d8819336d0d9d198103a3658717251f2afc564fa62ed818f2: Status 404 returned error can't find the container with id 757cefab9ef5b67d8819336d0d9d198103a3658717251f2afc564fa62ed818f2
Nov 03 20:39:35.674380 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:35.674349    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-78txb"]
Nov 03 20:39:35.675168 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:39:35.675137    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb65b5a79_c181_45ec_9cae_ebb5e5bbd865.slice/crio-546f029aa875954ca0cedc1c91eafd0303f19ccddff5926a3ceeee5717b5ad87 WatchSource:0}: Error finding container 546f029aa875954ca0cedc1c91eafd0303f19ccddff5926a3ceeee5717b5ad87: Status 404 returned error can't find the container with id 546f029aa875954ca0cedc1c91eafd0303f19ccddff5926a3ceeee5717b5ad87
Nov 03 20:39:35.677596 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:35.677565    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:39:35.677662 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:35.677636    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:39:35.677701 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:35.677661    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:39:35.689027 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:35.689001    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-msmxq"]
Nov 03 20:39:35.690543 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:39:35.690450    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ebdd5ef_b321_4dfa_8c3e_1ad7c249a0ff.slice/crio-d0e5f45bf350e42d111f243649fb1c551b93765f5665cf8b8fda15881a173e59 WatchSource:0}: Error finding container d0e5f45bf350e42d111f243649fb1c551b93765f5665cf8b8fda15881a173e59: Status 404 returned error can't find the container with id d0e5f45bf350e42d111f243649fb1c551b93765f5665cf8b8fda15881a173e59
Nov 03 20:39:35.692910 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:35.692850    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:39:35.692991 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:35.692918    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:39:35.692991 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:35.692957    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:39:35.702755 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:35.702733    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-wpxcz"]
Nov 03 20:39:35.708504 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:35.708469    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:39:35.708626 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:35.708551    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:39:35.708626 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:35.708598    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:39:35.767124 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:35.767083    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-66656cd6fd-qq5n6" event={"ID":"b2accca8-9328-4135-9957-d2e947b104a7","Type":"ContainerStarted","Data":"757cefab9ef5b67d8819336d0d9d198103a3658717251f2afc564fa62ed818f2"}
Nov 03 20:39:35.770560 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:35.769854    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-78txb" event={"ID":"b65b5a79-c181-45ec-9cae-ebb5e5bbd865","Type":"ContainerStarted","Data":"7e4d616821d9277a5763c3cfee0a7c0f670d038f09abb402d6be5a144dc00501"}
Nov 03 20:39:35.770560 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:35.769921    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-78txb" event={"ID":"b65b5a79-c181-45ec-9cae-ebb5e5bbd865","Type":"ContainerStarted","Data":"546f029aa875954ca0cedc1c91eafd0303f19ccddff5926a3ceeee5717b5ad87"}
Nov 03 20:39:35.773943 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:35.773909    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-wpxcz" event={"ID":"36477cba-9827-41ba-997d-d3d646ef4904","Type":"ContainerStarted","Data":"aa0a928db833f451841c27fde990456bd643463b54faef043753295e4c976464"}
Nov 03 20:39:35.774897 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:35.774855    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-wpxcz" event={"ID":"36477cba-9827-41ba-997d-d3d646ef4904","Type":"ContainerStarted","Data":"b444a7cfbf40158dea982a98147598c4bfda4c28bfd9c0f029038cc1974a3ab7"}
Nov 03 20:39:35.777065 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:35.776318    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-msmxq" event={"ID":"9ebdd5ef-b321-4dfa-8c3e-1ad7c249a0ff","Type":"ContainerStarted","Data":"f057a1e1ca52d1eb107219b391b3918a4e390f8fea5117695d32c8c821fcd24b"}
Nov 03 20:39:35.777065 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:35.776357    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-msmxq" event={"ID":"9ebdd5ef-b321-4dfa-8c3e-1ad7c249a0ff","Type":"ContainerStarted","Data":"d0e5f45bf350e42d111f243649fb1c551b93765f5665cf8b8fda15881a173e59"}
Nov 03 20:39:35.815239 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:35.814954    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-78txb" podStartSLOduration=0.814922257 podStartE2EDuration="814.922257ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:39:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:39:35.814019617 +0000 UTC m=+2109.966681505" watchObservedRunningTime="2026-11-03 20:39:35.814922257 +0000 UTC m=+2109.967584144"
Nov 03 20:39:35.857802 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:35.857561    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-wpxcz" podStartSLOduration=0.857542663 podStartE2EDuration="857.542663ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:39:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:39:35.856388647 +0000 UTC m=+2110.009050557" watchObservedRunningTime="2026-11-03 20:39:35.857542663 +0000 UTC m=+2110.010204550"
Nov 03 20:39:35.900527 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:35.900468    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-msmxq" podStartSLOduration=0.900451214 podStartE2EDuration="900.451214ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:39:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:39:35.898912815 +0000 UTC m=+2110.051574698" watchObservedRunningTime="2026-11-03 20:39:35.900451214 +0000 UTC m=+2110.053113097"
Nov 03 20:39:36.367921 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:36.367000    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-78txb"
Nov 03 20:39:36.367921 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:36.367192    5570 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-78txb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body=
Nov 03 20:39:36.367921 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:36.367250    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-78txb" podUID="b65b5a79-c181-45ec-9cae-ebb5e5bbd865" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused"
Nov 03 20:39:36.383809 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:36.379725    5570 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-msmxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" start-of-body=
Nov 03 20:39:36.383809 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:36.379874    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-msmxq" podUID="9ebdd5ef-b321-4dfa-8c3e-1ad7c249a0ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused"
Nov 03 20:39:36.383809 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:36.379995    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-msmxq"
Nov 03 20:39:36.393799 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:36.389820    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-wpxcz"
Nov 03 20:39:36.393799 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:36.392505    5570 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-wpxcz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body=
Nov 03 20:39:36.393799 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:36.392661    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-wpxcz" podUID="36477cba-9827-41ba-997d-d3d646ef4904" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused"
Nov 03 20:39:36.814799 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:36.813061    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-66656cd6fd-qq5n6" event={"ID":"b2accca8-9328-4135-9957-d2e947b104a7","Type":"ContainerStarted","Data":"1d8417e4b166248a4c6cf936402b33f9a1d78119c38dd79a8ba50f8a8e088211"}
Nov 03 20:39:36.815075 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:36.815058    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-66656cd6fd-qq5n6"
Nov 03 20:39:37.367805 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:37.367279    5570 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-78txb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body=
Nov 03 20:39:37.367805 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:37.367373    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-78txb" podUID="b65b5a79-c181-45ec-9cae-ebb5e5bbd865" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused"
Nov 03 20:39:37.391801 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:37.389085    5570 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-wpxcz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body=
Nov 03 20:39:37.391801 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:37.389160    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-wpxcz" podUID="36477cba-9827-41ba-997d-d3d646ef4904" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused"
Nov 03 20:39:37.391801 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:37.389228    5570 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-msmxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" start-of-body=
Nov 03 20:39:37.391801 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:37.389247    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-msmxq" podUID="9ebdd5ef-b321-4dfa-8c3e-1ad7c249a0ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused"
Nov 03 20:39:38.365283 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:38.365238    5570 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-78txb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body=
Nov 03 20:39:38.365592 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:38.365566    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-78txb" podUID="b65b5a79-c181-45ec-9cae-ebb5e5bbd865" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused"
Nov 03 20:39:38.379289 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:38.379245    5570 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-msmxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" start-of-body=
Nov 03 20:39:38.379963 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:38.379929    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-msmxq" podUID="9ebdd5ef-b321-4dfa-8c3e-1ad7c249a0ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused"
Nov 03 20:39:38.386796 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:38.385765    5570 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-wpxcz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body=
Nov 03 20:39:38.386796 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:38.385847    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-wpxcz" podUID="36477cba-9827-41ba-997d-d3d646ef4904" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused"
Nov 03 20:39:39.370801 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:39.367910    5570 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-78txb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body=
Nov 03 20:39:39.370801 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:39.367998    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-78txb" podUID="b65b5a79-c181-45ec-9cae-ebb5e5bbd865" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused"
Nov 03 20:39:39.382800 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:39.380239    5570 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-msmxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" start-of-body=
Nov 03 20:39:39.382800 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:39.380305    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-msmxq" podUID="9ebdd5ef-b321-4dfa-8c3e-1ad7c249a0ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused"
Nov 03 20:39:39.386799 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:39.385199    5570 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-wpxcz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body=
Nov 03 20:39:39.386799 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:39.385263    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-wpxcz" podUID="36477cba-9827-41ba-997d-d3d646ef4904" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused"
Nov 03 20:39:40.366797 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:40.366315    5570 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-78txb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body=
Nov 03 20:39:40.366797 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:40.366394    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-78txb" podUID="b65b5a79-c181-45ec-9cae-ebb5e5bbd865" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused"
Nov 03 20:39:40.382800 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:40.381335    5570 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-msmxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" start-of-body=
Nov 03 20:39:40.382800 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:40.381419    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-msmxq" podUID="9ebdd5ef-b321-4dfa-8c3e-1ad7c249a0ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused"
Nov 03 20:39:40.386798 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:40.385710    5570 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-wpxcz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body=
Nov 03 20:39:40.386798 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:40.385767    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-wpxcz" podUID="36477cba-9827-41ba-997d-d3d646ef4904" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused"
Nov 03 20:39:41.366800 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:41.364318    5570 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-78txb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body=
Nov 03 20:39:41.366800 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:41.364391    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-78txb" podUID="b65b5a79-c181-45ec-9cae-ebb5e5bbd865" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused"
Nov 03 20:39:41.379807 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:41.379509    5570 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-msmxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" start-of-body=
Nov 03 20:39:41.379807 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:41.379576    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-msmxq" podUID="9ebdd5ef-b321-4dfa-8c3e-1ad7c249a0ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused"
Nov 03 20:39:41.399811 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:41.399617    5570 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-wpxcz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body=
Nov 03 20:39:41.399811 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:41.399682    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-wpxcz" podUID="36477cba-9827-41ba-997d-d3d646ef4904" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused"
Nov 03 20:39:42.362893 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:42.362848    5570 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-78txb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body=
Nov 03 20:39:42.363104 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:42.362936    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-78txb" podUID="b65b5a79-c181-45ec-9cae-ebb5e5bbd865" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused"
Nov 03 20:39:42.377948 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:42.377898    5570 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-msmxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" start-of-body=
Nov 03 20:39:42.378170 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:42.377969    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-msmxq" podUID="9ebdd5ef-b321-4dfa-8c3e-1ad7c249a0ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused"
Nov 03 20:39:42.385232 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:42.385192    5570 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-wpxcz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body=
Nov 03 20:39:42.385455 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:42.385252    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-wpxcz" podUID="36477cba-9827-41ba-997d-d3d646ef4904" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused"
Nov 03 20:39:43.362949 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:43.362900    5570 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-78txb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body=
Nov 03 20:39:43.363399 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:43.362975    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-78txb" podUID="b65b5a79-c181-45ec-9cae-ebb5e5bbd865" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused"
Nov 03 20:39:43.378455 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:43.378408    5570 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-msmxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" start-of-body=
Nov 03 20:39:43.378615 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:43.378485    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-msmxq" podUID="9ebdd5ef-b321-4dfa-8c3e-1ad7c249a0ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused"
Nov 03 20:39:43.385555 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:43.385512    5570 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-wpxcz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body=
Nov 03 20:39:43.385731 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:43.385587    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-wpxcz" podUID="36477cba-9827-41ba-997d-d3d646ef4904" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused"
Nov 03 20:39:44.368986 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:44.368928    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-78txb"
Nov 03 20:39:44.369553 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:44.369528    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-78txb"
Nov 03 20:39:44.370635 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:44.370606    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-78txb"
Nov 03 20:39:44.427989 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:44.427912    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-66656cd6fd-qq5n6" podStartSLOduration=10.054860686 podStartE2EDuration="10.427874125s" totalImagesPullingTime="373.013439ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:39:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:39:36.863654949 +0000 UTC m=+2111.016316833" watchObservedRunningTime="2026-11-03 20:39:44.427874125 +0000 UTC m=+2118.580536011"
Nov 03 20:39:44.473948 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:44.473911    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-wpxcz"
Nov 03 20:39:44.474611 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:44.474582    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-wpxcz"
Nov 03 20:39:44.475859 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:44.475741    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-wpxcz"
Nov 03 20:39:44.573039 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:44.573004    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-msmxq"
Nov 03 20:39:44.573707 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:44.573682    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-msmxq"
Nov 03 20:39:44.575208 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:44.575187    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-msmxq"
Nov 03 20:39:48.832749 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:39:48.832718    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-66656cd6fd-qq5n6"
Nov 03 20:39:57.228276 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:39:57.228234    5570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8bwhz" containerName="extractor"
Nov 03 20:40:13.256566 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:40:13.256527    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g69f8"]
Nov 03 20:40:15.622799 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:40:15.621874    5570 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-msmxq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:40:15.622799 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:40:15.621941    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-msmxq" podUID="9ebdd5ef-b321-4dfa-8c3e-1ad7c249a0ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:40:18.773984 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:40:18.773951    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-tlwq8"]
Nov 03 20:40:18.779178 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:40:18.779152    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-tlwq8"
Nov 03 20:40:18.793499 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:40:18.793463    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-tlwq8"]
Nov 03 20:40:18.804408 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:40:18.804367    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:40:18.804617 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:40:18.804437    5570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 20:40:18.804617 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:40:18.804536    5570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 20:40:18.808125 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:40:18.808095    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l7s5w\""
Nov 03 20:40:18.839951 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:40:18.839914    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vf859\" (UniqueName: \"kubernetes.io/projected/07ea3e12-a2f0-496c-af2e-180c499058ef-kube-api-access-vf859\") pod \"maas-api-revoke-sub-c1e271a9-tlwq8\" (UID: \"07ea3e12-a2f0-496c-af2e-180c499058ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-tlwq8"
Nov 03 20:40:18.840182 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:40:18.840039    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07ea3e12-a2f0-496c-af2e-180c499058ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-tlwq8\" (UID: \"07ea3e12-a2f0-496c-af2e-180c499058ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-tlwq8"
Nov 03 20:40:18.840182 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:40:18.840063    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07ea3e12-a2f0-496c-af2e-180c499058ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-tlwq8\" (UID: \"07ea3e12-a2f0-496c-af2e-180c499058ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-tlwq8"
Nov 03 20:40:18.941577 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:40:18.941531    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07ea3e12-a2f0-496c-af2e-180c499058ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-tlwq8\" (UID: \"07ea3e12-a2f0-496c-af2e-180c499058ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-tlwq8"
Nov 03 20:40:18.941577 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:40:18.941578    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07ea3e12-a2f0-496c-af2e-180c499058ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-tlwq8\" (UID: \"07ea3e12-a2f0-496c-af2e-180c499058ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-tlwq8"
Nov 03 20:40:18.941918 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:40:18.941692    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vf859\" (UniqueName: \"kubernetes.io/projected/07ea3e12-a2f0-496c-af2e-180c499058ef-kube-api-access-vf859\") pod \"maas-api-revoke-sub-c1e271a9-tlwq8\" (UID: \"07ea3e12-a2f0-496c-af2e-180c499058ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-tlwq8"
Nov 03 20:40:18.942369 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:40:18.942345    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07ea3e12-a2f0-496c-af2e-180c499058ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-tlwq8\" (UID: \"07ea3e12-a2f0-496c-af2e-180c499058ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-tlwq8"
Nov 03 20:40:18.944085 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:40:18.944061    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07ea3e12-a2f0-496c-af2e-180c499058ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-tlwq8\" (UID: \"07ea3e12-a2f0-496c-af2e-180c499058ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-tlwq8"
Nov 03 20:40:18.962132 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:40:18.962093    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vf859\" (UniqueName: \"kubernetes.io/projected/07ea3e12-a2f0-496c-af2e-180c499058ef-kube-api-access-vf859\") pod \"maas-api-revoke-sub-c1e271a9-tlwq8\" (UID: \"07ea3e12-a2f0-496c-af2e-180c499058ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-tlwq8"
Nov 03 20:40:19.095591 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:40:19.095499    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-tlwq8"
Nov 03 20:40:19.391650 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:40:19.391619    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-tlwq8"]
Nov 03 20:40:20.110622 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:40:20.110584    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-tlwq8" event={"ID":"07ea3e12-a2f0-496c-af2e-180c499058ef","Type":"ContainerStarted","Data":"04cc4d4785163288501bd4e4d3c1be0be2cbe27dbd1711d005a320154383795c"}
Nov 03 20:40:20.111062 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:40:20.110641    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-tlwq8" event={"ID":"07ea3e12-a2f0-496c-af2e-180c499058ef","Type":"ContainerStarted","Data":"eefad4edfd65b6b0de96726f295bfd5f0d6fe5f5cff523e94c363c87fac4cb45"}
Nov 03 20:40:20.150496 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:40:20.150396    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-tlwq8" podStartSLOduration=2.150378219 podStartE2EDuration="2.150378219s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:40:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:40:20.150134433 +0000 UTC m=+2154.302796329" watchObservedRunningTime="2026-11-03 20:40:20.150378219 +0000 UTC m=+2154.303040107"
Nov 03 20:40:21.118062 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:40:21.118025    5570 generic.go:356] "Generic (PLEG): container finished" podID="07ea3e12-a2f0-496c-af2e-180c499058ef" containerID="04cc4d4785163288501bd4e4d3c1be0be2cbe27dbd1711d005a320154383795c" exitCode=0
Nov 03 20:40:21.118497 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:40:21.118073    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-tlwq8" event={"ID":"07ea3e12-a2f0-496c-af2e-180c499058ef","Type":"ContainerDied","Data":"04cc4d4785163288501bd4e4d3c1be0be2cbe27dbd1711d005a320154383795c"}
Nov 03 20:40:22.265385 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:40:22.265354    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-tlwq8"
Nov 03 20:40:22.377470 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:40:22.377377    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07ea3e12-a2f0-496c-af2e-180c499058ef-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07ea3e12-a2f0-496c-af2e-180c499058ef-maas-api-openshift-ca\") pod \"07ea3e12-a2f0-496c-af2e-180c499058ef\" (UID: \"07ea3e12-a2f0-496c-af2e-180c499058ef\") "
Nov 03 20:40:22.377470 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:40:22.377470    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07ea3e12-a2f0-496c-af2e-180c499058ef-kube-api-access-vf859\" (UniqueName: \"kubernetes.io/projected/07ea3e12-a2f0-496c-af2e-180c499058ef-kube-api-access-vf859\") pod \"07ea3e12-a2f0-496c-af2e-180c499058ef\" (UID: \"07ea3e12-a2f0-496c-af2e-180c499058ef\") "
Nov 03 20:40:22.377708 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:40:22.377515    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/07ea3e12-a2f0-496c-af2e-180c499058ef-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07ea3e12-a2f0-496c-af2e-180c499058ef-maas-api-xks-ca\") pod \"07ea3e12-a2f0-496c-af2e-180c499058ef\" (UID: \"07ea3e12-a2f0-496c-af2e-180c499058ef\") "
Nov 03 20:40:22.377813 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:40:22.377745    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07ea3e12-a2f0-496c-af2e-180c499058ef-maas-api-openshift-ca" pod "07ea3e12-a2f0-496c-af2e-180c499058ef" (UID: "07ea3e12-a2f0-496c-af2e-180c499058ef"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:40:22.377966 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:40:22.377947    5570 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07ea3e12-a2f0-496c-af2e-180c499058ef-maas-api-openshift-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:40:22.379710 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:40:22.379687    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07ea3e12-a2f0-496c-af2e-180c499058ef-maas-api-xks-ca" pod "07ea3e12-a2f0-496c-af2e-180c499058ef" (UID: "07ea3e12-a2f0-496c-af2e-180c499058ef"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:40:22.379808 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:40:22.379705    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07ea3e12-a2f0-496c-af2e-180c499058ef-kube-api-access-vf859" pod "07ea3e12-a2f0-496c-af2e-180c499058ef" (UID: "07ea3e12-a2f0-496c-af2e-180c499058ef"). InnerVolumeSpecName "kube-api-access-vf859". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:40:22.478482 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:40:22.478438    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vf859\" (UniqueName: \"kubernetes.io/projected/07ea3e12-a2f0-496c-af2e-180c499058ef-kube-api-access-vf859\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:40:22.478482 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:40:22.478474    5570 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07ea3e12-a2f0-496c-af2e-180c499058ef-maas-api-xks-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:40:23.130072 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:40:23.130038    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-tlwq8"
Nov 03 20:40:23.130277 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:40:23.130040    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-tlwq8" event={"ID":"07ea3e12-a2f0-496c-af2e-180c499058ef","Type":"ContainerDied","Data":"eefad4edfd65b6b0de96726f295bfd5f0d6fe5f5cff523e94c363c87fac4cb45"}
Nov 03 20:40:23.130277 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:40:23.130160    5570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eefad4edfd65b6b0de96726f295bfd5f0d6fe5f5cff523e94c363c87fac4cb45"
Nov 03 20:40:27.227490 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:40:27.227451    5570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8bwhz" containerName="extractor"
Nov 03 20:40:30.169208 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:40:30.169171    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g69f8"]
Nov 03 20:40:31.826603 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:40:31.826567    5570 scope.go:122] "RemoveContainer" containerID="bd865b0cd67a42e1a3cc7f30194b6c6b2aca22f3624bc77b1841477d22ba873b"
Nov 03 20:40:34.951151 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:40:34.951109    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g69f8"]
Nov 03 20:40:41.744236 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:40:41.744148    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-tggds"]
Nov 03 20:40:41.744822 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:40:41.744802    5570 cpu_manager.go:414] "containerMap: removing container" podUID="07ea3e12-a2f0-496c-af2e-180c499058ef" containerName="revoke-keys"
Nov 03 20:40:41.744929 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:40:41.744823    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07ea3e12-a2f0-496c-af2e-180c499058ef" containerName="revoke-keys"
Nov 03 20:40:41.744929 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:40:41.744920    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="07ea3e12-a2f0-496c-af2e-180c499058ef" containerName="revoke-keys"
Nov 03 20:40:41.750358 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:40:41.750329    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-tggds"
Nov 03 20:40:41.766558 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:40:41.766521    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-tggds"]
Nov 03 20:40:41.777448 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:40:41.777418    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:40:41.778000 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:40:41.777974    5570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 20:40:41.778099 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:40:41.778036    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l7s5w\""
Nov 03 20:40:41.779829 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:40:41.779801    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e708e575-b161-4cda-9e3b-1fc4b29994b6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-tggds\" (UID: \"e708e575-b161-4cda-9e3b-1fc4b29994b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-tggds"
Nov 03 20:40:41.779958 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:40:41.779894    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j47f4\" (UniqueName: \"kubernetes.io/projected/e708e575-b161-4cda-9e3b-1fc4b29994b6-kube-api-access-j47f4\") pod \"maas-api-revoke-sub-2e1f5b5f-tggds\" (UID: \"e708e575-b161-4cda-9e3b-1fc4b29994b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-tggds"
Nov 03 20:40:41.780082 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:40:41.780064    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e708e575-b161-4cda-9e3b-1fc4b29994b6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-tggds\" (UID: \"e708e575-b161-4cda-9e3b-1fc4b29994b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-tggds"
Nov 03 20:40:41.780358 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:40:41.780334    5570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 20:40:41.880647 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:40:41.880613    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e708e575-b161-4cda-9e3b-1fc4b29994b6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-tggds\" (UID: \"e708e575-b161-4cda-9e3b-1fc4b29994b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-tggds"
Nov 03 20:40:41.881155 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:40:41.880682    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j47f4\" (UniqueName: \"kubernetes.io/projected/e708e575-b161-4cda-9e3b-1fc4b29994b6-kube-api-access-j47f4\") pod \"maas-api-revoke-sub-2e1f5b5f-tggds\" (UID: \"e708e575-b161-4cda-9e3b-1fc4b29994b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-tggds"
Nov 03 20:40:41.881155 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:40:41.880800    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e708e575-b161-4cda-9e3b-1fc4b29994b6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-tggds\" (UID: \"e708e575-b161-4cda-9e3b-1fc4b29994b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-tggds"
Nov 03 20:40:41.881485 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:40:41.881465    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e708e575-b161-4cda-9e3b-1fc4b29994b6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-tggds\" (UID: \"e708e575-b161-4cda-9e3b-1fc4b29994b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-tggds"
Nov 03 20:40:41.883236 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:40:41.883211    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e708e575-b161-4cda-9e3b-1fc4b29994b6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-tggds\" (UID: \"e708e575-b161-4cda-9e3b-1fc4b29994b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-tggds"
Nov 03 20:40:41.901917 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:40:41.901884    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j47f4\" (UniqueName: \"kubernetes.io/projected/e708e575-b161-4cda-9e3b-1fc4b29994b6-kube-api-access-j47f4\") pod \"maas-api-revoke-sub-2e1f5b5f-tggds\" (UID: \"e708e575-b161-4cda-9e3b-1fc4b29994b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-tggds"
Nov 03 20:40:42.064219 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:40:42.064119    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-tggds"
Nov 03 20:40:42.361033 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:40:42.361004    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-tggds"]
Nov 03 20:40:42.363177 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:40:42.363144    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode708e575_b161_4cda_9e3b_1fc4b29994b6.slice/crio-e75fd42309a9e159fdce07c981b1918add64850ae3b260e72b6fbdefe14f920a WatchSource:0}: Error finding container e75fd42309a9e159fdce07c981b1918add64850ae3b260e72b6fbdefe14f920a: Status 404 returned error can't find the container with id e75fd42309a9e159fdce07c981b1918add64850ae3b260e72b6fbdefe14f920a
Nov 03 20:40:43.294106 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:40:43.294069    5570 generic.go:356] "Generic (PLEG): container finished" podID="e708e575-b161-4cda-9e3b-1fc4b29994b6" containerID="65d1b6094e465f0942980664226750d8ee6880970a51f6bd4708581ae855ce30" exitCode=0
Nov 03 20:40:43.294615 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:40:43.294157    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-tggds" event={"ID":"e708e575-b161-4cda-9e3b-1fc4b29994b6","Type":"ContainerDied","Data":"65d1b6094e465f0942980664226750d8ee6880970a51f6bd4708581ae855ce30"}
Nov 03 20:40:43.294615 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:40:43.294218    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-tggds" event={"ID":"e708e575-b161-4cda-9e3b-1fc4b29994b6","Type":"ContainerStarted","Data":"e75fd42309a9e159fdce07c981b1918add64850ae3b260e72b6fbdefe14f920a"}
Nov 03 20:40:44.436318 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:40:44.436291    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-tggds"
Nov 03 20:40:44.509902 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:40:44.509862    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e708e575-b161-4cda-9e3b-1fc4b29994b6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e708e575-b161-4cda-9e3b-1fc4b29994b6-maas-api-xks-ca\") pod \"e708e575-b161-4cda-9e3b-1fc4b29994b6\" (UID: \"e708e575-b161-4cda-9e3b-1fc4b29994b6\") "
Nov 03 20:40:44.510122 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:40:44.509944    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e708e575-b161-4cda-9e3b-1fc4b29994b6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e708e575-b161-4cda-9e3b-1fc4b29994b6-maas-api-openshift-ca\") pod \"e708e575-b161-4cda-9e3b-1fc4b29994b6\" (UID: \"e708e575-b161-4cda-9e3b-1fc4b29994b6\") "
Nov 03 20:40:44.510122 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:40:44.510078    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e708e575-b161-4cda-9e3b-1fc4b29994b6-kube-api-access-j47f4\" (UniqueName: \"kubernetes.io/projected/e708e575-b161-4cda-9e3b-1fc4b29994b6-kube-api-access-j47f4\") pod \"e708e575-b161-4cda-9e3b-1fc4b29994b6\" (UID: \"e708e575-b161-4cda-9e3b-1fc4b29994b6\") "
Nov 03 20:40:44.510390 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:40:44.510358    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e708e575-b161-4cda-9e3b-1fc4b29994b6-maas-api-openshift-ca" pod "e708e575-b161-4cda-9e3b-1fc4b29994b6" (UID: "e708e575-b161-4cda-9e3b-1fc4b29994b6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:40:44.512088 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:40:44.512057    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e708e575-b161-4cda-9e3b-1fc4b29994b6-maas-api-xks-ca" pod "e708e575-b161-4cda-9e3b-1fc4b29994b6" (UID: "e708e575-b161-4cda-9e3b-1fc4b29994b6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:40:44.512601 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:40:44.512578    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e708e575-b161-4cda-9e3b-1fc4b29994b6-kube-api-access-j47f4" pod "e708e575-b161-4cda-9e3b-1fc4b29994b6" (UID: "e708e575-b161-4cda-9e3b-1fc4b29994b6"). InnerVolumeSpecName "kube-api-access-j47f4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:40:44.611102 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:40:44.611001    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j47f4\" (UniqueName: \"kubernetes.io/projected/e708e575-b161-4cda-9e3b-1fc4b29994b6-kube-api-access-j47f4\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:40:44.611102 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:40:44.611042    5570 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e708e575-b161-4cda-9e3b-1fc4b29994b6-maas-api-xks-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:40:44.611102 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:40:44.611055    5570 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e708e575-b161-4cda-9e3b-1fc4b29994b6-maas-api-openshift-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:40:45.306816 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:40:45.306765    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-tggds" event={"ID":"e708e575-b161-4cda-9e3b-1fc4b29994b6","Type":"ContainerDied","Data":"e75fd42309a9e159fdce07c981b1918add64850ae3b260e72b6fbdefe14f920a"}
Nov 03 20:40:45.306816 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:40:45.306805    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-tggds"
Nov 03 20:40:45.306816 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:40:45.306826    5570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e75fd42309a9e159fdce07c981b1918add64850ae3b260e72b6fbdefe14f920a"
Nov 03 20:40:53.103953 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:40:53.103918    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-bm5ql"]
Nov 03 20:40:53.104407 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:40:53.104393    5570 cpu_manager.go:414] "containerMap: removing container" podUID="e708e575-b161-4cda-9e3b-1fc4b29994b6" containerName="revoke-keys"
Nov 03 20:40:53.104454 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:40:53.104409    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e708e575-b161-4cda-9e3b-1fc4b29994b6" containerName="revoke-keys"
Nov 03 20:40:53.104514 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:40:53.104503    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="e708e575-b161-4cda-9e3b-1fc4b29994b6" containerName="revoke-keys"
Nov 03 20:40:53.107716 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:40:53.107687    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-bm5ql"
Nov 03 20:40:53.125016 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:40:53.124971    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-bm5ql"]
Nov 03 20:40:53.134989 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:40:53.134957    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:40:53.134989 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:40:53.134970    5570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 20:40:53.135291 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:40:53.134996    5570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 20:40:53.136763 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:40:53.136742    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l7s5w\""
Nov 03 20:40:53.138762 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:40:53.138736    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g69f8"]
Nov 03 20:40:53.201354 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:40:53.201316    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrsgz\" (UniqueName: \"kubernetes.io/projected/f4e8bf4b-5796-4913-85f0-6997a7e0fcec-kube-api-access-vrsgz\") pod \"maas-api-revoke-sub-ec843018-bm5ql\" (UID: \"f4e8bf4b-5796-4913-85f0-6997a7e0fcec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-bm5ql"
Nov 03 20:40:53.201545 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:40:53.201438    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4e8bf4b-5796-4913-85f0-6997a7e0fcec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-bm5ql\" (UID: \"f4e8bf4b-5796-4913-85f0-6997a7e0fcec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-bm5ql"
Nov 03 20:40:53.201545 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:40:53.201522    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4e8bf4b-5796-4913-85f0-6997a7e0fcec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-bm5ql\" (UID: \"f4e8bf4b-5796-4913-85f0-6997a7e0fcec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-bm5ql"
Nov 03 20:40:53.302481 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:40:53.302436    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4e8bf4b-5796-4913-85f0-6997a7e0fcec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-bm5ql\" (UID: \"f4e8bf4b-5796-4913-85f0-6997a7e0fcec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-bm5ql"
Nov 03 20:40:53.302716 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:40:53.302536    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrsgz\" (UniqueName: \"kubernetes.io/projected/f4e8bf4b-5796-4913-85f0-6997a7e0fcec-kube-api-access-vrsgz\") pod \"maas-api-revoke-sub-ec843018-bm5ql\" (UID: \"f4e8bf4b-5796-4913-85f0-6997a7e0fcec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-bm5ql"
Nov 03 20:40:53.302716 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:40:53.302584    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4e8bf4b-5796-4913-85f0-6997a7e0fcec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-bm5ql\" (UID: \"f4e8bf4b-5796-4913-85f0-6997a7e0fcec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-bm5ql"
Nov 03 20:40:53.303371 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:40:53.303339    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4e8bf4b-5796-4913-85f0-6997a7e0fcec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-bm5ql\" (UID: \"f4e8bf4b-5796-4913-85f0-6997a7e0fcec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-bm5ql"
Nov 03 20:40:53.305655 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:40:53.305630    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4e8bf4b-5796-4913-85f0-6997a7e0fcec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-bm5ql\" (UID: \"f4e8bf4b-5796-4913-85f0-6997a7e0fcec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-bm5ql"
Nov 03 20:40:53.322437 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:40:53.322405    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrsgz\" (UniqueName: \"kubernetes.io/projected/f4e8bf4b-5796-4913-85f0-6997a7e0fcec-kube-api-access-vrsgz\") pod \"maas-api-revoke-sub-ec843018-bm5ql\" (UID: \"f4e8bf4b-5796-4913-85f0-6997a7e0fcec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-bm5ql"
Nov 03 20:40:53.421819 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:40:53.421694    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-bm5ql"
Nov 03 20:40:53.796087 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:40:53.796056    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-bm5ql"]
Nov 03 20:40:53.798007 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:40:53.797976    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4e8bf4b_5796_4913_85f0_6997a7e0fcec.slice/crio-6bb9d0df1c90a85b479d7f5f4f9409ec046776efde009361a008d1408b8bd35e WatchSource:0}: Error finding container 6bb9d0df1c90a85b479d7f5f4f9409ec046776efde009361a008d1408b8bd35e: Status 404 returned error can't find the container with id 6bb9d0df1c90a85b479d7f5f4f9409ec046776efde009361a008d1408b8bd35e
Nov 03 20:40:54.357416 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:40:54.357324    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-bm5ql" event={"ID":"f4e8bf4b-5796-4913-85f0-6997a7e0fcec","Type":"ContainerStarted","Data":"26303f6ea19af99d93b739eb6517a0b50fe499d28e10528e5ee1d8af34b77645"}
Nov 03 20:40:54.357416 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:40:54.357385    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-bm5ql" event={"ID":"f4e8bf4b-5796-4913-85f0-6997a7e0fcec","Type":"ContainerStarted","Data":"6bb9d0df1c90a85b479d7f5f4f9409ec046776efde009361a008d1408b8bd35e"}
Nov 03 20:40:54.397872 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:40:54.397816    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-bm5ql" podStartSLOduration=1.397796445 podStartE2EDuration="1.397796445s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:40:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:40:54.39727388 +0000 UTC m=+2188.549935767" watchObservedRunningTime="2026-11-03 20:40:54.397796445 +0000 UTC m=+2188.550458331"
Nov 03 20:40:55.364694 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:40:55.364654    5570 generic.go:356] "Generic (PLEG): container finished" podID="f4e8bf4b-5796-4913-85f0-6997a7e0fcec" containerID="26303f6ea19af99d93b739eb6517a0b50fe499d28e10528e5ee1d8af34b77645" exitCode=0
Nov 03 20:40:55.365164 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:40:55.364737    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-bm5ql" event={"ID":"f4e8bf4b-5796-4913-85f0-6997a7e0fcec","Type":"ContainerDied","Data":"26303f6ea19af99d93b739eb6517a0b50fe499d28e10528e5ee1d8af34b77645"}
Nov 03 20:40:56.514225 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:40:56.514194    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-bm5ql"
Nov 03 20:40:56.636081 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:40:56.635990    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f4e8bf4b-5796-4913-85f0-6997a7e0fcec-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4e8bf4b-5796-4913-85f0-6997a7e0fcec-maas-api-xks-ca\") pod \"f4e8bf4b-5796-4913-85f0-6997a7e0fcec\" (UID: \"f4e8bf4b-5796-4913-85f0-6997a7e0fcec\") "
Nov 03 20:40:56.636256 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:40:56.636103    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4e8bf4b-5796-4913-85f0-6997a7e0fcec-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4e8bf4b-5796-4913-85f0-6997a7e0fcec-maas-api-openshift-ca\") pod \"f4e8bf4b-5796-4913-85f0-6997a7e0fcec\" (UID: \"f4e8bf4b-5796-4913-85f0-6997a7e0fcec\") "
Nov 03 20:40:56.636256 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:40:56.636168    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4e8bf4b-5796-4913-85f0-6997a7e0fcec-kube-api-access-vrsgz\" (UniqueName: \"kubernetes.io/projected/f4e8bf4b-5796-4913-85f0-6997a7e0fcec-kube-api-access-vrsgz\") pod \"f4e8bf4b-5796-4913-85f0-6997a7e0fcec\" (UID: \"f4e8bf4b-5796-4913-85f0-6997a7e0fcec\") "
Nov 03 20:40:56.636532 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:40:56.636501    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4e8bf4b-5796-4913-85f0-6997a7e0fcec-maas-api-openshift-ca" pod "f4e8bf4b-5796-4913-85f0-6997a7e0fcec" (UID: "f4e8bf4b-5796-4913-85f0-6997a7e0fcec"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:40:56.636656 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:40:56.636585    5570 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4e8bf4b-5796-4913-85f0-6997a7e0fcec-maas-api-openshift-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:40:56.638268 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:40:56.638232    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4e8bf4b-5796-4913-85f0-6997a7e0fcec-kube-api-access-vrsgz" pod "f4e8bf4b-5796-4913-85f0-6997a7e0fcec" (UID: "f4e8bf4b-5796-4913-85f0-6997a7e0fcec"). InnerVolumeSpecName "kube-api-access-vrsgz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:40:56.638366 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:40:56.638236    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4e8bf4b-5796-4913-85f0-6997a7e0fcec-maas-api-xks-ca" pod "f4e8bf4b-5796-4913-85f0-6997a7e0fcec" (UID: "f4e8bf4b-5796-4913-85f0-6997a7e0fcec"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:40:56.737961 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:40:56.737919    5570 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4e8bf4b-5796-4913-85f0-6997a7e0fcec-maas-api-xks-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:40:56.737961 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:40:56.737957    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrsgz\" (UniqueName: \"kubernetes.io/projected/f4e8bf4b-5796-4913-85f0-6997a7e0fcec-kube-api-access-vrsgz\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:40:57.227731 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:40:57.227694    5570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8bwhz" containerName="extractor"
Nov 03 20:40:57.377520 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:40:57.377479    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-bm5ql" event={"ID":"f4e8bf4b-5796-4913-85f0-6997a7e0fcec","Type":"ContainerDied","Data":"6bb9d0df1c90a85b479d7f5f4f9409ec046776efde009361a008d1408b8bd35e"}
Nov 03 20:40:57.377703 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:40:57.377532    5570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6bb9d0df1c90a85b479d7f5f4f9409ec046776efde009361a008d1408b8bd35e"
Nov 03 20:40:57.377703 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:40:57.377498    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-bm5ql"
Nov 03 20:41:08.818651 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:41:08.818618    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g69f8"]
Nov 03 20:41:20.773235 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:41:20.773194    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-8hxlb"]
Nov 03 20:41:20.773746 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:41:20.773625    5570 cpu_manager.go:414] "containerMap: removing container" podUID="f4e8bf4b-5796-4913-85f0-6997a7e0fcec" containerName="revoke-keys"
Nov 03 20:41:20.773746 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:41:20.773636    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4e8bf4b-5796-4913-85f0-6997a7e0fcec" containerName="revoke-keys"
Nov 03 20:41:20.773746 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:41:20.773723    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4e8bf4b-5796-4913-85f0-6997a7e0fcec" containerName="revoke-keys"
Nov 03 20:41:20.776948 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:41:20.776929    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-8hxlb"
Nov 03 20:41:20.794026 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:41:20.793997    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-8hxlb"]
Nov 03 20:41:20.805581 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:41:20.805540    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:41:20.805581 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:41:20.805564    5570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 20:41:20.805822 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:41:20.805592    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l7s5w\""
Nov 03 20:41:20.807674 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:41:20.807655    5570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 20:41:20.827539 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:41:20.827502    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7skxx\" (UniqueName: \"kubernetes.io/projected/d8b8db97-5583-40e0-90b5-8076a3e052d0-kube-api-access-7skxx\") pod \"maas-api-revoke-sub-046e1343-8hxlb\" (UID: \"d8b8db97-5583-40e0-90b5-8076a3e052d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-8hxlb"
Nov 03 20:41:20.827732 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:41:20.827547    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8b8db97-5583-40e0-90b5-8076a3e052d0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-8hxlb\" (UID: \"d8b8db97-5583-40e0-90b5-8076a3e052d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-8hxlb"
Nov 03 20:41:20.827732 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:41:20.827687    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8b8db97-5583-40e0-90b5-8076a3e052d0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-8hxlb\" (UID: \"d8b8db97-5583-40e0-90b5-8076a3e052d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-8hxlb"
Nov 03 20:41:20.928458 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:41:20.928415    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7skxx\" (UniqueName: \"kubernetes.io/projected/d8b8db97-5583-40e0-90b5-8076a3e052d0-kube-api-access-7skxx\") pod \"maas-api-revoke-sub-046e1343-8hxlb\" (UID: \"d8b8db97-5583-40e0-90b5-8076a3e052d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-8hxlb"
Nov 03 20:41:20.928665 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:41:20.928468    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8b8db97-5583-40e0-90b5-8076a3e052d0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-8hxlb\" (UID: \"d8b8db97-5583-40e0-90b5-8076a3e052d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-8hxlb"
Nov 03 20:41:20.928665 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:41:20.928550    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8b8db97-5583-40e0-90b5-8076a3e052d0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-8hxlb\" (UID: \"d8b8db97-5583-40e0-90b5-8076a3e052d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-8hxlb"
Nov 03 20:41:20.929348 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:41:20.929318    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8b8db97-5583-40e0-90b5-8076a3e052d0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-8hxlb\" (UID: \"d8b8db97-5583-40e0-90b5-8076a3e052d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-8hxlb"
Nov 03 20:41:20.930990 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:41:20.930971    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8b8db97-5583-40e0-90b5-8076a3e052d0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-8hxlb\" (UID: \"d8b8db97-5583-40e0-90b5-8076a3e052d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-8hxlb"
Nov 03 20:41:20.948910 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:41:20.948871    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7skxx\" (UniqueName: \"kubernetes.io/projected/d8b8db97-5583-40e0-90b5-8076a3e052d0-kube-api-access-7skxx\") pod \"maas-api-revoke-sub-046e1343-8hxlb\" (UID: \"d8b8db97-5583-40e0-90b5-8076a3e052d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-8hxlb"
Nov 03 20:41:21.090544 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:41:21.090454    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-8hxlb"
Nov 03 20:41:21.387039 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:41:21.387005    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-8hxlb"]
Nov 03 20:41:21.388384 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:41:21.388355    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8b8db97_5583_40e0_90b5_8076a3e052d0.slice/crio-f5b9b0d7722f139141934b780d9c81d4c289209869ae7beb766d36416c7fe9a3 WatchSource:0}: Error finding container f5b9b0d7722f139141934b780d9c81d4c289209869ae7beb766d36416c7fe9a3: Status 404 returned error can't find the container with id f5b9b0d7722f139141934b780d9c81d4c289209869ae7beb766d36416c7fe9a3
Nov 03 20:41:21.543023 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:41:21.542991    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-8hxlb" event={"ID":"d8b8db97-5583-40e0-90b5-8076a3e052d0","Type":"ContainerStarted","Data":"f5b9b0d7722f139141934b780d9c81d4c289209869ae7beb766d36416c7fe9a3"}
Nov 03 20:41:22.550800 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:41:22.550671    5570 generic.go:356] "Generic (PLEG): container finished" podID="d8b8db97-5583-40e0-90b5-8076a3e052d0" containerID="874afa765839756afeb7c6a531db3ad2dbebce38b1c07f17d716078b7cf0322d" exitCode=0
Nov 03 20:41:22.550800 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:41:22.550756    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-8hxlb" event={"ID":"d8b8db97-5583-40e0-90b5-8076a3e052d0","Type":"ContainerDied","Data":"874afa765839756afeb7c6a531db3ad2dbebce38b1c07f17d716078b7cf0322d"}
Nov 03 20:41:23.702735 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:41:23.702708    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-8hxlb"
Nov 03 20:41:23.759617 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:41:23.759586    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8b8db97-5583-40e0-90b5-8076a3e052d0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8b8db97-5583-40e0-90b5-8076a3e052d0-maas-api-openshift-ca\") pod \"d8b8db97-5583-40e0-90b5-8076a3e052d0\" (UID: \"d8b8db97-5583-40e0-90b5-8076a3e052d0\") "
Nov 03 20:41:23.759836 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:41:23.759705    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d8b8db97-5583-40e0-90b5-8076a3e052d0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8b8db97-5583-40e0-90b5-8076a3e052d0-maas-api-xks-ca\") pod \"d8b8db97-5583-40e0-90b5-8076a3e052d0\" (UID: \"d8b8db97-5583-40e0-90b5-8076a3e052d0\") "
Nov 03 20:41:23.759836 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:41:23.759758    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8b8db97-5583-40e0-90b5-8076a3e052d0-kube-api-access-7skxx\" (UniqueName: \"kubernetes.io/projected/d8b8db97-5583-40e0-90b5-8076a3e052d0-kube-api-access-7skxx\") pod \"d8b8db97-5583-40e0-90b5-8076a3e052d0\" (UID: \"d8b8db97-5583-40e0-90b5-8076a3e052d0\") "
Nov 03 20:41:23.760131 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:41:23.760103    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8b8db97-5583-40e0-90b5-8076a3e052d0-maas-api-openshift-ca" pod "d8b8db97-5583-40e0-90b5-8076a3e052d0" (UID: "d8b8db97-5583-40e0-90b5-8076a3e052d0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:41:23.760242 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:41:23.760222    5570 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8b8db97-5583-40e0-90b5-8076a3e052d0-maas-api-openshift-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:41:23.762059 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:41:23.762033    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8b8db97-5583-40e0-90b5-8076a3e052d0-maas-api-xks-ca" pod "d8b8db97-5583-40e0-90b5-8076a3e052d0" (UID: "d8b8db97-5583-40e0-90b5-8076a3e052d0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:41:23.762059 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:41:23.762048    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8b8db97-5583-40e0-90b5-8076a3e052d0-kube-api-access-7skxx" pod "d8b8db97-5583-40e0-90b5-8076a3e052d0" (UID: "d8b8db97-5583-40e0-90b5-8076a3e052d0"). InnerVolumeSpecName "kube-api-access-7skxx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:41:23.861605 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:41:23.861512    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7skxx\" (UniqueName: \"kubernetes.io/projected/d8b8db97-5583-40e0-90b5-8076a3e052d0-kube-api-access-7skxx\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:41:23.861605 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:41:23.861547    5570 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8b8db97-5583-40e0-90b5-8076a3e052d0-maas-api-xks-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:41:24.563092 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:41:24.563052    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-8hxlb"
Nov 03 20:41:24.563290 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:41:24.563085    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-8hxlb" event={"ID":"d8b8db97-5583-40e0-90b5-8076a3e052d0","Type":"ContainerDied","Data":"f5b9b0d7722f139141934b780d9c81d4c289209869ae7beb766d36416c7fe9a3"}
Nov 03 20:41:24.563290 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:41:24.563162    5570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f5b9b0d7722f139141934b780d9c81d4c289209869ae7beb766d36416c7fe9a3"
Nov 03 20:41:27.227711 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:41:27.227670    5570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8bwhz" containerName="extractor"
Nov 03 20:41:31.110533 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:41:31.110490    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-br276"]
Nov 03 20:41:31.111010 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:41:31.110992    5570 cpu_manager.go:414] "containerMap: removing container" podUID="d8b8db97-5583-40e0-90b5-8076a3e052d0" containerName="revoke-keys"
Nov 03 20:41:31.111073 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:41:31.111010    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8b8db97-5583-40e0-90b5-8076a3e052d0" containerName="revoke-keys"
Nov 03 20:41:31.111106 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:41:31.111099    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8b8db97-5583-40e0-90b5-8076a3e052d0" containerName="revoke-keys"
Nov 03 20:41:31.114578 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:41:31.114560    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-br276"
Nov 03 20:41:31.130711 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:41:31.130679    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-br276"]
Nov 03 20:41:31.141246 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:41:31.141211    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:41:31.141422 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:41:31.141288    5570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 20:41:31.141422 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:41:31.141298    5570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 20:41:31.143303 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:41:31.143287    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l7s5w\""
Nov 03 20:41:31.236258 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:41:31.236219    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lx6j\" (UniqueName: \"kubernetes.io/projected/2375597e-e94e-433b-9b1b-d48abe92e870-kube-api-access-9lx6j\") pod \"maas-api-revoke-sub-d3fba056-br276\" (UID: \"2375597e-e94e-433b-9b1b-d48abe92e870\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-br276"
Nov 03 20:41:31.236516 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:41:31.236360    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2375597e-e94e-433b-9b1b-d48abe92e870-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-br276\" (UID: \"2375597e-e94e-433b-9b1b-d48abe92e870\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-br276"
Nov 03 20:41:31.236516 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:41:31.236453    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2375597e-e94e-433b-9b1b-d48abe92e870-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-br276\" (UID: \"2375597e-e94e-433b-9b1b-d48abe92e870\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-br276"
Nov 03 20:41:31.337510 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:41:31.337476    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2375597e-e94e-433b-9b1b-d48abe92e870-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-br276\" (UID: \"2375597e-e94e-433b-9b1b-d48abe92e870\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-br276"
Nov 03 20:41:31.337732 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:41:31.337539    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2375597e-e94e-433b-9b1b-d48abe92e870-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-br276\" (UID: \"2375597e-e94e-433b-9b1b-d48abe92e870\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-br276"
Nov 03 20:41:31.337732 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:41:31.337645    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lx6j\" (UniqueName: \"kubernetes.io/projected/2375597e-e94e-433b-9b1b-d48abe92e870-kube-api-access-9lx6j\") pod \"maas-api-revoke-sub-d3fba056-br276\" (UID: \"2375597e-e94e-433b-9b1b-d48abe92e870\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-br276"
Nov 03 20:41:31.338158 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:41:31.338135    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2375597e-e94e-433b-9b1b-d48abe92e870-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-br276\" (UID: \"2375597e-e94e-433b-9b1b-d48abe92e870\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-br276"
Nov 03 20:41:31.340003 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:41:31.339974    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2375597e-e94e-433b-9b1b-d48abe92e870-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-br276\" (UID: \"2375597e-e94e-433b-9b1b-d48abe92e870\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-br276"
Nov 03 20:41:31.358036 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:41:31.358009    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lx6j\" (UniqueName: \"kubernetes.io/projected/2375597e-e94e-433b-9b1b-d48abe92e870-kube-api-access-9lx6j\") pod \"maas-api-revoke-sub-d3fba056-br276\" (UID: \"2375597e-e94e-433b-9b1b-d48abe92e870\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-br276"
Nov 03 20:41:31.427687 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:41:31.427604    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-br276"
Nov 03 20:41:31.722633 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:41:31.722600    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-br276"]
Nov 03 20:41:31.723858 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:41:31.723765    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2375597e_e94e_433b_9b1b_d48abe92e870.slice/crio-bff10a76311eb956e13a5811b5063565f25d1b1e90ac0666aa2b9fcfb926903d WatchSource:0}: Error finding container bff10a76311eb956e13a5811b5063565f25d1b1e90ac0666aa2b9fcfb926903d: Status 404 returned error can't find the container with id bff10a76311eb956e13a5811b5063565f25d1b1e90ac0666aa2b9fcfb926903d
Nov 03 20:41:32.054245 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:41:32.054215    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g69f8"]
Nov 03 20:41:32.609862 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:41:32.609825    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-br276" event={"ID":"2375597e-e94e-433b-9b1b-d48abe92e870","Type":"ContainerStarted","Data":"2d24bb8477d0c4f0a6985b8b19df168880d0b7349042a43a2599e441ac23eb01"}
Nov 03 20:41:32.610488 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:41:32.610449    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-br276" event={"ID":"2375597e-e94e-433b-9b1b-d48abe92e870","Type":"ContainerStarted","Data":"bff10a76311eb956e13a5811b5063565f25d1b1e90ac0666aa2b9fcfb926903d"}
Nov 03 20:41:32.653285 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:41:32.653205    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-br276" podStartSLOduration=1.653183108 podStartE2EDuration="1.653183108s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:41:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:41:32.649260333 +0000 UTC m=+2226.801922233" watchObservedRunningTime="2026-11-03 20:41:32.653183108 +0000 UTC m=+2226.805844995"
Nov 03 20:41:33.636218 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:41:33.636162    5570 generic.go:356] "Generic (PLEG): container finished" podID="2375597e-e94e-433b-9b1b-d48abe92e870" containerID="2d24bb8477d0c4f0a6985b8b19df168880d0b7349042a43a2599e441ac23eb01" exitCode=0
Nov 03 20:41:33.637670 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:41:33.637641    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-br276" event={"ID":"2375597e-e94e-433b-9b1b-d48abe92e870","Type":"ContainerDied","Data":"2d24bb8477d0c4f0a6985b8b19df168880d0b7349042a43a2599e441ac23eb01"}
Nov 03 20:41:35.191917 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:41:35.191884    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-br276"
Nov 03 20:41:35.346205 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:41:35.346110    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2375597e-e94e-433b-9b1b-d48abe92e870-kube-api-access-9lx6j\" (UniqueName: \"kubernetes.io/projected/2375597e-e94e-433b-9b1b-d48abe92e870-kube-api-access-9lx6j\") pod \"2375597e-e94e-433b-9b1b-d48abe92e870\" (UID: \"2375597e-e94e-433b-9b1b-d48abe92e870\") "
Nov 03 20:41:35.346403 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:41:35.346232    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2375597e-e94e-433b-9b1b-d48abe92e870-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2375597e-e94e-433b-9b1b-d48abe92e870-maas-api-xks-ca\") pod \"2375597e-e94e-433b-9b1b-d48abe92e870\" (UID: \"2375597e-e94e-433b-9b1b-d48abe92e870\") "
Nov 03 20:41:35.346403 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:41:35.346274    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2375597e-e94e-433b-9b1b-d48abe92e870-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2375597e-e94e-433b-9b1b-d48abe92e870-maas-api-openshift-ca\") pod \"2375597e-e94e-433b-9b1b-d48abe92e870\" (UID: \"2375597e-e94e-433b-9b1b-d48abe92e870\") "
Nov 03 20:41:35.346757 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:41:35.346729    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2375597e-e94e-433b-9b1b-d48abe92e870-maas-api-openshift-ca" pod "2375597e-e94e-433b-9b1b-d48abe92e870" (UID: "2375597e-e94e-433b-9b1b-d48abe92e870"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:41:35.348449 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:41:35.348422    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2375597e-e94e-433b-9b1b-d48abe92e870-kube-api-access-9lx6j" pod "2375597e-e94e-433b-9b1b-d48abe92e870" (UID: "2375597e-e94e-433b-9b1b-d48abe92e870"). InnerVolumeSpecName "kube-api-access-9lx6j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:41:35.348560 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:41:35.348503    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2375597e-e94e-433b-9b1b-d48abe92e870-maas-api-xks-ca" pod "2375597e-e94e-433b-9b1b-d48abe92e870" (UID: "2375597e-e94e-433b-9b1b-d48abe92e870"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:41:35.447889 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:41:35.447818    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9lx6j\" (UniqueName: \"kubernetes.io/projected/2375597e-e94e-433b-9b1b-d48abe92e870-kube-api-access-9lx6j\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:41:35.447889 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:41:35.447885    5570 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2375597e-e94e-433b-9b1b-d48abe92e870-maas-api-xks-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:41:35.447889 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:41:35.447898    5570 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2375597e-e94e-433b-9b1b-d48abe92e870-maas-api-openshift-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:41:35.674492 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:41:35.674403    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-br276"
Nov 03 20:41:35.674492 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:41:35.674430    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-br276" event={"ID":"2375597e-e94e-433b-9b1b-d48abe92e870","Type":"ContainerDied","Data":"bff10a76311eb956e13a5811b5063565f25d1b1e90ac0666aa2b9fcfb926903d"}
Nov 03 20:41:35.674492 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:41:35.674483    5570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bff10a76311eb956e13a5811b5063565f25d1b1e90ac0666aa2b9fcfb926903d"
Nov 03 20:41:40.026617 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:41:40.026578    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e33e59be-fmxqt"]
Nov 03 20:41:40.040059 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:41:40.040024    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e33e59be-fmxqt"]
Nov 03 20:41:41.963019 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:41:41.962978    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="426afdc6-cfdd-47c9-9ba9-6da128c31913" path="/var/lib/kubelet/pods/426afdc6-cfdd-47c9-9ba9-6da128c31913/volumes"
Nov 03 20:41:42.373850 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:41:42.373735    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g69f8"]
Nov 03 20:41:46.941589 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:41:46.941550    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g69f8"]
Nov 03 20:41:52.507308 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:41:52.507270    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-kwvfl"]
Nov 03 20:41:52.507864 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:41:52.507696    5570 cpu_manager.go:414] "containerMap: removing container" podUID="2375597e-e94e-433b-9b1b-d48abe92e870" containerName="revoke-keys"
Nov 03 20:41:52.507864 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:41:52.507708    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2375597e-e94e-433b-9b1b-d48abe92e870" containerName="revoke-keys"
Nov 03 20:41:52.507864 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:41:52.507788    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="2375597e-e94e-433b-9b1b-d48abe92e870" containerName="revoke-keys"
Nov 03 20:41:52.510996 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:41:52.510978    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-kwvfl"
Nov 03 20:41:52.526554 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:41:52.526525    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-kwvfl"]
Nov 03 20:41:52.537212 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:41:52.537178    5570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 20:41:52.537394 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:41:52.537224    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:41:52.539821 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:41:52.539791    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l7s5w\""
Nov 03 20:41:52.540000 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:41:52.539824    5570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 20:41:52.595461 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:41:52.595422    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1f35880-1a34-4214-a893-b1e1ba5f27c4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-kwvfl\" (UID: \"d1f35880-1a34-4214-a893-b1e1ba5f27c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-kwvfl"
Nov 03 20:41:52.595461 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:41:52.595468    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1f35880-1a34-4214-a893-b1e1ba5f27c4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-kwvfl\" (UID: \"d1f35880-1a34-4214-a893-b1e1ba5f27c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-kwvfl"
Nov 03 20:41:52.595706 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:41:52.595488    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rjlt\" (UniqueName: \"kubernetes.io/projected/d1f35880-1a34-4214-a893-b1e1ba5f27c4-kube-api-access-7rjlt\") pod \"maas-api-revoke-sub-ae594979-kwvfl\" (UID: \"d1f35880-1a34-4214-a893-b1e1ba5f27c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-kwvfl"
Nov 03 20:41:52.696628 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:41:52.696595    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1f35880-1a34-4214-a893-b1e1ba5f27c4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-kwvfl\" (UID: \"d1f35880-1a34-4214-a893-b1e1ba5f27c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-kwvfl"
Nov 03 20:41:52.696838 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:41:52.696640    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1f35880-1a34-4214-a893-b1e1ba5f27c4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-kwvfl\" (UID: \"d1f35880-1a34-4214-a893-b1e1ba5f27c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-kwvfl"
Nov 03 20:41:52.696838 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:41:52.696670    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rjlt\" (UniqueName: \"kubernetes.io/projected/d1f35880-1a34-4214-a893-b1e1ba5f27c4-kube-api-access-7rjlt\") pod \"maas-api-revoke-sub-ae594979-kwvfl\" (UID: \"d1f35880-1a34-4214-a893-b1e1ba5f27c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-kwvfl"
Nov 03 20:41:52.697423 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:41:52.697402    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1f35880-1a34-4214-a893-b1e1ba5f27c4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-kwvfl\" (UID: \"d1f35880-1a34-4214-a893-b1e1ba5f27c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-kwvfl"
Nov 03 20:41:52.699082 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:41:52.699066    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1f35880-1a34-4214-a893-b1e1ba5f27c4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-kwvfl\" (UID: \"d1f35880-1a34-4214-a893-b1e1ba5f27c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-kwvfl"
Nov 03 20:41:52.716786 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:41:52.716755    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rjlt\" (UniqueName: \"kubernetes.io/projected/d1f35880-1a34-4214-a893-b1e1ba5f27c4-kube-api-access-7rjlt\") pod \"maas-api-revoke-sub-ae594979-kwvfl\" (UID: \"d1f35880-1a34-4214-a893-b1e1ba5f27c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-kwvfl"
Nov 03 20:41:52.824354 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:41:52.824267    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-kwvfl"
Nov 03 20:41:53.120232 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:41:53.120203    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-kwvfl"]
Nov 03 20:41:53.121685 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:41:53.121655    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1f35880_1a34_4214_a893_b1e1ba5f27c4.slice/crio-1bc2e68ed5f653f857a4469b150930866ef4e9b46aa3816fb8a081bfca67a836 WatchSource:0}: Error finding container 1bc2e68ed5f653f857a4469b150930866ef4e9b46aa3816fb8a081bfca67a836: Status 404 returned error can't find the container with id 1bc2e68ed5f653f857a4469b150930866ef4e9b46aa3816fb8a081bfca67a836
Nov 03 20:41:53.830795 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:41:53.830688    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-kwvfl" event={"ID":"d1f35880-1a34-4214-a893-b1e1ba5f27c4","Type":"ContainerStarted","Data":"6025faab011372ec2db92604e8c2aaa50d6d3c8653c916742a761bd62d7a5c5d"}
Nov 03 20:41:53.830795 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:41:53.830745    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-kwvfl" event={"ID":"d1f35880-1a34-4214-a893-b1e1ba5f27c4","Type":"ContainerStarted","Data":"1bc2e68ed5f653f857a4469b150930866ef4e9b46aa3816fb8a081bfca67a836"}
Nov 03 20:41:53.871391 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:41:53.871334    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-kwvfl" podStartSLOduration=1.871315544 podStartE2EDuration="1.871315544s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:41:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:41:53.870266936 +0000 UTC m=+2248.022928834" watchObservedRunningTime="2026-11-03 20:41:53.871315544 +0000 UTC m=+2248.023977430"
Nov 03 20:41:54.846044 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:41:54.846000    5570 generic.go:356] "Generic (PLEG): container finished" podID="d1f35880-1a34-4214-a893-b1e1ba5f27c4" containerID="6025faab011372ec2db92604e8c2aaa50d6d3c8653c916742a761bd62d7a5c5d" exitCode=0
Nov 03 20:41:54.846526 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:41:54.846101    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-kwvfl" event={"ID":"d1f35880-1a34-4214-a893-b1e1ba5f27c4","Type":"ContainerDied","Data":"6025faab011372ec2db92604e8c2aaa50d6d3c8653c916742a761bd62d7a5c5d"}
Nov 03 20:41:55.990440 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:41:55.990410    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-kwvfl"
Nov 03 20:41:56.130827 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:41:56.130709    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d1f35880-1a34-4214-a893-b1e1ba5f27c4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1f35880-1a34-4214-a893-b1e1ba5f27c4-maas-api-xks-ca\") pod \"d1f35880-1a34-4214-a893-b1e1ba5f27c4\" (UID: \"d1f35880-1a34-4214-a893-b1e1ba5f27c4\") "
Nov 03 20:41:56.130827 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:41:56.130808    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1f35880-1a34-4214-a893-b1e1ba5f27c4-kube-api-access-7rjlt\" (UniqueName: \"kubernetes.io/projected/d1f35880-1a34-4214-a893-b1e1ba5f27c4-kube-api-access-7rjlt\") pod \"d1f35880-1a34-4214-a893-b1e1ba5f27c4\" (UID: \"d1f35880-1a34-4214-a893-b1e1ba5f27c4\") "
Nov 03 20:41:56.131061 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:41:56.130907    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1f35880-1a34-4214-a893-b1e1ba5f27c4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1f35880-1a34-4214-a893-b1e1ba5f27c4-maas-api-openshift-ca\") pod \"d1f35880-1a34-4214-a893-b1e1ba5f27c4\" (UID: \"d1f35880-1a34-4214-a893-b1e1ba5f27c4\") "
Nov 03 20:41:56.131287 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:41:56.131265    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1f35880-1a34-4214-a893-b1e1ba5f27c4-maas-api-openshift-ca" pod "d1f35880-1a34-4214-a893-b1e1ba5f27c4" (UID: "d1f35880-1a34-4214-a893-b1e1ba5f27c4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:41:56.133162 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:41:56.133138    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1f35880-1a34-4214-a893-b1e1ba5f27c4-maas-api-xks-ca" pod "d1f35880-1a34-4214-a893-b1e1ba5f27c4" (UID: "d1f35880-1a34-4214-a893-b1e1ba5f27c4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:41:56.133255 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:41:56.133210    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1f35880-1a34-4214-a893-b1e1ba5f27c4-kube-api-access-7rjlt" pod "d1f35880-1a34-4214-a893-b1e1ba5f27c4" (UID: "d1f35880-1a34-4214-a893-b1e1ba5f27c4"). InnerVolumeSpecName "kube-api-access-7rjlt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:41:56.232149 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:41:56.232106    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rjlt\" (UniqueName: \"kubernetes.io/projected/d1f35880-1a34-4214-a893-b1e1ba5f27c4-kube-api-access-7rjlt\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:41:56.232149 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:41:56.232144    5570 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1f35880-1a34-4214-a893-b1e1ba5f27c4-maas-api-openshift-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:41:56.232387 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:41:56.232165    5570 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1f35880-1a34-4214-a893-b1e1ba5f27c4-maas-api-xks-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:41:56.858502 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:41:56.858465    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-kwvfl"
Nov 03 20:41:56.858699 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:41:56.858499    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-kwvfl" event={"ID":"d1f35880-1a34-4214-a893-b1e1ba5f27c4","Type":"ContainerDied","Data":"1bc2e68ed5f653f857a4469b150930866ef4e9b46aa3816fb8a081bfca67a836"}
Nov 03 20:41:56.858699 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:41:56.858553    5570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1bc2e68ed5f653f857a4469b150930866ef4e9b46aa3816fb8a081bfca67a836"
Nov 03 20:41:57.228356 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:41:57.228321    5570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8bwhz" containerName="extractor"
Nov 03 20:42:03.933476 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:42:03.933442    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g69f8"]
Nov 03 20:42:08.734977 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:42:08.734915    5570 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-657vf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:42:08.735549 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:42:08.735019    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-657vf" podUID="f089dd20-d6a6-4c63-88a5-3fbe9a2ff315" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:42:09.508254 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:42:09.508200    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g69f8"]
Nov 03 20:42:13.949028 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:42:13.948985    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-p4kmk"]
Nov 03 20:42:13.950114 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:42:13.950090    5570 cpu_manager.go:414] "containerMap: removing container" podUID="d1f35880-1a34-4214-a893-b1e1ba5f27c4" containerName="revoke-keys"
Nov 03 20:42:13.950302 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:42:13.950286    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1f35880-1a34-4214-a893-b1e1ba5f27c4" containerName="revoke-keys"
Nov 03 20:42:13.950616 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:42:13.950600    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1f35880-1a34-4214-a893-b1e1ba5f27c4" containerName="revoke-keys"
Nov 03 20:42:13.956725 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:42:13.956698    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-p4kmk"
Nov 03 20:42:13.969732 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:42:13.969695    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-p4kmk"]
Nov 03 20:42:13.978396 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:42:13.978369    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l7s5w\""
Nov 03 20:42:13.978615 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:42:13.978386    5570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 20:42:13.978615 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:42:13.978463    5570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 20:42:13.986064 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:42:13.986034    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:42:14.069670 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:42:14.069632    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d6c1202-d2b4-4066-b6b9-b8ba9d7e3421-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-p4kmk\" (UID: \"4d6c1202-d2b4-4066-b6b9-b8ba9d7e3421\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-p4kmk"
Nov 03 20:42:14.069906 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:42:14.069687    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2jfb\" (UniqueName: \"kubernetes.io/projected/4d6c1202-d2b4-4066-b6b9-b8ba9d7e3421-kube-api-access-p2jfb\") pod \"maas-api-revoke-sub-a4aef7d7-p4kmk\" (UID: \"4d6c1202-d2b4-4066-b6b9-b8ba9d7e3421\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-p4kmk"
Nov 03 20:42:14.069906 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:42:14.069712    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d6c1202-d2b4-4066-b6b9-b8ba9d7e3421-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-p4kmk\" (UID: \"4d6c1202-d2b4-4066-b6b9-b8ba9d7e3421\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-p4kmk"
Nov 03 20:42:14.170612 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:42:14.170574    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d6c1202-d2b4-4066-b6b9-b8ba9d7e3421-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-p4kmk\" (UID: \"4d6c1202-d2b4-4066-b6b9-b8ba9d7e3421\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-p4kmk"
Nov 03 20:42:14.170874 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:42:14.170629    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2jfb\" (UniqueName: \"kubernetes.io/projected/4d6c1202-d2b4-4066-b6b9-b8ba9d7e3421-kube-api-access-p2jfb\") pod \"maas-api-revoke-sub-a4aef7d7-p4kmk\" (UID: \"4d6c1202-d2b4-4066-b6b9-b8ba9d7e3421\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-p4kmk"
Nov 03 20:42:14.170874 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:42:14.170739    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d6c1202-d2b4-4066-b6b9-b8ba9d7e3421-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-p4kmk\" (UID: \"4d6c1202-d2b4-4066-b6b9-b8ba9d7e3421\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-p4kmk"
Nov 03 20:42:14.171405 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:42:14.171384    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d6c1202-d2b4-4066-b6b9-b8ba9d7e3421-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-p4kmk\" (UID: \"4d6c1202-d2b4-4066-b6b9-b8ba9d7e3421\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-p4kmk"
Nov 03 20:42:14.173087 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:42:14.173062    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d6c1202-d2b4-4066-b6b9-b8ba9d7e3421-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-p4kmk\" (UID: \"4d6c1202-d2b4-4066-b6b9-b8ba9d7e3421\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-p4kmk"
Nov 03 20:42:14.190343 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:42:14.190309    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2jfb\" (UniqueName: \"kubernetes.io/projected/4d6c1202-d2b4-4066-b6b9-b8ba9d7e3421-kube-api-access-p2jfb\") pod \"maas-api-revoke-sub-a4aef7d7-p4kmk\" (UID: \"4d6c1202-d2b4-4066-b6b9-b8ba9d7e3421\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-p4kmk"
Nov 03 20:42:14.287437 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:42:14.287345    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-p4kmk"
Nov 03 20:42:14.584699 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:42:14.584666    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-p4kmk"]
Nov 03 20:42:14.585849 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:42:14.585823    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d6c1202_d2b4_4066_b6b9_b8ba9d7e3421.slice/crio-3ec3ace9a0a30e5894c37fd46206576144d66d8a846154c3e3d99e95f0b616b1 WatchSource:0}: Error finding container 3ec3ace9a0a30e5894c37fd46206576144d66d8a846154c3e3d99e95f0b616b1: Status 404 returned error can't find the container with id 3ec3ace9a0a30e5894c37fd46206576144d66d8a846154c3e3d99e95f0b616b1
Nov 03 20:42:14.978134 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:42:14.978087    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-p4kmk" event={"ID":"4d6c1202-d2b4-4066-b6b9-b8ba9d7e3421","Type":"ContainerStarted","Data":"66c06a9cd82318108ea0c87d760f7f57ab3bf5f4dae231006e13ac8622f4f7d6"}
Nov 03 20:42:14.978643 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:42:14.978148    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-p4kmk" event={"ID":"4d6c1202-d2b4-4066-b6b9-b8ba9d7e3421","Type":"ContainerStarted","Data":"3ec3ace9a0a30e5894c37fd46206576144d66d8a846154c3e3d99e95f0b616b1"}
Nov 03 20:42:15.018347 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:42:15.018291    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-p4kmk" podStartSLOduration=2.018277527 podStartE2EDuration="2.018277527s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:42:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:42:15.017582473 +0000 UTC m=+2269.170244359" watchObservedRunningTime="2026-11-03 20:42:15.018277527 +0000 UTC m=+2269.170939413"
Nov 03 20:42:15.985749 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:42:15.985719    5570 generic.go:356] "Generic (PLEG): container finished" podID="4d6c1202-d2b4-4066-b6b9-b8ba9d7e3421" containerID="66c06a9cd82318108ea0c87d760f7f57ab3bf5f4dae231006e13ac8622f4f7d6" exitCode=0
Nov 03 20:42:15.986177 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:42:15.985801    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-p4kmk" event={"ID":"4d6c1202-d2b4-4066-b6b9-b8ba9d7e3421","Type":"ContainerDied","Data":"66c06a9cd82318108ea0c87d760f7f57ab3bf5f4dae231006e13ac8622f4f7d6"}
Nov 03 20:42:17.127282 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:42:17.127253    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-p4kmk"
Nov 03 20:42:17.202011 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:42:17.201974    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4d6c1202-d2b4-4066-b6b9-b8ba9d7e3421-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d6c1202-d2b4-4066-b6b9-b8ba9d7e3421-maas-api-xks-ca\") pod \"4d6c1202-d2b4-4066-b6b9-b8ba9d7e3421\" (UID: \"4d6c1202-d2b4-4066-b6b9-b8ba9d7e3421\") "
Nov 03 20:42:17.202218 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:42:17.202043    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d6c1202-d2b4-4066-b6b9-b8ba9d7e3421-kube-api-access-p2jfb\" (UniqueName: \"kubernetes.io/projected/4d6c1202-d2b4-4066-b6b9-b8ba9d7e3421-kube-api-access-p2jfb\") pod \"4d6c1202-d2b4-4066-b6b9-b8ba9d7e3421\" (UID: \"4d6c1202-d2b4-4066-b6b9-b8ba9d7e3421\") "
Nov 03 20:42:17.202218 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:42:17.202105    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d6c1202-d2b4-4066-b6b9-b8ba9d7e3421-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d6c1202-d2b4-4066-b6b9-b8ba9d7e3421-maas-api-openshift-ca\") pod \"4d6c1202-d2b4-4066-b6b9-b8ba9d7e3421\" (UID: \"4d6c1202-d2b4-4066-b6b9-b8ba9d7e3421\") "
Nov 03 20:42:17.202576 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:42:17.202546    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d6c1202-d2b4-4066-b6b9-b8ba9d7e3421-maas-api-openshift-ca" pod "4d6c1202-d2b4-4066-b6b9-b8ba9d7e3421" (UID: "4d6c1202-d2b4-4066-b6b9-b8ba9d7e3421"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:42:17.204345 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:42:17.204308    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d6c1202-d2b4-4066-b6b9-b8ba9d7e3421-maas-api-xks-ca" pod "4d6c1202-d2b4-4066-b6b9-b8ba9d7e3421" (UID: "4d6c1202-d2b4-4066-b6b9-b8ba9d7e3421"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:42:17.204459 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:42:17.204415    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d6c1202-d2b4-4066-b6b9-b8ba9d7e3421-kube-api-access-p2jfb" pod "4d6c1202-d2b4-4066-b6b9-b8ba9d7e3421" (UID: "4d6c1202-d2b4-4066-b6b9-b8ba9d7e3421"). InnerVolumeSpecName "kube-api-access-p2jfb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:42:17.303862 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:42:17.303681    5570 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d6c1202-d2b4-4066-b6b9-b8ba9d7e3421-maas-api-xks-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:42:17.303862 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:42:17.303716    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2jfb\" (UniqueName: \"kubernetes.io/projected/4d6c1202-d2b4-4066-b6b9-b8ba9d7e3421-kube-api-access-p2jfb\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:42:17.303862 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:42:17.303726    5570 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d6c1202-d2b4-4066-b6b9-b8ba9d7e3421-maas-api-openshift-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:42:17.998262 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:42:17.998231    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-p4kmk"
Nov 03 20:42:17.998439 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:42:17.998266    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-p4kmk" event={"ID":"4d6c1202-d2b4-4066-b6b9-b8ba9d7e3421","Type":"ContainerDied","Data":"3ec3ace9a0a30e5894c37fd46206576144d66d8a846154c3e3d99e95f0b616b1"}
Nov 03 20:42:17.998439 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:42:17.998341    5570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3ec3ace9a0a30e5894c37fd46206576144d66d8a846154c3e3d99e95f0b616b1"
Nov 03 20:42:25.324078 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:42:25.324031    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g69f8"]
Nov 03 20:42:27.228371 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:42:27.228332    5570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8bwhz" containerName="extractor"
Nov 03 20:42:29.938126 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:42:29.938077    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g69f8"]
Nov 03 20:42:31.944801 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:42:31.938417    5570 scope.go:122] "RemoveContainer" containerID="155122ce70bbb1e1a1de2f68df5cf6b21b6860787f71654a7080ff6bea9b0bc4"
Nov 03 20:42:33.918051 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:42:33.918019    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-2ds6v"]
Nov 03 20:42:33.918589 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:42:33.918570    5570 cpu_manager.go:414] "containerMap: removing container" podUID="4d6c1202-d2b4-4066-b6b9-b8ba9d7e3421" containerName="revoke-keys"
Nov 03 20:42:33.918589 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:42:33.918590    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d6c1202-d2b4-4066-b6b9-b8ba9d7e3421" containerName="revoke-keys"
Nov 03 20:42:33.918727 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:42:33.918676    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d6c1202-d2b4-4066-b6b9-b8ba9d7e3421" containerName="revoke-keys"
Nov 03 20:42:33.923688 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:42:33.923655    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-2ds6v"
Nov 03 20:42:33.937887 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:42:33.937853    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-2ds6v"]
Nov 03 20:42:33.948419 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:42:33.948379    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:42:33.948615 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:42:33.948480    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l7s5w\""
Nov 03 20:42:34.015328 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:42:34.015293    5570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 20:42:34.015516 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:42:34.015349    5570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 20:42:34.023930 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:42:34.023898    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2t6t\" (UniqueName: \"kubernetes.io/projected/e19047f4-09c4-4a2d-b576-6988366782d8-kube-api-access-h2t6t\") pod \"maas-api-revoke-sub-79805ca9-2ds6v\" (UID: \"e19047f4-09c4-4a2d-b576-6988366782d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-2ds6v"
Nov 03 20:42:34.024000 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:42:34.023952    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e19047f4-09c4-4a2d-b576-6988366782d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-2ds6v\" (UID: \"e19047f4-09c4-4a2d-b576-6988366782d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-2ds6v"
Nov 03 20:42:34.024078 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:42:34.024058    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e19047f4-09c4-4a2d-b576-6988366782d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-2ds6v\" (UID: \"e19047f4-09c4-4a2d-b576-6988366782d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-2ds6v"
Nov 03 20:42:34.125471 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:42:34.125436    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2t6t\" (UniqueName: \"kubernetes.io/projected/e19047f4-09c4-4a2d-b576-6988366782d8-kube-api-access-h2t6t\") pod \"maas-api-revoke-sub-79805ca9-2ds6v\" (UID: \"e19047f4-09c4-4a2d-b576-6988366782d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-2ds6v"
Nov 03 20:42:34.125687 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:42:34.125480    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e19047f4-09c4-4a2d-b576-6988366782d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-2ds6v\" (UID: \"e19047f4-09c4-4a2d-b576-6988366782d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-2ds6v"
Nov 03 20:42:34.125687 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:42:34.125511    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e19047f4-09c4-4a2d-b576-6988366782d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-2ds6v\" (UID: \"e19047f4-09c4-4a2d-b576-6988366782d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-2ds6v"
Nov 03 20:42:34.126262 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:42:34.126233    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e19047f4-09c4-4a2d-b576-6988366782d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-2ds6v\" (UID: \"e19047f4-09c4-4a2d-b576-6988366782d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-2ds6v"
Nov 03 20:42:34.127943 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:42:34.127919    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e19047f4-09c4-4a2d-b576-6988366782d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-2ds6v\" (UID: \"e19047f4-09c4-4a2d-b576-6988366782d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-2ds6v"
Nov 03 20:42:34.152582 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:42:34.152541    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2t6t\" (UniqueName: \"kubernetes.io/projected/e19047f4-09c4-4a2d-b576-6988366782d8-kube-api-access-h2t6t\") pod \"maas-api-revoke-sub-79805ca9-2ds6v\" (UID: \"e19047f4-09c4-4a2d-b576-6988366782d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-2ds6v"
Nov 03 20:42:34.242495 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:42:34.242446    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-2ds6v"
Nov 03 20:42:34.538475 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:42:34.538442    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-2ds6v"]
Nov 03 20:42:34.540458 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:42:34.540427    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode19047f4_09c4_4a2d_b576_6988366782d8.slice/crio-44d102ac95e2d570984e145eaafc5d1e2e25f232aa9408fd101d74247a028f10 WatchSource:0}: Error finding container 44d102ac95e2d570984e145eaafc5d1e2e25f232aa9408fd101d74247a028f10: Status 404 returned error can't find the container with id 44d102ac95e2d570984e145eaafc5d1e2e25f232aa9408fd101d74247a028f10
Nov 03 20:42:35.127434 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:42:35.127341    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-2ds6v" event={"ID":"e19047f4-09c4-4a2d-b576-6988366782d8","Type":"ContainerStarted","Data":"db16a74203a9d0b80d98387b7dd504dd91b24824dff2ae882fda264c4a09475d"}
Nov 03 20:42:35.127434 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:42:35.127396    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-2ds6v" event={"ID":"e19047f4-09c4-4a2d-b576-6988366782d8","Type":"ContainerStarted","Data":"44d102ac95e2d570984e145eaafc5d1e2e25f232aa9408fd101d74247a028f10"}
Nov 03 20:42:35.166824 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:42:35.166753    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-2ds6v" podStartSLOduration=2.166737988 podStartE2EDuration="2.166737988s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:42:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:42:35.165975578 +0000 UTC m=+2289.318637465" watchObservedRunningTime="2026-11-03 20:42:35.166737988 +0000 UTC m=+2289.319399875"
Nov 03 20:42:36.134677 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:42:36.134642    5570 generic.go:356] "Generic (PLEG): container finished" podID="e19047f4-09c4-4a2d-b576-6988366782d8" containerID="db16a74203a9d0b80d98387b7dd504dd91b24824dff2ae882fda264c4a09475d" exitCode=0
Nov 03 20:42:36.135117 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:42:36.134708    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-2ds6v" event={"ID":"e19047f4-09c4-4a2d-b576-6988366782d8","Type":"ContainerDied","Data":"db16a74203a9d0b80d98387b7dd504dd91b24824dff2ae882fda264c4a09475d"}
Nov 03 20:42:37.281515 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:42:37.281484    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-2ds6v"
Nov 03 20:42:37.360751 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:42:37.360714    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e19047f4-09c4-4a2d-b576-6988366782d8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e19047f4-09c4-4a2d-b576-6988366782d8-maas-api-openshift-ca\") pod \"e19047f4-09c4-4a2d-b576-6988366782d8\" (UID: \"e19047f4-09c4-4a2d-b576-6988366782d8\") "
Nov 03 20:42:37.360950 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:42:37.360854    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e19047f4-09c4-4a2d-b576-6988366782d8-kube-api-access-h2t6t\" (UniqueName: \"kubernetes.io/projected/e19047f4-09c4-4a2d-b576-6988366782d8-kube-api-access-h2t6t\") pod \"e19047f4-09c4-4a2d-b576-6988366782d8\" (UID: \"e19047f4-09c4-4a2d-b576-6988366782d8\") "
Nov 03 20:42:37.360950 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:42:37.360929    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e19047f4-09c4-4a2d-b576-6988366782d8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e19047f4-09c4-4a2d-b576-6988366782d8-maas-api-xks-ca\") pod \"e19047f4-09c4-4a2d-b576-6988366782d8\" (UID: \"e19047f4-09c4-4a2d-b576-6988366782d8\") "
Nov 03 20:42:37.361195 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:42:37.361171    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e19047f4-09c4-4a2d-b576-6988366782d8-maas-api-openshift-ca" pod "e19047f4-09c4-4a2d-b576-6988366782d8" (UID: "e19047f4-09c4-4a2d-b576-6988366782d8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:42:37.361312 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:42:37.361298    5570 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e19047f4-09c4-4a2d-b576-6988366782d8-maas-api-openshift-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:42:37.363036 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:42:37.363001    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e19047f4-09c4-4a2d-b576-6988366782d8-kube-api-access-h2t6t" pod "e19047f4-09c4-4a2d-b576-6988366782d8" (UID: "e19047f4-09c4-4a2d-b576-6988366782d8"). InnerVolumeSpecName "kube-api-access-h2t6t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:42:37.363160 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:42:37.363039    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e19047f4-09c4-4a2d-b576-6988366782d8-maas-api-xks-ca" pod "e19047f4-09c4-4a2d-b576-6988366782d8" (UID: "e19047f4-09c4-4a2d-b576-6988366782d8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:42:37.462344 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:42:37.462300    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2t6t\" (UniqueName: \"kubernetes.io/projected/e19047f4-09c4-4a2d-b576-6988366782d8-kube-api-access-h2t6t\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:42:37.462344 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:42:37.462341    5570 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e19047f4-09c4-4a2d-b576-6988366782d8-maas-api-xks-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:42:38.148217 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:42:38.148180    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-2ds6v" event={"ID":"e19047f4-09c4-4a2d-b576-6988366782d8","Type":"ContainerDied","Data":"44d102ac95e2d570984e145eaafc5d1e2e25f232aa9408fd101d74247a028f10"}
Nov 03 20:42:38.148217 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:42:38.148219    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-2ds6v"
Nov 03 20:42:38.148454 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:42:38.148233    5570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="44d102ac95e2d570984e145eaafc5d1e2e25f232aa9408fd101d74247a028f10"
Nov 03 20:42:45.239715 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:42:45.239679    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g69f8"]
Nov 03 20:42:50.681357 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:42:50.681315    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g69f8"]
Nov 03 20:42:53.951518 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:42:53.951472    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-fjm82"]
Nov 03 20:42:53.952115 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:42:53.952082    5570 cpu_manager.go:414] "containerMap: removing container" podUID="e19047f4-09c4-4a2d-b576-6988366782d8" containerName="revoke-keys"
Nov 03 20:42:53.952115 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:42:53.952097    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e19047f4-09c4-4a2d-b576-6988366782d8" containerName="revoke-keys"
Nov 03 20:42:53.952302 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:42:53.952183    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="e19047f4-09c4-4a2d-b576-6988366782d8" containerName="revoke-keys"
Nov 03 20:42:53.955404 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:42:53.955382    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-fjm82"
Nov 03 20:42:53.971262 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:42:53.971230    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-fjm82"]
Nov 03 20:42:53.981648 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:42:53.981613    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:42:53.981893 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:42:53.981704    5570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 20:42:53.981893 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:42:53.981623    5570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 20:42:53.981893 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:42:53.981841    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l7s5w\""
Nov 03 20:42:53.999695 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:42:53.999662    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bc5d581-0ac5-4623-b41a-0e1b81e964f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-fjm82\" (UID: \"9bc5d581-0ac5-4623-b41a-0e1b81e964f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-fjm82"
Nov 03 20:42:53.999934 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:42:53.999747    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zj89k\" (UniqueName: \"kubernetes.io/projected/9bc5d581-0ac5-4623-b41a-0e1b81e964f3-kube-api-access-zj89k\") pod \"maas-api-revoke-sub-0b7e8edb-fjm82\" (UID: \"9bc5d581-0ac5-4623-b41a-0e1b81e964f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-fjm82"
Nov 03 20:42:53.999934 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:42:53.999896    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bc5d581-0ac5-4623-b41a-0e1b81e964f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-fjm82\" (UID: \"9bc5d581-0ac5-4623-b41a-0e1b81e964f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-fjm82"
Nov 03 20:42:54.100989 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:42:54.100946    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bc5d581-0ac5-4623-b41a-0e1b81e964f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-fjm82\" (UID: \"9bc5d581-0ac5-4623-b41a-0e1b81e964f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-fjm82"
Nov 03 20:42:54.101206 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:42:54.101016    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zj89k\" (UniqueName: \"kubernetes.io/projected/9bc5d581-0ac5-4623-b41a-0e1b81e964f3-kube-api-access-zj89k\") pod \"maas-api-revoke-sub-0b7e8edb-fjm82\" (UID: \"9bc5d581-0ac5-4623-b41a-0e1b81e964f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-fjm82"
Nov 03 20:42:54.101206 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:42:54.101075    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bc5d581-0ac5-4623-b41a-0e1b81e964f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-fjm82\" (UID: \"9bc5d581-0ac5-4623-b41a-0e1b81e964f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-fjm82"
Nov 03 20:42:54.101723 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:42:54.101696    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bc5d581-0ac5-4623-b41a-0e1b81e964f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-fjm82\" (UID: \"9bc5d581-0ac5-4623-b41a-0e1b81e964f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-fjm82"
Nov 03 20:42:54.103524 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:42:54.103498    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bc5d581-0ac5-4623-b41a-0e1b81e964f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-fjm82\" (UID: \"9bc5d581-0ac5-4623-b41a-0e1b81e964f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-fjm82"
Nov 03 20:42:54.121359 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:42:54.121327    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zj89k\" (UniqueName: \"kubernetes.io/projected/9bc5d581-0ac5-4623-b41a-0e1b81e964f3-kube-api-access-zj89k\") pod \"maas-api-revoke-sub-0b7e8edb-fjm82\" (UID: \"9bc5d581-0ac5-4623-b41a-0e1b81e964f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-fjm82"
Nov 03 20:42:54.271940 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:42:54.271825    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-fjm82"
Nov 03 20:42:54.568222 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:42:54.568191    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-fjm82"]
Nov 03 20:42:54.568941 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:42:54.568913    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bc5d581_0ac5_4623_b41a_0e1b81e964f3.slice/crio-295038308cbbf8f580b916f7a0fe2113e148ac0fb029f47c6f1ba6360bf3463e WatchSource:0}: Error finding container 295038308cbbf8f580b916f7a0fe2113e148ac0fb029f47c6f1ba6360bf3463e: Status 404 returned error can't find the container with id 295038308cbbf8f580b916f7a0fe2113e148ac0fb029f47c6f1ba6360bf3463e
Nov 03 20:42:55.328816 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:42:55.328756    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-fjm82" event={"ID":"9bc5d581-0ac5-4623-b41a-0e1b81e964f3","Type":"ContainerStarted","Data":"278fc687282538009a3be94bba855d1f3814b710572018392f1fe809703c6a96"}
Nov 03 20:42:55.329377 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:42:55.328845    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-fjm82" event={"ID":"9bc5d581-0ac5-4623-b41a-0e1b81e964f3","Type":"ContainerStarted","Data":"295038308cbbf8f580b916f7a0fe2113e148ac0fb029f47c6f1ba6360bf3463e"}
Nov 03 20:42:55.370435 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:42:55.370374    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-fjm82" podStartSLOduration=2.370358662 podStartE2EDuration="2.370358662s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:42:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:42:55.367902496 +0000 UTC m=+2309.520564386" watchObservedRunningTime="2026-11-03 20:42:55.370358662 +0000 UTC m=+2309.523020587"
Nov 03 20:42:56.335428 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:42:56.335391    5570 generic.go:356] "Generic (PLEG): container finished" podID="9bc5d581-0ac5-4623-b41a-0e1b81e964f3" containerID="278fc687282538009a3be94bba855d1f3814b710572018392f1fe809703c6a96" exitCode=0
Nov 03 20:42:56.335967 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:42:56.335476    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-fjm82" event={"ID":"9bc5d581-0ac5-4623-b41a-0e1b81e964f3","Type":"ContainerDied","Data":"278fc687282538009a3be94bba855d1f3814b710572018392f1fe809703c6a96"}
Nov 03 20:42:57.227466 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:42:57.227431    5570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8bwhz" containerName="extractor"
Nov 03 20:42:57.480225 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:42:57.480148    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-fjm82"
Nov 03 20:42:57.535617 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:42:57.535581    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bc5d581-0ac5-4623-b41a-0e1b81e964f3-kube-api-access-zj89k\" (UniqueName: \"kubernetes.io/projected/9bc5d581-0ac5-4623-b41a-0e1b81e964f3-kube-api-access-zj89k\") pod \"9bc5d581-0ac5-4623-b41a-0e1b81e964f3\" (UID: \"9bc5d581-0ac5-4623-b41a-0e1b81e964f3\") "
Nov 03 20:42:57.535819 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:42:57.535635    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9bc5d581-0ac5-4623-b41a-0e1b81e964f3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bc5d581-0ac5-4623-b41a-0e1b81e964f3-maas-api-xks-ca\") pod \"9bc5d581-0ac5-4623-b41a-0e1b81e964f3\" (UID: \"9bc5d581-0ac5-4623-b41a-0e1b81e964f3\") "
Nov 03 20:42:57.535819 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:42:57.535766    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9bc5d581-0ac5-4623-b41a-0e1b81e964f3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bc5d581-0ac5-4623-b41a-0e1b81e964f3-maas-api-openshift-ca\") pod \"9bc5d581-0ac5-4623-b41a-0e1b81e964f3\" (UID: \"9bc5d581-0ac5-4623-b41a-0e1b81e964f3\") "
Nov 03 20:42:57.536157 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:42:57.536130    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bc5d581-0ac5-4623-b41a-0e1b81e964f3-maas-api-openshift-ca" pod "9bc5d581-0ac5-4623-b41a-0e1b81e964f3" (UID: "9bc5d581-0ac5-4623-b41a-0e1b81e964f3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:42:57.537946 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:42:57.537920    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9bc5d581-0ac5-4623-b41a-0e1b81e964f3-maas-api-xks-ca" pod "9bc5d581-0ac5-4623-b41a-0e1b81e964f3" (UID: "9bc5d581-0ac5-4623-b41a-0e1b81e964f3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:42:57.538038 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:42:57.537950    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bc5d581-0ac5-4623-b41a-0e1b81e964f3-kube-api-access-zj89k" pod "9bc5d581-0ac5-4623-b41a-0e1b81e964f3" (UID: "9bc5d581-0ac5-4623-b41a-0e1b81e964f3"). InnerVolumeSpecName "kube-api-access-zj89k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:42:57.637309 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:42:57.637265    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zj89k\" (UniqueName: \"kubernetes.io/projected/9bc5d581-0ac5-4623-b41a-0e1b81e964f3-kube-api-access-zj89k\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:42:57.637309 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:42:57.637302    5570 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bc5d581-0ac5-4623-b41a-0e1b81e964f3-maas-api-xks-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:42:57.637309 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:42:57.637314    5570 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bc5d581-0ac5-4623-b41a-0e1b81e964f3-maas-api-openshift-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:42:58.348385 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:42:58.348297    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-fjm82"
Nov 03 20:42:58.348385 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:42:58.348309    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-fjm82" event={"ID":"9bc5d581-0ac5-4623-b41a-0e1b81e964f3","Type":"ContainerDied","Data":"295038308cbbf8f580b916f7a0fe2113e148ac0fb029f47c6f1ba6360bf3463e"}
Nov 03 20:42:58.348385 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:42:58.348352    5570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="295038308cbbf8f580b916f7a0fe2113e148ac0fb029f47c6f1ba6360bf3463e"
Nov 03 20:43:04.672760 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:43:04.672724    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rmhxp"]
Nov 03 20:43:04.673219 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:43:04.673202    5570 cpu_manager.go:414] "containerMap: removing container" podUID="9bc5d581-0ac5-4623-b41a-0e1b81e964f3" containerName="revoke-keys"
Nov 03 20:43:04.673266 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:43:04.673221    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bc5d581-0ac5-4623-b41a-0e1b81e964f3" containerName="revoke-keys"
Nov 03 20:43:04.673358 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:43:04.673348    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bc5d581-0ac5-4623-b41a-0e1b81e964f3" containerName="revoke-keys"
Nov 03 20:43:04.676743 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:43:04.676706    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rmhxp"
Nov 03 20:43:04.692522 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:43:04.692483    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rmhxp"]
Nov 03 20:43:04.702809 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:43:04.702764    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:43:04.702985 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:43:04.702868    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l7s5w\""
Nov 03 20:43:04.713011 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:43:04.712974    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26f6c315-fa59-482a-9df2-47381546cb87-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-rmhxp\" (UID: \"26f6c315-fa59-482a-9df2-47381546cb87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rmhxp"
Nov 03 20:43:04.713224 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:43:04.713165    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26f6c315-fa59-482a-9df2-47381546cb87-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-rmhxp\" (UID: \"26f6c315-fa59-482a-9df2-47381546cb87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rmhxp"
Nov 03 20:43:04.713373 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:43:04.713335    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2mff\" (UniqueName: \"kubernetes.io/projected/26f6c315-fa59-482a-9df2-47381546cb87-kube-api-access-z2mff\") pod \"maas-api-revoke-sub-3d4156f1-rmhxp\" (UID: \"26f6c315-fa59-482a-9df2-47381546cb87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rmhxp"
Nov 03 20:43:04.764037 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:43:04.763999    5570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 20:43:04.764037 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:43:04.764025    5570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 20:43:04.815096 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:43:04.815054    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2mff\" (UniqueName: \"kubernetes.io/projected/26f6c315-fa59-482a-9df2-47381546cb87-kube-api-access-z2mff\") pod \"maas-api-revoke-sub-3d4156f1-rmhxp\" (UID: \"26f6c315-fa59-482a-9df2-47381546cb87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rmhxp"
Nov 03 20:43:04.815294 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:43:04.815114    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26f6c315-fa59-482a-9df2-47381546cb87-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-rmhxp\" (UID: \"26f6c315-fa59-482a-9df2-47381546cb87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rmhxp"
Nov 03 20:43:04.815294 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:43:04.815162    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26f6c315-fa59-482a-9df2-47381546cb87-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-rmhxp\" (UID: \"26f6c315-fa59-482a-9df2-47381546cb87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rmhxp"
Nov 03 20:43:04.815898 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:43:04.815856    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26f6c315-fa59-482a-9df2-47381546cb87-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-rmhxp\" (UID: \"26f6c315-fa59-482a-9df2-47381546cb87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rmhxp"
Nov 03 20:43:04.817673 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:43:04.817647    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26f6c315-fa59-482a-9df2-47381546cb87-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-rmhxp\" (UID: \"26f6c315-fa59-482a-9df2-47381546cb87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rmhxp"
Nov 03 20:43:04.836270 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:43:04.836232    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2mff\" (UniqueName: \"kubernetes.io/projected/26f6c315-fa59-482a-9df2-47381546cb87-kube-api-access-z2mff\") pod \"maas-api-revoke-sub-3d4156f1-rmhxp\" (UID: \"26f6c315-fa59-482a-9df2-47381546cb87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rmhxp"
Nov 03 20:43:04.990624 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:43:04.990585    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rmhxp"
Nov 03 20:43:05.264289 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:43:05.264203    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g69f8"]
Nov 03 20:43:05.292903 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:43:05.292878    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rmhxp"]
Nov 03 20:43:05.294469 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:43:05.294439    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26f6c315_fa59_482a_9df2_47381546cb87.slice/crio-4a9f7efe6a73d35edf633891db561a44d73cb696e3bd30d17c285c110e0351da WatchSource:0}: Error finding container 4a9f7efe6a73d35edf633891db561a44d73cb696e3bd30d17c285c110e0351da: Status 404 returned error can't find the container with id 4a9f7efe6a73d35edf633891db561a44d73cb696e3bd30d17c285c110e0351da
Nov 03 20:43:05.389274 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:43:05.389237    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rmhxp" event={"ID":"26f6c315-fa59-482a-9df2-47381546cb87","Type":"ContainerStarted","Data":"4a9f7efe6a73d35edf633891db561a44d73cb696e3bd30d17c285c110e0351da"}
Nov 03 20:43:06.400440 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:43:06.400340    5570 generic.go:356] "Generic (PLEG): container finished" podID="26f6c315-fa59-482a-9df2-47381546cb87" containerID="a94392be6662a165b2ec36410dfb91d76f3b80c6a8f08efb99ddba52df5ada1a" exitCode=0
Nov 03 20:43:06.400853 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:43:06.400442    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rmhxp" event={"ID":"26f6c315-fa59-482a-9df2-47381546cb87","Type":"ContainerDied","Data":"a94392be6662a165b2ec36410dfb91d76f3b80c6a8f08efb99ddba52df5ada1a"}
Nov 03 20:43:07.027348 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:43:07.027312    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa4571df-5g52m"]
Nov 03 20:43:07.040579 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:43:07.040545    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa4571df-5g52m"]
Nov 03 20:43:07.544492 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:43:07.544466    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rmhxp"
Nov 03 20:43:07.644763 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:43:07.644716    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/26f6c315-fa59-482a-9df2-47381546cb87-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26f6c315-fa59-482a-9df2-47381546cb87-maas-api-xks-ca\") pod \"26f6c315-fa59-482a-9df2-47381546cb87\" (UID: \"26f6c315-fa59-482a-9df2-47381546cb87\") "
Nov 03 20:43:07.645018 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:43:07.644920    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26f6c315-fa59-482a-9df2-47381546cb87-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26f6c315-fa59-482a-9df2-47381546cb87-maas-api-openshift-ca\") pod \"26f6c315-fa59-482a-9df2-47381546cb87\" (UID: \"26f6c315-fa59-482a-9df2-47381546cb87\") "
Nov 03 20:43:07.645018 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:43:07.644982    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26f6c315-fa59-482a-9df2-47381546cb87-kube-api-access-z2mff\" (UniqueName: \"kubernetes.io/projected/26f6c315-fa59-482a-9df2-47381546cb87-kube-api-access-z2mff\") pod \"26f6c315-fa59-482a-9df2-47381546cb87\" (UID: \"26f6c315-fa59-482a-9df2-47381546cb87\") "
Nov 03 20:43:07.645319 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:43:07.645292    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26f6c315-fa59-482a-9df2-47381546cb87-maas-api-openshift-ca" pod "26f6c315-fa59-482a-9df2-47381546cb87" (UID: "26f6c315-fa59-482a-9df2-47381546cb87"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:43:07.646874 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:43:07.646846    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26f6c315-fa59-482a-9df2-47381546cb87-maas-api-xks-ca" pod "26f6c315-fa59-482a-9df2-47381546cb87" (UID: "26f6c315-fa59-482a-9df2-47381546cb87"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:43:07.646994 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:43:07.646974    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26f6c315-fa59-482a-9df2-47381546cb87-kube-api-access-z2mff" pod "26f6c315-fa59-482a-9df2-47381546cb87" (UID: "26f6c315-fa59-482a-9df2-47381546cb87"). InnerVolumeSpecName "kube-api-access-z2mff". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:43:07.746310 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:43:07.746267    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2mff\" (UniqueName: \"kubernetes.io/projected/26f6c315-fa59-482a-9df2-47381546cb87-kube-api-access-z2mff\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:43:07.746310 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:43:07.746305    5570 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26f6c315-fa59-482a-9df2-47381546cb87-maas-api-xks-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:43:07.746310 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:43:07.746313    5570 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26f6c315-fa59-482a-9df2-47381546cb87-maas-api-openshift-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:43:07.962813 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:43:07.962763    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="746cdf44-bdd0-4040-87a9-43ff9b48cf99" path="/var/lib/kubelet/pods/746cdf44-bdd0-4040-87a9-43ff9b48cf99/volumes"
Nov 03 20:43:08.413349 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:43:08.413316    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rmhxp"
Nov 03 20:43:08.413551 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:43:08.413358    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rmhxp" event={"ID":"26f6c315-fa59-482a-9df2-47381546cb87","Type":"ContainerDied","Data":"4a9f7efe6a73d35edf633891db561a44d73cb696e3bd30d17c285c110e0351da"}
Nov 03 20:43:08.413551 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:43:08.413411    5570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4a9f7efe6a73d35edf633891db561a44d73cb696e3bd30d17c285c110e0351da"
Nov 03 20:43:15.985889 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:43:15.985847    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g69f8"]
Nov 03 20:43:21.800798 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:43:21.795655    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g69f8"]
Nov 03 20:43:24.955268 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:43:24.955228    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rmhxp"]
Nov 03 20:43:24.968584 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:43:24.968549    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rmhxp"]
Nov 03 20:43:25.003059 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:43:25.003019    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2tfnh"]
Nov 03 20:43:25.003530 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:43:25.003513    5570 cpu_manager.go:414] "containerMap: removing container" podUID="26f6c315-fa59-482a-9df2-47381546cb87" containerName="revoke-keys"
Nov 03 20:43:25.003580 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:43:25.003533    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26f6c315-fa59-482a-9df2-47381546cb87" containerName="revoke-keys"
Nov 03 20:43:25.003629 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:43:25.003605    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="26f6c315-fa59-482a-9df2-47381546cb87" containerName="revoke-keys"
Nov 03 20:43:25.006854 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:43:25.006827    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2tfnh"
Nov 03 20:43:25.022547 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:43:25.022499    5570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 20:43:25.023221 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:43:25.023197    5570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 20:43:25.023303 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:43:25.023220    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l7s5w\""
Nov 03 20:43:25.023303 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:43:25.023275    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:43:25.025620 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:43:25.025586    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2tfnh"]
Nov 03 20:43:25.105262 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:43:25.105220    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8212f373-2ed9-4518-9736-a02b538b7164-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-2tfnh\" (UID: \"8212f373-2ed9-4518-9736-a02b538b7164\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2tfnh"
Nov 03 20:43:25.105466 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:43:25.105351    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8212f373-2ed9-4518-9736-a02b538b7164-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-2tfnh\" (UID: \"8212f373-2ed9-4518-9736-a02b538b7164\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2tfnh"
Nov 03 20:43:25.105527 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:43:25.105477    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtpp2\" (UniqueName: \"kubernetes.io/projected/8212f373-2ed9-4518-9736-a02b538b7164-kube-api-access-gtpp2\") pod \"maas-api-revoke-sub-3d4156f1-2tfnh\" (UID: \"8212f373-2ed9-4518-9736-a02b538b7164\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2tfnh"
Nov 03 20:43:25.207204 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:43:25.207110    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8212f373-2ed9-4518-9736-a02b538b7164-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-2tfnh\" (UID: \"8212f373-2ed9-4518-9736-a02b538b7164\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2tfnh"
Nov 03 20:43:25.207204 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:43:25.207195    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8212f373-2ed9-4518-9736-a02b538b7164-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-2tfnh\" (UID: \"8212f373-2ed9-4518-9736-a02b538b7164\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2tfnh"
Nov 03 20:43:25.207394 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:43:25.207227    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtpp2\" (UniqueName: \"kubernetes.io/projected/8212f373-2ed9-4518-9736-a02b538b7164-kube-api-access-gtpp2\") pod \"maas-api-revoke-sub-3d4156f1-2tfnh\" (UID: \"8212f373-2ed9-4518-9736-a02b538b7164\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2tfnh"
Nov 03 20:43:25.207939 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:43:25.207899    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8212f373-2ed9-4518-9736-a02b538b7164-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-2tfnh\" (UID: \"8212f373-2ed9-4518-9736-a02b538b7164\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2tfnh"
Nov 03 20:43:25.209845 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:43:25.209803    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8212f373-2ed9-4518-9736-a02b538b7164-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-2tfnh\" (UID: \"8212f373-2ed9-4518-9736-a02b538b7164\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2tfnh"
Nov 03 20:43:25.228035 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:43:25.227999    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtpp2\" (UniqueName: \"kubernetes.io/projected/8212f373-2ed9-4518-9736-a02b538b7164-kube-api-access-gtpp2\") pod \"maas-api-revoke-sub-3d4156f1-2tfnh\" (UID: \"8212f373-2ed9-4518-9736-a02b538b7164\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2tfnh"
Nov 03 20:43:25.321337 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:43:25.321296    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2tfnh"
Nov 03 20:43:25.618223 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:43:25.618192    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2tfnh"]
Nov 03 20:43:25.618875 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:43:25.618849    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8212f373_2ed9_4518_9736_a02b538b7164.slice/crio-79523d789810d581f29411ed33764f4a968384b59e5616794a68d3a768b4b450 WatchSource:0}: Error finding container 79523d789810d581f29411ed33764f4a968384b59e5616794a68d3a768b4b450: Status 404 returned error can't find the container with id 79523d789810d581f29411ed33764f4a968384b59e5616794a68d3a768b4b450
Nov 03 20:43:25.964494 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:43:25.964461    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26f6c315-fa59-482a-9df2-47381546cb87" path="/var/lib/kubelet/pods/26f6c315-fa59-482a-9df2-47381546cb87/volumes"
Nov 03 20:43:26.567093 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:43:26.567054    5570 generic.go:356] "Generic (PLEG): container finished" podID="8212f373-2ed9-4518-9736-a02b538b7164" containerID="f1c9d6f66d29f0c3c930fb4dec17b43a9739ee52ae8954d58967fd6f30f1db56" exitCode=0
Nov 03 20:43:26.567287 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:43:26.567139    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2tfnh" event={"ID":"8212f373-2ed9-4518-9736-a02b538b7164","Type":"ContainerDied","Data":"f1c9d6f66d29f0c3c930fb4dec17b43a9739ee52ae8954d58967fd6f30f1db56"}
Nov 03 20:43:26.567287 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:43:26.567191    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2tfnh" event={"ID":"8212f373-2ed9-4518-9736-a02b538b7164","Type":"ContainerStarted","Data":"79523d789810d581f29411ed33764f4a968384b59e5616794a68d3a768b4b450"}
Nov 03 20:43:27.228879 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:43:27.228837    5570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8bwhz" containerName="extractor"
Nov 03 20:43:27.717021 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:43:27.716993    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2tfnh"
Nov 03 20:43:27.834295 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:43:27.834199    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8212f373-2ed9-4518-9736-a02b538b7164-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8212f373-2ed9-4518-9736-a02b538b7164-maas-api-openshift-ca\") pod \"8212f373-2ed9-4518-9736-a02b538b7164\" (UID: \"8212f373-2ed9-4518-9736-a02b538b7164\") "
Nov 03 20:43:27.834466 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:43:27.834337    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8212f373-2ed9-4518-9736-a02b538b7164-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8212f373-2ed9-4518-9736-a02b538b7164-maas-api-xks-ca\") pod \"8212f373-2ed9-4518-9736-a02b538b7164\" (UID: \"8212f373-2ed9-4518-9736-a02b538b7164\") "
Nov 03 20:43:27.834552 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:43:27.834531    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8212f373-2ed9-4518-9736-a02b538b7164-kube-api-access-gtpp2\" (UniqueName: \"kubernetes.io/projected/8212f373-2ed9-4518-9736-a02b538b7164-kube-api-access-gtpp2\") pod \"8212f373-2ed9-4518-9736-a02b538b7164\" (UID: \"8212f373-2ed9-4518-9736-a02b538b7164\") "
Nov 03 20:43:27.834631 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:43:27.834614    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8212f373-2ed9-4518-9736-a02b538b7164-maas-api-openshift-ca" pod "8212f373-2ed9-4518-9736-a02b538b7164" (UID: "8212f373-2ed9-4518-9736-a02b538b7164"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:43:27.834888 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:43:27.834861    5570 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8212f373-2ed9-4518-9736-a02b538b7164-maas-api-openshift-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:43:27.836431 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:43:27.836402    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8212f373-2ed9-4518-9736-a02b538b7164-maas-api-xks-ca" pod "8212f373-2ed9-4518-9736-a02b538b7164" (UID: "8212f373-2ed9-4518-9736-a02b538b7164"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:43:27.836558 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:43:27.836540    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8212f373-2ed9-4518-9736-a02b538b7164-kube-api-access-gtpp2" pod "8212f373-2ed9-4518-9736-a02b538b7164" (UID: "8212f373-2ed9-4518-9736-a02b538b7164"). InnerVolumeSpecName "kube-api-access-gtpp2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:43:27.935707 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:43:27.935661    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gtpp2\" (UniqueName: \"kubernetes.io/projected/8212f373-2ed9-4518-9736-a02b538b7164-kube-api-access-gtpp2\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:43:27.935707 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:43:27.935700    5570 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8212f373-2ed9-4518-9736-a02b538b7164-maas-api-xks-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:43:28.579046 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:43:28.579016    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2tfnh"
Nov 03 20:43:28.579457 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:43:28.579046    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2tfnh" event={"ID":"8212f373-2ed9-4518-9736-a02b538b7164","Type":"ContainerDied","Data":"79523d789810d581f29411ed33764f4a968384b59e5616794a68d3a768b4b450"}
Nov 03 20:43:28.579457 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:43:28.579124    5570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="79523d789810d581f29411ed33764f4a968384b59e5616794a68d3a768b4b450"
Nov 03 20:43:32.074445 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:43:32.074413    5570 scope.go:122] "RemoveContainer" containerID="b88ff525475e91c8387174eb572c2b939820a5b2c8fac17dc14258e4bbca7c25"
Nov 03 20:43:32.083703 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:43:32.083677    5570 scope.go:122] "RemoveContainer" containerID="be059e412f1853a8c5debbfb8af9c7ba18de02b070219f07598fbd3f5bde7924"
Nov 03 20:43:32.092469 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:43:32.092446    5570 scope.go:122] "RemoveContainer" containerID="b525da23b42a2bfa208703a4df2247e35b213257c3859f49c2a4771f1a785802"
Nov 03 20:43:32.100816 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:43:32.100796    5570 scope.go:122] "RemoveContainer" containerID="68afaae48b42fbeeb53562b5cbe83c10698a53b81cb7c101633a459964020902"
Nov 03 20:43:32.109425 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:43:32.109403    5570 scope.go:122] "RemoveContainer" containerID="c208803397eb8d4054c1b188174da01748b3d7cad8a5135b673477bfe42743f9"
Nov 03 20:43:37.027349 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:43:37.027266    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-966ac553-x29lb"]
Nov 03 20:43:37.040911 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:43:37.040880    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-966ac553-x29lb"]
Nov 03 20:43:37.158418 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:43:37.158377    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g69f8"]
Nov 03 20:43:37.963761 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:43:37.963710    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f075a0ee-7327-4a89-bd17-a02edd236f9f" path="/var/lib/kubelet/pods/f075a0ee-7327-4a89-bd17-a02edd236f9f/volumes"
Nov 03 20:43:42.669883 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:43:42.669851    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g69f8"]
Nov 03 20:43:54.286897 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:43:54.286854    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g69f8"]
Nov 03 20:43:57.311800 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:43:57.311079    5570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8bwhz" containerName="extractor"
Nov 03 20:44:00.575162 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:00.575122    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-pbxdh"]
Nov 03 20:44:00.575665 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:00.575556    5570 cpu_manager.go:414] "containerMap: removing container" podUID="8212f373-2ed9-4518-9736-a02b538b7164" containerName="revoke-keys"
Nov 03 20:44:00.575665 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:00.575567    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8212f373-2ed9-4518-9736-a02b538b7164" containerName="revoke-keys"
Nov 03 20:44:00.575742 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:00.575669    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="8212f373-2ed9-4518-9736-a02b538b7164" containerName="revoke-keys"
Nov 03 20:44:00.579169 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:00.579142    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-pbxdh"
Nov 03 20:44:00.594215 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:00.594183    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-pbxdh"]
Nov 03 20:44:00.605227 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:00.605194    5570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 20:44:00.605413 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:00.605235    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:44:00.605413 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:00.605280    5570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 20:44:00.608179 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:00.608160    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l7s5w\""
Nov 03 20:44:00.732827 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:00.732784    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e07a89e4-f07f-4a03-ba49-f310f25343a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-pbxdh\" (UID: \"e07a89e4-f07f-4a03-ba49-f310f25343a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-pbxdh"
Nov 03 20:44:00.733030 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:00.732892    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcpwg\" (UniqueName: \"kubernetes.io/projected/e07a89e4-f07f-4a03-ba49-f310f25343a0-kube-api-access-wcpwg\") pod \"maas-api-revoke-sub-e2fc9c5b-pbxdh\" (UID: \"e07a89e4-f07f-4a03-ba49-f310f25343a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-pbxdh"
Nov 03 20:44:00.733030 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:00.733007    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e07a89e4-f07f-4a03-ba49-f310f25343a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-pbxdh\" (UID: \"e07a89e4-f07f-4a03-ba49-f310f25343a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-pbxdh"
Nov 03 20:44:00.833921 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:00.833819    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e07a89e4-f07f-4a03-ba49-f310f25343a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-pbxdh\" (UID: \"e07a89e4-f07f-4a03-ba49-f310f25343a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-pbxdh"
Nov 03 20:44:00.833921 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:00.833879    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e07a89e4-f07f-4a03-ba49-f310f25343a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-pbxdh\" (UID: \"e07a89e4-f07f-4a03-ba49-f310f25343a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-pbxdh"
Nov 03 20:44:00.834162 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:00.833975    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcpwg\" (UniqueName: \"kubernetes.io/projected/e07a89e4-f07f-4a03-ba49-f310f25343a0-kube-api-access-wcpwg\") pod \"maas-api-revoke-sub-e2fc9c5b-pbxdh\" (UID: \"e07a89e4-f07f-4a03-ba49-f310f25343a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-pbxdh"
Nov 03 20:44:00.834623 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:00.834598    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e07a89e4-f07f-4a03-ba49-f310f25343a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-pbxdh\" (UID: \"e07a89e4-f07f-4a03-ba49-f310f25343a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-pbxdh"
Nov 03 20:44:00.836318 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:00.836291    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e07a89e4-f07f-4a03-ba49-f310f25343a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-pbxdh\" (UID: \"e07a89e4-f07f-4a03-ba49-f310f25343a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-pbxdh"
Nov 03 20:44:00.854252 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:00.854220    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcpwg\" (UniqueName: \"kubernetes.io/projected/e07a89e4-f07f-4a03-ba49-f310f25343a0-kube-api-access-wcpwg\") pod \"maas-api-revoke-sub-e2fc9c5b-pbxdh\" (UID: \"e07a89e4-f07f-4a03-ba49-f310f25343a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-pbxdh"
Nov 03 20:44:00.893272 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:00.893236    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-pbxdh"
Nov 03 20:44:01.188262 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:01.188233    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-pbxdh"]
Nov 03 20:44:01.189478 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:44:01.189455    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode07a89e4_f07f_4a03_ba49_f310f25343a0.slice/crio-16c4b413edd24d1fad04b1b605ae85406bd917c9c7bd0c1630874439346eb4cb WatchSource:0}: Error finding container 16c4b413edd24d1fad04b1b605ae85406bd917c9c7bd0c1630874439346eb4cb: Status 404 returned error can't find the container with id 16c4b413edd24d1fad04b1b605ae85406bd917c9c7bd0c1630874439346eb4cb
Nov 03 20:44:01.782925 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:01.782825    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-pbxdh" event={"ID":"e07a89e4-f07f-4a03-ba49-f310f25343a0","Type":"ContainerStarted","Data":"6e4efd932684956a5baa16178e0fed43d20483d3c9b924f8be5307c6c710adb4"}
Nov 03 20:44:01.782925 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:01.782886    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-pbxdh" event={"ID":"e07a89e4-f07f-4a03-ba49-f310f25343a0","Type":"ContainerStarted","Data":"16c4b413edd24d1fad04b1b605ae85406bd917c9c7bd0c1630874439346eb4cb"}
Nov 03 20:44:01.837717 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:01.837646    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-pbxdh" podStartSLOduration=1.8376279869999999 podStartE2EDuration="1.837627987s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:44:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:44:01.822798936 +0000 UTC m=+2375.975460813" watchObservedRunningTime="2026-11-03 20:44:01.837627987 +0000 UTC m=+2375.990289873"
Nov 03 20:44:02.789979 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:02.789943    5570 generic.go:356] "Generic (PLEG): container finished" podID="e07a89e4-f07f-4a03-ba49-f310f25343a0" containerID="6e4efd932684956a5baa16178e0fed43d20483d3c9b924f8be5307c6c710adb4" exitCode=0
Nov 03 20:44:02.790406 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:02.790012    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-pbxdh" event={"ID":"e07a89e4-f07f-4a03-ba49-f310f25343a0","Type":"ContainerDied","Data":"6e4efd932684956a5baa16178e0fed43d20483d3c9b924f8be5307c6c710adb4"}
Nov 03 20:44:03.934308 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:03.934279    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-pbxdh"
Nov 03 20:44:04.068506 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:04.068397    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e07a89e4-f07f-4a03-ba49-f310f25343a0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e07a89e4-f07f-4a03-ba49-f310f25343a0-maas-api-openshift-ca\") pod \"e07a89e4-f07f-4a03-ba49-f310f25343a0\" (UID: \"e07a89e4-f07f-4a03-ba49-f310f25343a0\") "
Nov 03 20:44:04.068506 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:04.068467    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e07a89e4-f07f-4a03-ba49-f310f25343a0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e07a89e4-f07f-4a03-ba49-f310f25343a0-maas-api-xks-ca\") pod \"e07a89e4-f07f-4a03-ba49-f310f25343a0\" (UID: \"e07a89e4-f07f-4a03-ba49-f310f25343a0\") "
Nov 03 20:44:04.068815 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:04.068577    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e07a89e4-f07f-4a03-ba49-f310f25343a0-kube-api-access-wcpwg\" (UniqueName: \"kubernetes.io/projected/e07a89e4-f07f-4a03-ba49-f310f25343a0-kube-api-access-wcpwg\") pod \"e07a89e4-f07f-4a03-ba49-f310f25343a0\" (UID: \"e07a89e4-f07f-4a03-ba49-f310f25343a0\") "
Nov 03 20:44:04.068903 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:04.068824    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e07a89e4-f07f-4a03-ba49-f310f25343a0-maas-api-openshift-ca" pod "e07a89e4-f07f-4a03-ba49-f310f25343a0" (UID: "e07a89e4-f07f-4a03-ba49-f310f25343a0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:44:04.069129 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:04.069091    5570 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e07a89e4-f07f-4a03-ba49-f310f25343a0-maas-api-openshift-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:44:04.070674 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:04.070646    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e07a89e4-f07f-4a03-ba49-f310f25343a0-maas-api-xks-ca" pod "e07a89e4-f07f-4a03-ba49-f310f25343a0" (UID: "e07a89e4-f07f-4a03-ba49-f310f25343a0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:44:04.070819 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:04.070788    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e07a89e4-f07f-4a03-ba49-f310f25343a0-kube-api-access-wcpwg" pod "e07a89e4-f07f-4a03-ba49-f310f25343a0" (UID: "e07a89e4-f07f-4a03-ba49-f310f25343a0"). InnerVolumeSpecName "kube-api-access-wcpwg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:44:04.170282 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:04.170234    5570 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e07a89e4-f07f-4a03-ba49-f310f25343a0-maas-api-xks-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:44:04.170282 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:04.170280    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wcpwg\" (UniqueName: \"kubernetes.io/projected/e07a89e4-f07f-4a03-ba49-f310f25343a0-kube-api-access-wcpwg\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:44:04.802486 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:04.802448    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-pbxdh"
Nov 03 20:44:04.802486 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:04.802471    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-pbxdh" event={"ID":"e07a89e4-f07f-4a03-ba49-f310f25343a0","Type":"ContainerDied","Data":"16c4b413edd24d1fad04b1b605ae85406bd917c9c7bd0c1630874439346eb4cb"}
Nov 03 20:44:04.802735 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:04.802560    5570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="16c4b413edd24d1fad04b1b605ae85406bd917c9c7bd0c1630874439346eb4cb"
Nov 03 20:44:10.928250 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:10.928214    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-qldvw"]
Nov 03 20:44:10.928917 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:10.928893    5570 cpu_manager.go:414] "containerMap: removing container" podUID="e07a89e4-f07f-4a03-ba49-f310f25343a0" containerName="revoke-keys"
Nov 03 20:44:10.929038 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:10.928920    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e07a89e4-f07f-4a03-ba49-f310f25343a0" containerName="revoke-keys"
Nov 03 20:44:10.929038 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:10.929011    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="e07a89e4-f07f-4a03-ba49-f310f25343a0" containerName="revoke-keys"
Nov 03 20:44:10.932425 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:10.932400    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-qldvw"
Nov 03 20:44:10.948161 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:10.948119    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-qldvw"]
Nov 03 20:44:10.958116 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:10.958087    5570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 20:44:10.958677 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:10.958649    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:44:10.958677 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:10.958667    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l7s5w\""
Nov 03 20:44:10.961631 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:10.961603    5570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 20:44:11.032175 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:11.032137    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9fa4c49-b441-489d-b635-b483a1eddeb0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-qldvw\" (UID: \"c9fa4c49-b441-489d-b635-b483a1eddeb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-qldvw"
Nov 03 20:44:11.032375 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:11.032254    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9fa4c49-b441-489d-b635-b483a1eddeb0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-qldvw\" (UID: \"c9fa4c49-b441-489d-b635-b483a1eddeb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-qldvw"
Nov 03 20:44:11.032375 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:11.032370    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsclx\" (UniqueName: \"kubernetes.io/projected/c9fa4c49-b441-489d-b635-b483a1eddeb0-kube-api-access-fsclx\") pod \"maas-api-revoke-sub-5a29efb3-qldvw\" (UID: \"c9fa4c49-b441-489d-b635-b483a1eddeb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-qldvw"
Nov 03 20:44:11.133544 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:11.133504    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsclx\" (UniqueName: \"kubernetes.io/projected/c9fa4c49-b441-489d-b635-b483a1eddeb0-kube-api-access-fsclx\") pod \"maas-api-revoke-sub-5a29efb3-qldvw\" (UID: \"c9fa4c49-b441-489d-b635-b483a1eddeb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-qldvw"
Nov 03 20:44:11.133744 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:11.133592    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9fa4c49-b441-489d-b635-b483a1eddeb0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-qldvw\" (UID: \"c9fa4c49-b441-489d-b635-b483a1eddeb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-qldvw"
Nov 03 20:44:11.133744 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:11.133620    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9fa4c49-b441-489d-b635-b483a1eddeb0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-qldvw\" (UID: \"c9fa4c49-b441-489d-b635-b483a1eddeb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-qldvw"
Nov 03 20:44:11.134336 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:11.134313    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9fa4c49-b441-489d-b635-b483a1eddeb0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-qldvw\" (UID: \"c9fa4c49-b441-489d-b635-b483a1eddeb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-qldvw"
Nov 03 20:44:11.136028 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:11.136009    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9fa4c49-b441-489d-b635-b483a1eddeb0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-qldvw\" (UID: \"c9fa4c49-b441-489d-b635-b483a1eddeb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-qldvw"
Nov 03 20:44:11.153544 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:11.153513    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsclx\" (UniqueName: \"kubernetes.io/projected/c9fa4c49-b441-489d-b635-b483a1eddeb0-kube-api-access-fsclx\") pod \"maas-api-revoke-sub-5a29efb3-qldvw\" (UID: \"c9fa4c49-b441-489d-b635-b483a1eddeb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-qldvw"
Nov 03 20:44:11.246743 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:11.246705    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-qldvw"
Nov 03 20:44:11.541418 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:11.541381    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-qldvw"]
Nov 03 20:44:11.541550 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:44:11.541461    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9fa4c49_b441_489d_b635_b483a1eddeb0.slice/crio-10723580b1cb5957afc3484347ec8439d82de812993ab080f6183ceefcc9666b WatchSource:0}: Error finding container 10723580b1cb5957afc3484347ec8439d82de812993ab080f6183ceefcc9666b: Status 404 returned error can't find the container with id 10723580b1cb5957afc3484347ec8439d82de812993ab080f6183ceefcc9666b
Nov 03 20:44:11.840511 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:11.840473    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-qldvw" event={"ID":"c9fa4c49-b441-489d-b635-b483a1eddeb0","Type":"ContainerStarted","Data":"10723580b1cb5957afc3484347ec8439d82de812993ab080f6183ceefcc9666b"}
Nov 03 20:44:11.968620 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:11.968581    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g69f8"]
Nov 03 20:44:12.851566 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:12.851468    5570 generic.go:356] "Generic (PLEG): container finished" podID="c9fa4c49-b441-489d-b635-b483a1eddeb0" containerID="634b2cb11b71e6851773e59803ad8e2dbfa102ab389c5fce3c723daacee82667" exitCode=0
Nov 03 20:44:12.851566 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:12.851516    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-qldvw" event={"ID":"c9fa4c49-b441-489d-b635-b483a1eddeb0","Type":"ContainerDied","Data":"634b2cb11b71e6851773e59803ad8e2dbfa102ab389c5fce3c723daacee82667"}
Nov 03 20:44:14.732152 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:14.732118    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-qldvw"
Nov 03 20:44:14.749387 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:14.748408    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9fa4c49-b441-489d-b635-b483a1eddeb0-kube-api-access-fsclx\" (UniqueName: \"kubernetes.io/projected/c9fa4c49-b441-489d-b635-b483a1eddeb0-kube-api-access-fsclx\") pod \"c9fa4c49-b441-489d-b635-b483a1eddeb0\" (UID: \"c9fa4c49-b441-489d-b635-b483a1eddeb0\") "
Nov 03 20:44:14.749387 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:14.748560    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9fa4c49-b441-489d-b635-b483a1eddeb0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9fa4c49-b441-489d-b635-b483a1eddeb0-maas-api-openshift-ca\") pod \"c9fa4c49-b441-489d-b635-b483a1eddeb0\" (UID: \"c9fa4c49-b441-489d-b635-b483a1eddeb0\") "
Nov 03 20:44:14.749387 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:14.748617    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c9fa4c49-b441-489d-b635-b483a1eddeb0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9fa4c49-b441-489d-b635-b483a1eddeb0-maas-api-xks-ca\") pod \"c9fa4c49-b441-489d-b635-b483a1eddeb0\" (UID: \"c9fa4c49-b441-489d-b635-b483a1eddeb0\") "
Nov 03 20:44:14.750363 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:14.750332    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9fa4c49-b441-489d-b635-b483a1eddeb0-maas-api-openshift-ca" pod "c9fa4c49-b441-489d-b635-b483a1eddeb0" (UID: "c9fa4c49-b441-489d-b635-b483a1eddeb0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:44:14.772045 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:14.768451    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9fa4c49-b441-489d-b635-b483a1eddeb0-kube-api-access-fsclx" pod "c9fa4c49-b441-489d-b635-b483a1eddeb0" (UID: "c9fa4c49-b441-489d-b635-b483a1eddeb0"). InnerVolumeSpecName "kube-api-access-fsclx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:44:14.772806 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:14.772755    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9fa4c49-b441-489d-b635-b483a1eddeb0-maas-api-xks-ca" pod "c9fa4c49-b441-489d-b635-b483a1eddeb0" (UID: "c9fa4c49-b441-489d-b635-b483a1eddeb0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:44:14.898867 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:14.874489    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fsclx\" (UniqueName: \"kubernetes.io/projected/c9fa4c49-b441-489d-b635-b483a1eddeb0-kube-api-access-fsclx\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:44:14.898867 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:14.879478    5570 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9fa4c49-b441-489d-b635-b483a1eddeb0-maas-api-openshift-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:44:14.898867 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:14.879517    5570 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9fa4c49-b441-489d-b635-b483a1eddeb0-maas-api-xks-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:44:14.898867 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:14.891646    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-qldvw" event={"ID":"c9fa4c49-b441-489d-b635-b483a1eddeb0","Type":"ContainerDied","Data":"10723580b1cb5957afc3484347ec8439d82de812993ab080f6183ceefcc9666b"}
Nov 03 20:44:14.898867 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:14.891712    5570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="10723580b1cb5957afc3484347ec8439d82de812993ab080f6183ceefcc9666b"
Nov 03 20:44:14.898867 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:14.892001    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-qldvw"
Nov 03 20:44:22.462176 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:22.462137    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g69f8"]
Nov 03 20:44:25.026042 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:25.026003    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-62krd"]
Nov 03 20:44:25.039188 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:25.039154    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-62krd"]
Nov 03 20:44:25.973351 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:25.973306    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="721644ce-8cbd-4514-ae21-586c296a6773" path="/var/lib/kubelet/pods/721644ce-8cbd-4514-ae21-586c296a6773/volumes"
Nov 03 20:44:27.228623 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:44:27.228584    5570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8bwhz" containerName="extractor"
Nov 03 20:44:29.590536 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:29.590497    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2tfnh"]
Nov 03 20:44:29.603818 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:29.603770    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2tfnh"]
Nov 03 20:44:29.637012 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:29.636974    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kdxp9"]
Nov 03 20:44:29.637432 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:29.637418    5570 cpu_manager.go:414] "containerMap: removing container" podUID="c9fa4c49-b441-489d-b635-b483a1eddeb0" containerName="revoke-keys"
Nov 03 20:44:29.637484 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:29.637434    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9fa4c49-b441-489d-b635-b483a1eddeb0" containerName="revoke-keys"
Nov 03 20:44:29.637535 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:29.637526    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9fa4c49-b441-489d-b635-b483a1eddeb0" containerName="revoke-keys"
Nov 03 20:44:29.641025 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:29.641001    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kdxp9"
Nov 03 20:44:29.656324 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:29.656295    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kdxp9"]
Nov 03 20:44:29.656764 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:29.656738    5570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 20:44:29.656946 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:29.656788    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:44:29.656946 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:29.656818    5570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 20:44:29.656946 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:29.656836    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l7s5w\""
Nov 03 20:44:29.742141 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:29.742094    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjvn9\" (UniqueName: \"kubernetes.io/projected/f7697bff-4483-4412-b34a-2f8d1f57299b-kube-api-access-tjvn9\") pod \"maas-api-revoke-sub-3d4156f1-kdxp9\" (UID: \"f7697bff-4483-4412-b34a-2f8d1f57299b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kdxp9"
Nov 03 20:44:29.742337 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:29.742161    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7697bff-4483-4412-b34a-2f8d1f57299b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-kdxp9\" (UID: \"f7697bff-4483-4412-b34a-2f8d1f57299b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kdxp9"
Nov 03 20:44:29.742337 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:29.742265    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7697bff-4483-4412-b34a-2f8d1f57299b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-kdxp9\" (UID: \"f7697bff-4483-4412-b34a-2f8d1f57299b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kdxp9"
Nov 03 20:44:29.843140 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:29.843044    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7697bff-4483-4412-b34a-2f8d1f57299b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-kdxp9\" (UID: \"f7697bff-4483-4412-b34a-2f8d1f57299b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kdxp9"
Nov 03 20:44:29.843140 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:29.843092    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7697bff-4483-4412-b34a-2f8d1f57299b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-kdxp9\" (UID: \"f7697bff-4483-4412-b34a-2f8d1f57299b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kdxp9"
Nov 03 20:44:29.843346 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:29.843163    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjvn9\" (UniqueName: \"kubernetes.io/projected/f7697bff-4483-4412-b34a-2f8d1f57299b-kube-api-access-tjvn9\") pod \"maas-api-revoke-sub-3d4156f1-kdxp9\" (UID: \"f7697bff-4483-4412-b34a-2f8d1f57299b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kdxp9"
Nov 03 20:44:29.843709 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:29.843681    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7697bff-4483-4412-b34a-2f8d1f57299b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-kdxp9\" (UID: \"f7697bff-4483-4412-b34a-2f8d1f57299b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kdxp9"
Nov 03 20:44:29.845623 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:29.845587    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7697bff-4483-4412-b34a-2f8d1f57299b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-kdxp9\" (UID: \"f7697bff-4483-4412-b34a-2f8d1f57299b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kdxp9"
Nov 03 20:44:29.864419 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:29.864381    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjvn9\" (UniqueName: \"kubernetes.io/projected/f7697bff-4483-4412-b34a-2f8d1f57299b-kube-api-access-tjvn9\") pod \"maas-api-revoke-sub-3d4156f1-kdxp9\" (UID: \"f7697bff-4483-4412-b34a-2f8d1f57299b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kdxp9"
Nov 03 20:44:29.956186 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:29.956140    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kdxp9"
Nov 03 20:44:29.963734 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:29.963705    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8212f373-2ed9-4518-9736-a02b538b7164" path="/var/lib/kubelet/pods/8212f373-2ed9-4518-9736-a02b538b7164/volumes"
Nov 03 20:44:30.261107 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:30.261061    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kdxp9"]
Nov 03 20:44:30.261909 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:44:30.261882    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7697bff_4483_4412_b34a_2f8d1f57299b.slice/crio-abebdf01521b00a989279398416703ffb286b1bdea27e795879e89162a028ecf WatchSource:0}: Error finding container abebdf01521b00a989279398416703ffb286b1bdea27e795879e89162a028ecf: Status 404 returned error can't find the container with id abebdf01521b00a989279398416703ffb286b1bdea27e795879e89162a028ecf
Nov 03 20:44:30.984864 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:30.984820    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kdxp9" event={"ID":"f7697bff-4483-4412-b34a-2f8d1f57299b","Type":"ContainerStarted","Data":"ea563f33d3b9a256ba021a4c8c61099219fd7b2b4a1d62f055c81084c6b32e4f"}
Nov 03 20:44:30.985282 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:30.984879    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kdxp9" event={"ID":"f7697bff-4483-4412-b34a-2f8d1f57299b","Type":"ContainerStarted","Data":"abebdf01521b00a989279398416703ffb286b1bdea27e795879e89162a028ecf"}
Nov 03 20:44:31.025455 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:31.025397    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kdxp9" podStartSLOduration=2.025381986 podStartE2EDuration="2.025381986s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:44:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:44:31.024056777 +0000 UTC m=+2405.176718663" watchObservedRunningTime="2026-11-03 20:44:31.025381986 +0000 UTC m=+2405.178043872"
Nov 03 20:44:31.992434 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:31.992392    5570 generic.go:356] "Generic (PLEG): container finished" podID="f7697bff-4483-4412-b34a-2f8d1f57299b" containerID="ea563f33d3b9a256ba021a4c8c61099219fd7b2b4a1d62f055c81084c6b32e4f" exitCode=0
Nov 03 20:44:31.993000 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:31.992478    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kdxp9" event={"ID":"f7697bff-4483-4412-b34a-2f8d1f57299b","Type":"ContainerDied","Data":"ea563f33d3b9a256ba021a4c8c61099219fd7b2b4a1d62f055c81084c6b32e4f"}
Nov 03 20:44:32.150984 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:32.150953    5570 scope.go:122] "RemoveContainer" containerID="5b66695b9eca6d2eeff4b82a563f532b79d89d3e60e41624cb3a5d6752d28d4b"
Nov 03 20:44:32.160138 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:32.160113    5570 scope.go:122] "RemoveContainer" containerID="692cceb18d53e33b40c86198829996e7618e74182139285f83968bf7b8263a99"
Nov 03 20:44:32.168825 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:32.168801    5570 scope.go:122] "RemoveContainer" containerID="ff0f070ec195c347623c77fe2142ec5efacfe5e7aab04f2ab730b1a463624952"
Nov 03 20:44:33.143238 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:33.143209    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kdxp9"
Nov 03 20:44:33.279054 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:33.278960    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f7697bff-4483-4412-b34a-2f8d1f57299b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7697bff-4483-4412-b34a-2f8d1f57299b-maas-api-xks-ca\") pod \"f7697bff-4483-4412-b34a-2f8d1f57299b\" (UID: \"f7697bff-4483-4412-b34a-2f8d1f57299b\") "
Nov 03 20:44:33.279225 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:33.279056    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7697bff-4483-4412-b34a-2f8d1f57299b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7697bff-4483-4412-b34a-2f8d1f57299b-maas-api-openshift-ca\") pod \"f7697bff-4483-4412-b34a-2f8d1f57299b\" (UID: \"f7697bff-4483-4412-b34a-2f8d1f57299b\") "
Nov 03 20:44:33.279225 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:33.279116    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7697bff-4483-4412-b34a-2f8d1f57299b-kube-api-access-tjvn9\" (UniqueName: \"kubernetes.io/projected/f7697bff-4483-4412-b34a-2f8d1f57299b-kube-api-access-tjvn9\") pod \"f7697bff-4483-4412-b34a-2f8d1f57299b\" (UID: \"f7697bff-4483-4412-b34a-2f8d1f57299b\") "
Nov 03 20:44:33.279412 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:33.279390    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7697bff-4483-4412-b34a-2f8d1f57299b-maas-api-openshift-ca" pod "f7697bff-4483-4412-b34a-2f8d1f57299b" (UID: "f7697bff-4483-4412-b34a-2f8d1f57299b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:44:33.281253 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:33.281221    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7697bff-4483-4412-b34a-2f8d1f57299b-maas-api-xks-ca" pod "f7697bff-4483-4412-b34a-2f8d1f57299b" (UID: "f7697bff-4483-4412-b34a-2f8d1f57299b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:44:33.281377 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:33.281331    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7697bff-4483-4412-b34a-2f8d1f57299b-kube-api-access-tjvn9" pod "f7697bff-4483-4412-b34a-2f8d1f57299b" (UID: "f7697bff-4483-4412-b34a-2f8d1f57299b"). InnerVolumeSpecName "kube-api-access-tjvn9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:44:33.380681 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:33.380637    5570 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7697bff-4483-4412-b34a-2f8d1f57299b-maas-api-xks-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:44:33.380681 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:33.380672    5570 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7697bff-4483-4412-b34a-2f8d1f57299b-maas-api-openshift-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:44:33.380681 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:33.380682    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjvn9\" (UniqueName: \"kubernetes.io/projected/f7697bff-4483-4412-b34a-2f8d1f57299b-kube-api-access-tjvn9\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:44:34.005156 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:34.005123    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kdxp9"
Nov 03 20:44:34.005156 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:34.005144    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kdxp9" event={"ID":"f7697bff-4483-4412-b34a-2f8d1f57299b","Type":"ContainerDied","Data":"abebdf01521b00a989279398416703ffb286b1bdea27e795879e89162a028ecf"}
Nov 03 20:44:34.005394 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:34.005193    5570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="abebdf01521b00a989279398416703ffb286b1bdea27e795879e89162a028ecf"
Nov 03 20:44:45.084292 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:45.084247    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-229f0a-openshift-default-d94c48585-9jfkr"]
Nov 03 20:44:45.085148 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:45.085116    5570 cpu_manager.go:414] "containerMap: removing container" podUID="f7697bff-4483-4412-b34a-2f8d1f57299b" containerName="revoke-keys"
Nov 03 20:44:45.085360 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:45.085335    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7697bff-4483-4412-b34a-2f8d1f57299b" containerName="revoke-keys"
Nov 03 20:44:45.085625 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:45.085598    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7697bff-4483-4412-b34a-2f8d1f57299b" containerName="revoke-keys"
Nov 03 20:44:45.089997 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:45.089968    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-229f0a-openshift-default-d94c48585-9jfkr"
Nov 03 20:44:45.109827 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:45.109794    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-229f0a-openshift-default-d94c48585-9jfkr"]
Nov 03 20:44:45.119793 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:45.119739    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-229f0a-openshift-default-dockercfg-mnzt8\""
Nov 03 20:44:45.200070 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:45.200025    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ef1c1f9-e02e-41e8-ba84-1f5bf0f31c53-credential-socket\") pod \"e2e-worker-single-main-229f0a-openshift-default-d94c48585-9jfkr\" (UID: \"1ef1c1f9-e02e-41e8-ba84-1f5bf0f31c53\") " pod="openshift-ingress/e2e-worker-single-main-229f0a-openshift-default-d94c48585-9jfkr"
Nov 03 20:44:45.200268 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:45.200073    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ef1c1f9-e02e-41e8-ba84-1f5bf0f31c53-istio-data\") pod \"e2e-worker-single-main-229f0a-openshift-default-d94c48585-9jfkr\" (UID: \"1ef1c1f9-e02e-41e8-ba84-1f5bf0f31c53\") " pod="openshift-ingress/e2e-worker-single-main-229f0a-openshift-default-d94c48585-9jfkr"
Nov 03 20:44:45.200268 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:45.200203    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ef1c1f9-e02e-41e8-ba84-1f5bf0f31c53-istio-envoy\") pod \"e2e-worker-single-main-229f0a-openshift-default-d94c48585-9jfkr\" (UID: \"1ef1c1f9-e02e-41e8-ba84-1f5bf0f31c53\") " pod="openshift-ingress/e2e-worker-single-main-229f0a-openshift-default-d94c48585-9jfkr"
Nov 03 20:44:45.200268 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:45.200238    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ef1c1f9-e02e-41e8-ba84-1f5bf0f31c53-istio-token\") pod \"e2e-worker-single-main-229f0a-openshift-default-d94c48585-9jfkr\" (UID: \"1ef1c1f9-e02e-41e8-ba84-1f5bf0f31c53\") " pod="openshift-ingress/e2e-worker-single-main-229f0a-openshift-default-d94c48585-9jfkr"
Nov 03 20:44:45.200395 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:45.200271    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ef1c1f9-e02e-41e8-ba84-1f5bf0f31c53-istiod-ca-cert\") pod \"e2e-worker-single-main-229f0a-openshift-default-d94c48585-9jfkr\" (UID: \"1ef1c1f9-e02e-41e8-ba84-1f5bf0f31c53\") " pod="openshift-ingress/e2e-worker-single-main-229f0a-openshift-default-d94c48585-9jfkr"
Nov 03 20:44:45.200395 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:45.200334    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ef1c1f9-e02e-41e8-ba84-1f5bf0f31c53-istio-podinfo\") pod \"e2e-worker-single-main-229f0a-openshift-default-d94c48585-9jfkr\" (UID: \"1ef1c1f9-e02e-41e8-ba84-1f5bf0f31c53\") " pod="openshift-ingress/e2e-worker-single-main-229f0a-openshift-default-d94c48585-9jfkr"
Nov 03 20:44:45.200481 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:45.200403    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ef1c1f9-e02e-41e8-ba84-1f5bf0f31c53-workload-socket\") pod \"e2e-worker-single-main-229f0a-openshift-default-d94c48585-9jfkr\" (UID: \"1ef1c1f9-e02e-41e8-ba84-1f5bf0f31c53\") " pod="openshift-ingress/e2e-worker-single-main-229f0a-openshift-default-d94c48585-9jfkr"
Nov 03 20:44:45.200481 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:45.200422    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ef1c1f9-e02e-41e8-ba84-1f5bf0f31c53-workload-certs\") pod \"e2e-worker-single-main-229f0a-openshift-default-d94c48585-9jfkr\" (UID: \"1ef1c1f9-e02e-41e8-ba84-1f5bf0f31c53\") " pod="openshift-ingress/e2e-worker-single-main-229f0a-openshift-default-d94c48585-9jfkr"
Nov 03 20:44:45.200481 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:45.200466    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbpkw\" (UniqueName: \"kubernetes.io/projected/1ef1c1f9-e02e-41e8-ba84-1f5bf0f31c53-kube-api-access-tbpkw\") pod \"e2e-worker-single-main-229f0a-openshift-default-d94c48585-9jfkr\" (UID: \"1ef1c1f9-e02e-41e8-ba84-1f5bf0f31c53\") " pod="openshift-ingress/e2e-worker-single-main-229f0a-openshift-default-d94c48585-9jfkr"
Nov 03 20:44:45.304412 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:45.304370    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ef1c1f9-e02e-41e8-ba84-1f5bf0f31c53-workload-certs\") pod \"e2e-worker-single-main-229f0a-openshift-default-d94c48585-9jfkr\" (UID: \"1ef1c1f9-e02e-41e8-ba84-1f5bf0f31c53\") " pod="openshift-ingress/e2e-worker-single-main-229f0a-openshift-default-d94c48585-9jfkr"
Nov 03 20:44:45.305236 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:45.305200    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbpkw\" (UniqueName: \"kubernetes.io/projected/1ef1c1f9-e02e-41e8-ba84-1f5bf0f31c53-kube-api-access-tbpkw\") pod \"e2e-worker-single-main-229f0a-openshift-default-d94c48585-9jfkr\" (UID: \"1ef1c1f9-e02e-41e8-ba84-1f5bf0f31c53\") " pod="openshift-ingress/e2e-worker-single-main-229f0a-openshift-default-d94c48585-9jfkr"
Nov 03 20:44:45.305236 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:45.305108    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ef1c1f9-e02e-41e8-ba84-1f5bf0f31c53-workload-certs\") pod \"e2e-worker-single-main-229f0a-openshift-default-d94c48585-9jfkr\" (UID: \"1ef1c1f9-e02e-41e8-ba84-1f5bf0f31c53\") " pod="openshift-ingress/e2e-worker-single-main-229f0a-openshift-default-d94c48585-9jfkr"
Nov 03 20:44:45.305461 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:45.305339    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ef1c1f9-e02e-41e8-ba84-1f5bf0f31c53-credential-socket\") pod \"e2e-worker-single-main-229f0a-openshift-default-d94c48585-9jfkr\" (UID: \"1ef1c1f9-e02e-41e8-ba84-1f5bf0f31c53\") " pod="openshift-ingress/e2e-worker-single-main-229f0a-openshift-default-d94c48585-9jfkr"
Nov 03 20:44:45.305461 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:45.305375    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ef1c1f9-e02e-41e8-ba84-1f5bf0f31c53-istio-data\") pod \"e2e-worker-single-main-229f0a-openshift-default-d94c48585-9jfkr\" (UID: \"1ef1c1f9-e02e-41e8-ba84-1f5bf0f31c53\") " pod="openshift-ingress/e2e-worker-single-main-229f0a-openshift-default-d94c48585-9jfkr"
Nov 03 20:44:45.305461 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:45.305447    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ef1c1f9-e02e-41e8-ba84-1f5bf0f31c53-istio-envoy\") pod \"e2e-worker-single-main-229f0a-openshift-default-d94c48585-9jfkr\" (UID: \"1ef1c1f9-e02e-41e8-ba84-1f5bf0f31c53\") " pod="openshift-ingress/e2e-worker-single-main-229f0a-openshift-default-d94c48585-9jfkr"
Nov 03 20:44:45.305636 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:45.305477    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ef1c1f9-e02e-41e8-ba84-1f5bf0f31c53-istio-token\") pod \"e2e-worker-single-main-229f0a-openshift-default-d94c48585-9jfkr\" (UID: \"1ef1c1f9-e02e-41e8-ba84-1f5bf0f31c53\") " pod="openshift-ingress/e2e-worker-single-main-229f0a-openshift-default-d94c48585-9jfkr"
Nov 03 20:44:45.305636 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:45.305507    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ef1c1f9-e02e-41e8-ba84-1f5bf0f31c53-istiod-ca-cert\") pod \"e2e-worker-single-main-229f0a-openshift-default-d94c48585-9jfkr\" (UID: \"1ef1c1f9-e02e-41e8-ba84-1f5bf0f31c53\") " pod="openshift-ingress/e2e-worker-single-main-229f0a-openshift-default-d94c48585-9jfkr"
Nov 03 20:44:45.305636 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:45.305572    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ef1c1f9-e02e-41e8-ba84-1f5bf0f31c53-istio-podinfo\") pod \"e2e-worker-single-main-229f0a-openshift-default-d94c48585-9jfkr\" (UID: \"1ef1c1f9-e02e-41e8-ba84-1f5bf0f31c53\") " pod="openshift-ingress/e2e-worker-single-main-229f0a-openshift-default-d94c48585-9jfkr"
Nov 03 20:44:45.305636 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:45.305619    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ef1c1f9-e02e-41e8-ba84-1f5bf0f31c53-workload-socket\") pod \"e2e-worker-single-main-229f0a-openshift-default-d94c48585-9jfkr\" (UID: \"1ef1c1f9-e02e-41e8-ba84-1f5bf0f31c53\") " pod="openshift-ingress/e2e-worker-single-main-229f0a-openshift-default-d94c48585-9jfkr"
Nov 03 20:44:45.305890 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:45.305833    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ef1c1f9-e02e-41e8-ba84-1f5bf0f31c53-credential-socket\") pod \"e2e-worker-single-main-229f0a-openshift-default-d94c48585-9jfkr\" (UID: \"1ef1c1f9-e02e-41e8-ba84-1f5bf0f31c53\") " pod="openshift-ingress/e2e-worker-single-main-229f0a-openshift-default-d94c48585-9jfkr"
Nov 03 20:44:45.306022 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:45.306001    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ef1c1f9-e02e-41e8-ba84-1f5bf0f31c53-workload-socket\") pod \"e2e-worker-single-main-229f0a-openshift-default-d94c48585-9jfkr\" (UID: \"1ef1c1f9-e02e-41e8-ba84-1f5bf0f31c53\") " pod="openshift-ingress/e2e-worker-single-main-229f0a-openshift-default-d94c48585-9jfkr"
Nov 03 20:44:45.306241 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:45.306221    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ef1c1f9-e02e-41e8-ba84-1f5bf0f31c53-istio-data\") pod \"e2e-worker-single-main-229f0a-openshift-default-d94c48585-9jfkr\" (UID: \"1ef1c1f9-e02e-41e8-ba84-1f5bf0f31c53\") " pod="openshift-ingress/e2e-worker-single-main-229f0a-openshift-default-d94c48585-9jfkr"
Nov 03 20:44:45.306569 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:45.306541    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ef1c1f9-e02e-41e8-ba84-1f5bf0f31c53-istiod-ca-cert\") pod \"e2e-worker-single-main-229f0a-openshift-default-d94c48585-9jfkr\" (UID: \"1ef1c1f9-e02e-41e8-ba84-1f5bf0f31c53\") " pod="openshift-ingress/e2e-worker-single-main-229f0a-openshift-default-d94c48585-9jfkr"
Nov 03 20:44:45.309672 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:45.309654    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ef1c1f9-e02e-41e8-ba84-1f5bf0f31c53-istio-podinfo\") pod \"e2e-worker-single-main-229f0a-openshift-default-d94c48585-9jfkr\" (UID: \"1ef1c1f9-e02e-41e8-ba84-1f5bf0f31c53\") " pod="openshift-ingress/e2e-worker-single-main-229f0a-openshift-default-d94c48585-9jfkr"
Nov 03 20:44:45.312311 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:45.312286    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ef1c1f9-e02e-41e8-ba84-1f5bf0f31c53-istio-envoy\") pod \"e2e-worker-single-main-229f0a-openshift-default-d94c48585-9jfkr\" (UID: \"1ef1c1f9-e02e-41e8-ba84-1f5bf0f31c53\") " pod="openshift-ingress/e2e-worker-single-main-229f0a-openshift-default-d94c48585-9jfkr"
Nov 03 20:44:45.325580 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:45.325547    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbpkw\" (UniqueName: \"kubernetes.io/projected/1ef1c1f9-e02e-41e8-ba84-1f5bf0f31c53-kube-api-access-tbpkw\") pod \"e2e-worker-single-main-229f0a-openshift-default-d94c48585-9jfkr\" (UID: \"1ef1c1f9-e02e-41e8-ba84-1f5bf0f31c53\") " pod="openshift-ingress/e2e-worker-single-main-229f0a-openshift-default-d94c48585-9jfkr"
Nov 03 20:44:45.326205 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:45.326182    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ef1c1f9-e02e-41e8-ba84-1f5bf0f31c53-istio-token\") pod \"e2e-worker-single-main-229f0a-openshift-default-d94c48585-9jfkr\" (UID: \"1ef1c1f9-e02e-41e8-ba84-1f5bf0f31c53\") " pod="openshift-ingress/e2e-worker-single-main-229f0a-openshift-default-d94c48585-9jfkr"
Nov 03 20:44:45.414157 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:45.414066    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-229f0a-openshift-default-d94c48585-9jfkr"
Nov 03 20:44:45.722924 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:45.722889    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-229f0a-openshift-default-d94c48585-9jfkr"]
Nov 03 20:44:45.723526 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:44:45.723501    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ef1c1f9_e02e_41e8_ba84_1f5bf0f31c53.slice/crio-accbae5c201580d5cb444dd63c0de29273b6fb474aa8555e95ce48be44504fb1 WatchSource:0}: Error finding container accbae5c201580d5cb444dd63c0de29273b6fb474aa8555e95ce48be44504fb1: Status 404 returned error can't find the container with id accbae5c201580d5cb444dd63c0de29273b6fb474aa8555e95ce48be44504fb1
Nov 03 20:44:45.725788 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:45.725745    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:44:45.725901 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:45.725826    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:44:45.725901 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:45.725856    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:44:46.069239 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:46.069148    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-229f0a-openshift-default-d94c48585-9jfkr" event={"ID":"1ef1c1f9-e02e-41e8-ba84-1f5bf0f31c53","Type":"ContainerStarted","Data":"5c34a85e9a943eaed5cbb5da874bca7ef8764c88549063a88686cff90882ec34"}
Nov 03 20:44:46.069239 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:46.069211    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-229f0a-openshift-default-d94c48585-9jfkr" event={"ID":"1ef1c1f9-e02e-41e8-ba84-1f5bf0f31c53","Type":"ContainerStarted","Data":"accbae5c201580d5cb444dd63c0de29273b6fb474aa8555e95ce48be44504fb1"}
Nov 03 20:44:46.113557 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:46.113495    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-229f0a-openshift-default-d94c48585-9jfkr" podStartSLOduration=1.113477902 podStartE2EDuration="1.113477902s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:44:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:44:46.110921036 +0000 UTC m=+2420.263582922" watchObservedRunningTime="2026-11-03 20:44:46.113477902 +0000 UTC m=+2420.266139788"
Nov 03 20:44:46.414467 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:46.414368    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-229f0a-openshift-default-d94c48585-9jfkr"
Nov 03 20:44:46.420373 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:46.420338    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-229f0a-openshift-default-d94c48585-9jfkr"
Nov 03 20:44:47.074954 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:47.074922    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-229f0a-openshift-default-d94c48585-9jfkr"
Nov 03 20:44:47.075930 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:47.075907    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-229f0a-openshift-default-d94c48585-9jfkr"
Nov 03 20:44:50.323689 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:50.323649    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-229f0a-5bf59kb62m"]
Nov 03 20:44:50.327598 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:50.327578    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-229f0a-5bf59kb62m"
Nov 03 20:44:50.342967 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:50.342934    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-229f0a-5bf59kb62m"]
Nov 03 20:44:50.354131 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:50.354097    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-67936329-dockercfg-h7zk5\""
Nov 03 20:44:50.354131 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:50.354097    5570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-229f0a\""
Nov 03 20:44:50.441009 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:50.440974    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-229f0a-7684d76d8znhfl"]
Nov 03 20:44:50.446445 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:50.446412    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-229f0a-7684d76d8znhfl"
Nov 03 20:44:50.462293 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:50.462250    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gh8rv\" (UniqueName: \"kubernetes.io/projected/1308f0c5-ae02-4373-bf57-b557de9c0da8-kube-api-access-gh8rv\") pod \"payload-pre-processing-e2e-worker-single-main-229f0a-5bf59kb62m\" (UID: \"1308f0c5-ae02-4373-bf57-b557de9c0da8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-229f0a-5bf59kb62m"
Nov 03 20:44:50.462583 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:50.462363    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1308f0c5-ae02-4373-bf57-b557de9c0da8-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-229f0a-5bf59kb62m\" (UID: \"1308f0c5-ae02-4373-bf57-b557de9c0da8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-229f0a-5bf59kb62m"
Nov 03 20:44:50.464001 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:50.463972    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-229f0a-7684d76d8znhfl"]
Nov 03 20:44:50.564013 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:50.563972    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffvlw\" (UniqueName: \"kubernetes.io/projected/af829965-4b63-42ac-82e8-f5e9c3e49bcc-kube-api-access-ffvlw\") pod \"payload-processing-e2e-worker-single-main-229f0a-7684d76d8znhfl\" (UID: \"af829965-4b63-42ac-82e8-f5e9c3e49bcc\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-229f0a-7684d76d8znhfl"
Nov 03 20:44:50.564206 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:50.564027    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af829965-4b63-42ac-82e8-f5e9c3e49bcc-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-229f0a-7684d76d8znhfl\" (UID: \"af829965-4b63-42ac-82e8-f5e9c3e49bcc\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-229f0a-7684d76d8znhfl"
Nov 03 20:44:50.564206 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:50.564169    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gh8rv\" (UniqueName: \"kubernetes.io/projected/1308f0c5-ae02-4373-bf57-b557de9c0da8-kube-api-access-gh8rv\") pod \"payload-pre-processing-e2e-worker-single-main-229f0a-5bf59kb62m\" (UID: \"1308f0c5-ae02-4373-bf57-b557de9c0da8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-229f0a-5bf59kb62m"
Nov 03 20:44:50.564282 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:50.564269    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1308f0c5-ae02-4373-bf57-b557de9c0da8-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-229f0a-5bf59kb62m\" (UID: \"1308f0c5-ae02-4373-bf57-b557de9c0da8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-229f0a-5bf59kb62m"
Nov 03 20:44:50.564993 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:50.564972    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1308f0c5-ae02-4373-bf57-b557de9c0da8-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-229f0a-5bf59kb62m\" (UID: \"1308f0c5-ae02-4373-bf57-b557de9c0da8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-229f0a-5bf59kb62m"
Nov 03 20:44:50.584718 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:50.584641    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gh8rv\" (UniqueName: \"kubernetes.io/projected/1308f0c5-ae02-4373-bf57-b557de9c0da8-kube-api-access-gh8rv\") pod \"payload-pre-processing-e2e-worker-single-main-229f0a-5bf59kb62m\" (UID: \"1308f0c5-ae02-4373-bf57-b557de9c0da8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-229f0a-5bf59kb62m"
Nov 03 20:44:50.642920 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:50.642880    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-229f0a-5bf59kb62m"
Nov 03 20:44:50.665619 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:50.665587    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffvlw\" (UniqueName: \"kubernetes.io/projected/af829965-4b63-42ac-82e8-f5e9c3e49bcc-kube-api-access-ffvlw\") pod \"payload-processing-e2e-worker-single-main-229f0a-7684d76d8znhfl\" (UID: \"af829965-4b63-42ac-82e8-f5e9c3e49bcc\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-229f0a-7684d76d8znhfl"
Nov 03 20:44:50.665770 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:50.665637    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af829965-4b63-42ac-82e8-f5e9c3e49bcc-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-229f0a-7684d76d8znhfl\" (UID: \"af829965-4b63-42ac-82e8-f5e9c3e49bcc\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-229f0a-7684d76d8znhfl"
Nov 03 20:44:50.666350 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:50.666330    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af829965-4b63-42ac-82e8-f5e9c3e49bcc-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-229f0a-7684d76d8znhfl\" (UID: \"af829965-4b63-42ac-82e8-f5e9c3e49bcc\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-229f0a-7684d76d8znhfl"
Nov 03 20:44:50.687193 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:50.686311    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffvlw\" (UniqueName: \"kubernetes.io/projected/af829965-4b63-42ac-82e8-f5e9c3e49bcc-kube-api-access-ffvlw\") pod \"payload-processing-e2e-worker-single-main-229f0a-7684d76d8znhfl\" (UID: \"af829965-4b63-42ac-82e8-f5e9c3e49bcc\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-229f0a-7684d76d8znhfl"
Nov 03 20:44:50.777963 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:50.777925    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-229f0a-7684d76d8znhfl"
Nov 03 20:44:50.943556 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:50.943524    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-229f0a-5bf59kb62m"]
Nov 03 20:44:50.944402 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:44:50.944373    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1308f0c5_ae02_4373_bf57_b557de9c0da8.slice/crio-9bf36fea3bbeb86396b1a917155833dad76d2ba2d6349130d724cd3d9688cac0 WatchSource:0}: Error finding container 9bf36fea3bbeb86396b1a917155833dad76d2ba2d6349130d724cd3d9688cac0: Status 404 returned error can't find the container with id 9bf36fea3bbeb86396b1a917155833dad76d2ba2d6349130d724cd3d9688cac0
Nov 03 20:44:51.086250 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:51.086171    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-229f0a-7684d76d8znhfl"]
Nov 03 20:44:51.086548 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:44:51.086524    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf829965_4b63_42ac_82e8_f5e9c3e49bcc.slice/crio-ca5dea995ba004e78aa6a23bdcc76cb7037d55132d0493746b3a77f4bbf3dd93 WatchSource:0}: Error finding container ca5dea995ba004e78aa6a23bdcc76cb7037d55132d0493746b3a77f4bbf3dd93: Status 404 returned error can't find the container with id ca5dea995ba004e78aa6a23bdcc76cb7037d55132d0493746b3a77f4bbf3dd93
Nov 03 20:44:51.099003 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:51.098973    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-229f0a-7684d76d8znhfl" event={"ID":"af829965-4b63-42ac-82e8-f5e9c3e49bcc","Type":"ContainerStarted","Data":"ca5dea995ba004e78aa6a23bdcc76cb7037d55132d0493746b3a77f4bbf3dd93"}
Nov 03 20:44:51.100787 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:51.100747    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-229f0a-5bf59kb62m" event={"ID":"1308f0c5-ae02-4373-bf57-b557de9c0da8","Type":"ContainerStarted","Data":"3812ac83953641c6027333ec9d9b67e6506328070971ee56dcad5faa3d859973"}
Nov 03 20:44:51.100922 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:51.100825    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-229f0a-5bf59kb62m" event={"ID":"1308f0c5-ae02-4373-bf57-b557de9c0da8","Type":"ContainerStarted","Data":"9bf36fea3bbeb86396b1a917155833dad76d2ba2d6349130d724cd3d9688cac0"}
Nov 03 20:44:51.100922 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:51.100877    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-229f0a-5bf59kb62m"
Nov 03 20:44:51.142160 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:51.142113    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-229f0a-5bf59kb62m" podStartSLOduration=1.142098527 podStartE2EDuration="1.142098527s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:44:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:44:51.140126658 +0000 UTC m=+2425.292788542" watchObservedRunningTime="2026-11-03 20:44:51.142098527 +0000 UTC m=+2425.294760413"
Nov 03 20:44:52.107941 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:52.107896    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-229f0a-7684d76d8znhfl" event={"ID":"af829965-4b63-42ac-82e8-f5e9c3e49bcc","Type":"ContainerStarted","Data":"f752b46e884f2916c3fd122da9dbb1c0f2a41c7ca2e7f3f89c36715ad89807f4"}
Nov 03 20:44:52.108426 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:52.108211    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-229f0a-7684d76d8znhfl"
Nov 03 20:44:52.149845 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:52.149763    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-229f0a-7684d76d8znhfl" podStartSLOduration=2.1497476620000002 podStartE2EDuration="2.149747662s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:44:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:44:52.147169895 +0000 UTC m=+2426.299831784" watchObservedRunningTime="2026-11-03 20:44:52.149747662 +0000 UTC m=+2426.302409548"
Nov 03 20:44:54.135236 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:44:54.135198    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g69f8"]
Nov 03 20:44:57.228894 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:44:57.228856    5570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8bwhz" containerName="extractor"
Nov 03 20:45:00.158800 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:00.146660    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-229f0a-298956467ctx"]
Nov 03 20:45:00.163644 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:00.163613    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-229f0a-298956467ctx"
Nov 03 20:45:00.175989 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:00.175953    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-229f0a-298956467ctx"]
Nov 03 20:45:00.188738 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:00.188696    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkl5l\" (UniqueName: \"kubernetes.io/projected/9920f3ed-7cc7-44c9-9677-3fff18412fbb-kube-api-access-jkl5l\") pod \"maas-api-key-cleanup-e2e-worker-single-main-229f0a-298956467ctx\" (UID: \"9920f3ed-7cc7-44c9-9677-3fff18412fbb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-229f0a-298956467ctx"
Nov 03 20:45:00.188953 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:00.188888    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9920f3ed-7cc7-44c9-9677-3fff18412fbb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-single-main-229f0a-298956467ctx\" (UID: \"9920f3ed-7cc7-44c9-9677-3fff18412fbb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-229f0a-298956467ctx"
Nov 03 20:45:00.263089 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:00.263053    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l7s5w\""
Nov 03 20:45:00.263540 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:00.263519    5570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 20:45:00.263540 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:00.263520    5570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 20:45:00.289447 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:00.289407    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9920f3ed-7cc7-44c9-9677-3fff18412fbb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-single-main-229f0a-298956467ctx\" (UID: \"9920f3ed-7cc7-44c9-9677-3fff18412fbb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-229f0a-298956467ctx"
Nov 03 20:45:00.289610 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:00.289486    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkl5l\" (UniqueName: \"kubernetes.io/projected/9920f3ed-7cc7-44c9-9677-3fff18412fbb-kube-api-access-jkl5l\") pod \"maas-api-key-cleanup-e2e-worker-single-main-229f0a-298956467ctx\" (UID: \"9920f3ed-7cc7-44c9-9677-3fff18412fbb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-229f0a-298956467ctx"
Nov 03 20:45:00.290364 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:00.290337    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9920f3ed-7cc7-44c9-9677-3fff18412fbb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-single-main-229f0a-298956467ctx\" (UID: \"9920f3ed-7cc7-44c9-9677-3fff18412fbb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-229f0a-298956467ctx"
Nov 03 20:45:00.310293 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:00.310254    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkl5l\" (UniqueName: \"kubernetes.io/projected/9920f3ed-7cc7-44c9-9677-3fff18412fbb-kube-api-access-jkl5l\") pod \"maas-api-key-cleanup-e2e-worker-single-main-229f0a-298956467ctx\" (UID: \"9920f3ed-7cc7-44c9-9677-3fff18412fbb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-229f0a-298956467ctx"
Nov 03 20:45:00.485594 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:00.485553    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-229f0a-298956467ctx"
Nov 03 20:45:00.787571 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:45:00.787539    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9920f3ed_7cc7_44c9_9677_3fff18412fbb.slice/crio-88f83f518210deb7e07fc45c5c58d06bdb06c445ba0e8fff85b96b045d104f6f WatchSource:0}: Error finding container 88f83f518210deb7e07fc45c5c58d06bdb06c445ba0e8fff85b96b045d104f6f: Status 404 returned error can't find the container with id 88f83f518210deb7e07fc45c5c58d06bdb06c445ba0e8fff85b96b045d104f6f
Nov 03 20:45:00.787571 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:00.787572    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-229f0a-298956467ctx"]
Nov 03 20:45:01.207690 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:01.207651    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-229f0a-298956467ctx" event={"ID":"9920f3ed-7cc7-44c9-9677-3fff18412fbb","Type":"ContainerStarted","Data":"6786279c64375a1f7c7ab83ee88f13384000bed4a78a1dd4650361782acd7267"}
Nov 03 20:45:01.208173 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:01.207711    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-229f0a-298956467ctx" event={"ID":"9920f3ed-7cc7-44c9-9677-3fff18412fbb","Type":"ContainerStarted","Data":"88f83f518210deb7e07fc45c5c58d06bdb06c445ba0e8fff85b96b045d104f6f"}
Nov 03 20:45:01.250540 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:01.250468    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-229f0a-298956467ctx" podStartSLOduration=1.250447505 podStartE2EDuration="1.250447505s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:45:01.248105435 +0000 UTC m=+2435.400767323" watchObservedRunningTime="2026-11-03 20:45:01.250447505 +0000 UTC m=+2435.403109596"
Nov 03 20:45:02.111643 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:02.111611    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-229f0a-5bf59kb62m"
Nov 03 20:45:02.217738 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:02.217705    5570 generic.go:356] "Generic (PLEG): container finished" podID="9920f3ed-7cc7-44c9-9677-3fff18412fbb" containerID="6786279c64375a1f7c7ab83ee88f13384000bed4a78a1dd4650361782acd7267" exitCode=60
Nov 03 20:45:02.218279 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:02.217827    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-229f0a-298956467ctx" event={"ID":"9920f3ed-7cc7-44c9-9677-3fff18412fbb","Type":"ContainerDied","Data":"6786279c64375a1f7c7ab83ee88f13384000bed4a78a1dd4650361782acd7267"}
Nov 03 20:45:02.218279 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:02.218145    5570 scope.go:122] "RemoveContainer" containerID="6786279c64375a1f7c7ab83ee88f13384000bed4a78a1dd4650361782acd7267"
Nov 03 20:45:03.118238 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:03.118204    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-229f0a-7684d76d8znhfl"
Nov 03 20:45:03.118897 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:03.118869    5570 patch_prober.go:29] interesting pod/e2e-worker-single-main-229f0a-openshift-default-d94c48585-9jfkr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.205:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:45:03.119032 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:03.118916    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-229f0a-openshift-default-d94c48585-9jfkr" podUID="1ef1c1f9-e02e-41e8-ba84-1f5bf0f31c53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.205:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:45:03.229001 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:03.228965    5570 generic.go:356] "Generic (PLEG): container finished" podID="9920f3ed-7cc7-44c9-9677-3fff18412fbb" containerID="ae3f6c076e149f21096b52c044cecb03ef3e7f9a2b5911463a7463987b07fdf8" exitCode=60
Nov 03 20:45:03.229463 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:03.229064    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-229f0a-298956467ctx" event={"ID":"9920f3ed-7cc7-44c9-9677-3fff18412fbb","Type":"ContainerDied","Data":"ae3f6c076e149f21096b52c044cecb03ef3e7f9a2b5911463a7463987b07fdf8"}
Nov 03 20:45:03.229463 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:03.229189    5570 scope.go:122] "RemoveContainer" containerID="6786279c64375a1f7c7ab83ee88f13384000bed4a78a1dd4650361782acd7267"
Nov 03 20:45:03.229565 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:03.229495    5570 scope.go:122] "RemoveContainer" containerID="ae3f6c076e149f21096b52c044cecb03ef3e7f9a2b5911463a7463987b07fdf8"
Nov 03 20:45:03.229947 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:45:03.229911    5570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-single-main-229f0a-298956467ctx_odh-ai-gateway-infra(9920f3ed-7cc7-44c9-9677-3fff18412fbb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-229f0a-298956467ctx" podUID="9920f3ed-7cc7-44c9-9677-3fff18412fbb"
Nov 03 20:45:04.236369 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:04.236335    5570 generic.go:356] "Generic (PLEG): container finished" podID="1ef1c1f9-e02e-41e8-ba84-1f5bf0f31c53" containerID="5c34a85e9a943eaed5cbb5da874bca7ef8764c88549063a88686cff90882ec34" exitCode=0
Nov 03 20:45:04.236974 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:04.236411    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-229f0a-openshift-default-d94c48585-9jfkr" event={"ID":"1ef1c1f9-e02e-41e8-ba84-1f5bf0f31c53","Type":"ContainerDied","Data":"5c34a85e9a943eaed5cbb5da874bca7ef8764c88549063a88686cff90882ec34"}
Nov 03 20:45:04.236974 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:04.236957    5570 scope.go:122] "RemoveContainer" containerID="5c34a85e9a943eaed5cbb5da874bca7ef8764c88549063a88686cff90882ec34"
Nov 03 20:45:04.238341 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:04.238306    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:45:04.238449 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:04.238364    5570 scope.go:122] "RemoveContainer" containerID="ae3f6c076e149f21096b52c044cecb03ef3e7f9a2b5911463a7463987b07fdf8"
Nov 03 20:45:04.238449 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:04.238380    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:45:04.238449 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:04.238414    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:45:04.238567 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:45:04.238549    5570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-single-main-229f0a-298956467ctx_odh-ai-gateway-infra(9920f3ed-7cc7-44c9-9677-3fff18412fbb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-229f0a-298956467ctx" podUID="9920f3ed-7cc7-44c9-9677-3fff18412fbb"
Nov 03 20:45:05.268673 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:05.268631    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-229f0a-openshift-default-d94c48585-9jfkr" event={"ID":"1ef1c1f9-e02e-41e8-ba84-1f5bf0f31c53","Type":"ContainerStarted","Data":"fdf362554e7d1a5be0ffd71d8ba5fc90d760767a51faa5b4ca1cf144f1bda66d"}
Nov 03 20:45:05.414650 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:05.414610    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-229f0a-openshift-default-d94c48585-9jfkr"
Nov 03 20:45:05.415002 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:05.414974    5570 patch_prober.go:29] interesting pod/e2e-worker-single-main-229f0a-openshift-default-d94c48585-9jfkr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" start-of-body=
Nov 03 20:45:05.415063 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:05.415027    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-229f0a-openshift-default-d94c48585-9jfkr" podUID="1ef1c1f9-e02e-41e8-ba84-1f5bf0f31c53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused"
Nov 03 20:45:06.415375 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:06.415333    5570 patch_prober.go:29] interesting pod/e2e-worker-single-main-229f0a-openshift-default-d94c48585-9jfkr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" start-of-body=
Nov 03 20:45:06.415858 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:06.415399    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-229f0a-openshift-default-d94c48585-9jfkr" podUID="1ef1c1f9-e02e-41e8-ba84-1f5bf0f31c53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused"
Nov 03 20:45:07.415016 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:07.414933    5570 patch_prober.go:29] interesting pod/e2e-worker-single-main-229f0a-openshift-default-d94c48585-9jfkr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" start-of-body=
Nov 03 20:45:07.415016 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:07.414983    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-229f0a-openshift-default-d94c48585-9jfkr" podUID="1ef1c1f9-e02e-41e8-ba84-1f5bf0f31c53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused"
Nov 03 20:45:08.414620 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:08.414572    5570 patch_prober.go:29] interesting pod/e2e-worker-single-main-229f0a-openshift-default-d94c48585-9jfkr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" start-of-body=
Nov 03 20:45:08.415058 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:08.414640    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-229f0a-openshift-default-d94c48585-9jfkr" podUID="1ef1c1f9-e02e-41e8-ba84-1f5bf0f31c53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused"
Nov 03 20:45:08.657856 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:08.656761    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g69f8"]
Nov 03 20:45:09.414546 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:09.414511    5570 patch_prober.go:29] interesting pod/e2e-worker-single-main-229f0a-openshift-default-d94c48585-9jfkr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" start-of-body=
Nov 03 20:45:09.414759 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:09.414570    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-229f0a-openshift-default-d94c48585-9jfkr" podUID="1ef1c1f9-e02e-41e8-ba84-1f5bf0f31c53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused"
Nov 03 20:45:10.693475 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:10.693437    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-229f0a-openshift-default-d94c48585-9jfkr"
Nov 03 20:45:10.694236 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:10.694204    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-229f0a-openshift-default-d94c48585-9jfkr"
Nov 03 20:45:10.695281 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:10.695257    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-229f0a-openshift-default-d94c48585-9jfkr"
Nov 03 20:45:13.958249 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:13.958216    5570 scope.go:122] "RemoveContainer" containerID="ae3f6c076e149f21096b52c044cecb03ef3e7f9a2b5911463a7463987b07fdf8"
Nov 03 20:45:14.329139 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:14.329082    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-229f0a-298956467ctx" event={"ID":"9920f3ed-7cc7-44c9-9677-3fff18412fbb","Type":"ContainerStarted","Data":"5e493a6f9aff2a895a088e4da0924f29009a5ae5ca402d53c433106bfa107e27"}
Nov 03 20:45:15.336962 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:15.336927    5570 generic.go:356] "Generic (PLEG): container finished" podID="9920f3ed-7cc7-44c9-9677-3fff18412fbb" containerID="5e493a6f9aff2a895a088e4da0924f29009a5ae5ca402d53c433106bfa107e27" exitCode=60
Nov 03 20:45:15.337428 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:15.337003    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-229f0a-298956467ctx" event={"ID":"9920f3ed-7cc7-44c9-9677-3fff18412fbb","Type":"ContainerDied","Data":"5e493a6f9aff2a895a088e4da0924f29009a5ae5ca402d53c433106bfa107e27"}
Nov 03 20:45:15.337428 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:15.337063    5570 scope.go:122] "RemoveContainer" containerID="ae3f6c076e149f21096b52c044cecb03ef3e7f9a2b5911463a7463987b07fdf8"
Nov 03 20:45:15.337539 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:15.337452    5570 scope.go:122] "RemoveContainer" containerID="5e493a6f9aff2a895a088e4da0924f29009a5ae5ca402d53c433106bfa107e27"
Nov 03 20:45:15.337819 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:45:15.337762    5570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-single-main-229f0a-298956467ctx_odh-ai-gateway-infra(9920f3ed-7cc7-44c9-9677-3fff18412fbb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-229f0a-298956467ctx" podUID="9920f3ed-7cc7-44c9-9677-3fff18412fbb"
Nov 03 20:45:15.383828 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:15.383791    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-229f0a-298956467ctx"]
Nov 03 20:45:16.484560 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:16.484531    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-229f0a-298956467ctx"
Nov 03 20:45:16.595988 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:16.595874    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9920f3ed-7cc7-44c9-9677-3fff18412fbb-kube-api-access-jkl5l\" (UniqueName: \"kubernetes.io/projected/9920f3ed-7cc7-44c9-9677-3fff18412fbb-kube-api-access-jkl5l\") pod \"9920f3ed-7cc7-44c9-9677-3fff18412fbb\" (UID: \"9920f3ed-7cc7-44c9-9677-3fff18412fbb\") "
Nov 03 20:45:16.595988 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:16.595943    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9920f3ed-7cc7-44c9-9677-3fff18412fbb-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9920f3ed-7cc7-44c9-9677-3fff18412fbb-maas-api-service-ca\") pod \"9920f3ed-7cc7-44c9-9677-3fff18412fbb\" (UID: \"9920f3ed-7cc7-44c9-9677-3fff18412fbb\") "
Nov 03 20:45:16.596372 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:16.596343    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9920f3ed-7cc7-44c9-9677-3fff18412fbb-maas-api-service-ca" pod "9920f3ed-7cc7-44c9-9677-3fff18412fbb" (UID: "9920f3ed-7cc7-44c9-9677-3fff18412fbb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:45:16.598179 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:16.598147    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9920f3ed-7cc7-44c9-9677-3fff18412fbb-kube-api-access-jkl5l" pod "9920f3ed-7cc7-44c9-9677-3fff18412fbb" (UID: "9920f3ed-7cc7-44c9-9677-3fff18412fbb"). InnerVolumeSpecName "kube-api-access-jkl5l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:45:16.696826 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:16.696765    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkl5l\" (UniqueName: \"kubernetes.io/projected/9920f3ed-7cc7-44c9-9677-3fff18412fbb-kube-api-access-jkl5l\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:45:16.696826 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:16.696818    5570 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9920f3ed-7cc7-44c9-9677-3fff18412fbb-maas-api-service-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:45:17.354315 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:17.354272    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-229f0a-298956467ctx" event={"ID":"9920f3ed-7cc7-44c9-9677-3fff18412fbb","Type":"ContainerDied","Data":"88f83f518210deb7e07fc45c5c58d06bdb06c445ba0e8fff85b96b045d104f6f"}
Nov 03 20:45:17.354537 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:17.354342    5570 scope.go:122] "RemoveContainer" containerID="5e493a6f9aff2a895a088e4da0924f29009a5ae5ca402d53c433106bfa107e27"
Nov 03 20:45:17.354537 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:17.354298    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-229f0a-298956467ctx"
Nov 03 20:45:17.410946 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:17.410903    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-229f0a-298956467ctx"]
Nov 03 20:45:17.423829 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:17.423795    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-229f0a-298956467ctx"]
Nov 03 20:45:17.963580 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:17.963550    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9920f3ed-7cc7-44c9-9677-3fff18412fbb" path="/var/lib/kubelet/pods/9920f3ed-7cc7-44c9-9677-3fff18412fbb/volumes"
Nov 03 20:45:23.036264 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:23.036221    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-tlwq8"]
Nov 03 20:45:23.049524 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:23.049484    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-tlwq8"]
Nov 03 20:45:23.753235 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:23.753198    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g69f8"]
Nov 03 20:45:24.010225 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:24.010141    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07ea3e12-a2f0-496c-af2e-180c499058ef" path="/var/lib/kubelet/pods/07ea3e12-a2f0-496c-af2e-180c499058ef/volumes"
Nov 03 20:45:27.228424 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:45:27.228385    5570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8bwhz" containerName="extractor"
Nov 03 20:45:32.199662 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:32.199628    5570 scope.go:122] "RemoveContainer" containerID="3f35e269fd588cf53f7e120fc53f6cb2d495d5c1d02046acc4d72e1bd36e3480"
Nov 03 20:45:32.208649 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:32.208623    5570 scope.go:122] "RemoveContainer" containerID="ad57d654eb4ed4dcb230c25d2971137825f128848dba78ebe27ed2618b8ad233"
Nov 03 20:45:37.273550 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:37.273508    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd9d743c-zb28r"]
Nov 03 20:45:37.274083 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:37.273989    5570 cpu_manager.go:414] "containerMap: removing container" podUID="9920f3ed-7cc7-44c9-9677-3fff18412fbb" containerName="cleanup"
Nov 03 20:45:37.274083 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:37.274001    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9920f3ed-7cc7-44c9-9677-3fff18412fbb" containerName="cleanup"
Nov 03 20:45:37.274083 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:37.274024    5570 cpu_manager.go:414] "containerMap: removing container" podUID="9920f3ed-7cc7-44c9-9677-3fff18412fbb" containerName="cleanup"
Nov 03 20:45:37.274083 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:37.274030    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9920f3ed-7cc7-44c9-9677-3fff18412fbb" containerName="cleanup"
Nov 03 20:45:37.274222 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:37.274106    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="9920f3ed-7cc7-44c9-9677-3fff18412fbb" containerName="cleanup"
Nov 03 20:45:37.274222 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:37.274115    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="9920f3ed-7cc7-44c9-9677-3fff18412fbb" containerName="cleanup"
Nov 03 20:45:37.274222 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:37.274123    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="9920f3ed-7cc7-44c9-9677-3fff18412fbb" containerName="cleanup"
Nov 03 20:45:37.277332 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:37.277312    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd9d743c-zb28r"
Nov 03 20:45:37.293635 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:37.293603    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd9d743c-zb28r"]
Nov 03 20:45:37.304184 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:37.304152    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:45:37.304386 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:37.304189    5570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 20:45:37.304386 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:37.304236    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l7s5w\""
Nov 03 20:45:37.306362 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:37.306341    5570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 20:45:37.345020 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:37.344980    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81beac03-bfcc-4f4b-8ed8-3c6672dd4782-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd9d743c-zb28r\" (UID: \"81beac03-bfcc-4f4b-8ed8-3c6672dd4782\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd9d743c-zb28r"
Nov 03 20:45:37.345020 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:37.345025    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgzr2\" (UniqueName: \"kubernetes.io/projected/81beac03-bfcc-4f4b-8ed8-3c6672dd4782-kube-api-access-tgzr2\") pod \"maas-api-revoke-sub-cd9d743c-zb28r\" (UID: \"81beac03-bfcc-4f4b-8ed8-3c6672dd4782\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd9d743c-zb28r"
Nov 03 20:45:37.345266 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:37.345126    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81beac03-bfcc-4f4b-8ed8-3c6672dd4782-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd9d743c-zb28r\" (UID: \"81beac03-bfcc-4f4b-8ed8-3c6672dd4782\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd9d743c-zb28r"
Nov 03 20:45:37.445788 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:37.445747    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81beac03-bfcc-4f4b-8ed8-3c6672dd4782-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd9d743c-zb28r\" (UID: \"81beac03-bfcc-4f4b-8ed8-3c6672dd4782\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd9d743c-zb28r"
Nov 03 20:45:37.445996 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:37.445821    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgzr2\" (UniqueName: \"kubernetes.io/projected/81beac03-bfcc-4f4b-8ed8-3c6672dd4782-kube-api-access-tgzr2\") pod \"maas-api-revoke-sub-cd9d743c-zb28r\" (UID: \"81beac03-bfcc-4f4b-8ed8-3c6672dd4782\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd9d743c-zb28r"
Nov 03 20:45:37.445996 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:37.445882    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81beac03-bfcc-4f4b-8ed8-3c6672dd4782-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd9d743c-zb28r\" (UID: \"81beac03-bfcc-4f4b-8ed8-3c6672dd4782\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd9d743c-zb28r"
Nov 03 20:45:37.446563 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:37.446527    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81beac03-bfcc-4f4b-8ed8-3c6672dd4782-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd9d743c-zb28r\" (UID: \"81beac03-bfcc-4f4b-8ed8-3c6672dd4782\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd9d743c-zb28r"
Nov 03 20:45:37.448687 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:37.448660    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81beac03-bfcc-4f4b-8ed8-3c6672dd4782-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd9d743c-zb28r\" (UID: \"81beac03-bfcc-4f4b-8ed8-3c6672dd4782\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd9d743c-zb28r"
Nov 03 20:45:37.466265 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:37.466237    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgzr2\" (UniqueName: \"kubernetes.io/projected/81beac03-bfcc-4f4b-8ed8-3c6672dd4782-kube-api-access-tgzr2\") pod \"maas-api-revoke-sub-cd9d743c-zb28r\" (UID: \"81beac03-bfcc-4f4b-8ed8-3c6672dd4782\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd9d743c-zb28r"
Nov 03 20:45:37.589822 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:37.589706    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd9d743c-zb28r"
Nov 03 20:45:37.883219 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:37.883186    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd9d743c-zb28r"]
Nov 03 20:45:37.884669 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:45:37.884640    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81beac03_bfcc_4f4b_8ed8_3c6672dd4782.slice/crio-68c3f28381a3f4a4ef02295ce2ac72688af309bba3691afeff648e8e41d3eef5 WatchSource:0}: Error finding container 68c3f28381a3f4a4ef02295ce2ac72688af309bba3691afeff648e8e41d3eef5: Status 404 returned error can't find the container with id 68c3f28381a3f4a4ef02295ce2ac72688af309bba3691afeff648e8e41d3eef5
Nov 03 20:45:38.487650 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:38.487610    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd9d743c-zb28r" event={"ID":"81beac03-bfcc-4f4b-8ed8-3c6672dd4782","Type":"ContainerStarted","Data":"8d6f14cc58f2af1a5a47d0e79d092137f2e24702e6c1317bab260ada50eb49d4"}
Nov 03 20:45:38.488176 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:38.487668    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd9d743c-zb28r" event={"ID":"81beac03-bfcc-4f4b-8ed8-3c6672dd4782","Type":"ContainerStarted","Data":"68c3f28381a3f4a4ef02295ce2ac72688af309bba3691afeff648e8e41d3eef5"}
Nov 03 20:45:38.529050 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:38.528995    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd9d743c-zb28r" podStartSLOduration=1.52897798 podStartE2EDuration="1.52897798s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:45:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:45:38.527148089 +0000 UTC m=+2472.679809975" watchObservedRunningTime="2026-11-03 20:45:38.52897798 +0000 UTC m=+2472.681639866"
Nov 03 20:45:39.495139 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:39.495103    5570 generic.go:356] "Generic (PLEG): container finished" podID="81beac03-bfcc-4f4b-8ed8-3c6672dd4782" containerID="8d6f14cc58f2af1a5a47d0e79d092137f2e24702e6c1317bab260ada50eb49d4" exitCode=0
Nov 03 20:45:39.495595 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:39.495177    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd9d743c-zb28r" event={"ID":"81beac03-bfcc-4f4b-8ed8-3c6672dd4782","Type":"ContainerDied","Data":"8d6f14cc58f2af1a5a47d0e79d092137f2e24702e6c1317bab260ada50eb49d4"}
Nov 03 20:45:40.641366 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:40.641335    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd9d743c-zb28r"
Nov 03 20:45:40.679973 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:40.679917    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/81beac03-bfcc-4f4b-8ed8-3c6672dd4782-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81beac03-bfcc-4f4b-8ed8-3c6672dd4782-maas-api-xks-ca\") pod \"81beac03-bfcc-4f4b-8ed8-3c6672dd4782\" (UID: \"81beac03-bfcc-4f4b-8ed8-3c6672dd4782\") "
Nov 03 20:45:40.680186 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:40.679986    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81beac03-bfcc-4f4b-8ed8-3c6672dd4782-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81beac03-bfcc-4f4b-8ed8-3c6672dd4782-maas-api-openshift-ca\") pod \"81beac03-bfcc-4f4b-8ed8-3c6672dd4782\" (UID: \"81beac03-bfcc-4f4b-8ed8-3c6672dd4782\") "
Nov 03 20:45:40.680186 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:40.680029    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81beac03-bfcc-4f4b-8ed8-3c6672dd4782-kube-api-access-tgzr2\" (UniqueName: \"kubernetes.io/projected/81beac03-bfcc-4f4b-8ed8-3c6672dd4782-kube-api-access-tgzr2\") pod \"81beac03-bfcc-4f4b-8ed8-3c6672dd4782\" (UID: \"81beac03-bfcc-4f4b-8ed8-3c6672dd4782\") "
Nov 03 20:45:40.680353 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:40.680328    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81beac03-bfcc-4f4b-8ed8-3c6672dd4782-maas-api-openshift-ca" pod "81beac03-bfcc-4f4b-8ed8-3c6672dd4782" (UID: "81beac03-bfcc-4f4b-8ed8-3c6672dd4782"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:45:40.682358 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:40.682314    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81beac03-bfcc-4f4b-8ed8-3c6672dd4782-maas-api-xks-ca" pod "81beac03-bfcc-4f4b-8ed8-3c6672dd4782" (UID: "81beac03-bfcc-4f4b-8ed8-3c6672dd4782"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:45:40.682481 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:40.682411    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81beac03-bfcc-4f4b-8ed8-3c6672dd4782-kube-api-access-tgzr2" pod "81beac03-bfcc-4f4b-8ed8-3c6672dd4782" (UID: "81beac03-bfcc-4f4b-8ed8-3c6672dd4782"). InnerVolumeSpecName "kube-api-access-tgzr2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:45:40.780975 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:40.780883    5570 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81beac03-bfcc-4f4b-8ed8-3c6672dd4782-maas-api-openshift-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:45:40.780975 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:40.780917    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tgzr2\" (UniqueName: \"kubernetes.io/projected/81beac03-bfcc-4f4b-8ed8-3c6672dd4782-kube-api-access-tgzr2\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:45:40.780975 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:40.780928    5570 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81beac03-bfcc-4f4b-8ed8-3c6672dd4782-maas-api-xks-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:45:41.507214 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:41.507174    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd9d743c-zb28r" event={"ID":"81beac03-bfcc-4f4b-8ed8-3c6672dd4782","Type":"ContainerDied","Data":"68c3f28381a3f4a4ef02295ce2ac72688af309bba3691afeff648e8e41d3eef5"}
Nov 03 20:45:41.507214 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:41.507199    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd9d743c-zb28r"
Nov 03 20:45:41.507438 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:41.507228    5570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="68c3f28381a3f4a4ef02295ce2ac72688af309bba3691afeff648e8e41d3eef5"
Nov 03 20:45:45.026114 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:45.026075    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-tggds"]
Nov 03 20:45:45.039328 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:45.039291    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-tggds"]
Nov 03 20:45:45.964289 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:45.964252    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e708e575-b161-4cda-9e3b-1fc4b29994b6" path="/var/lib/kubelet/pods/e708e575-b161-4cda-9e3b-1fc4b29994b6/volumes"
Nov 03 20:45:48.533618 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:48.533573    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-229f0a-t5p4b"]
Nov 03 20:45:48.534315 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:48.534293    5570 cpu_manager.go:414] "containerMap: removing container" podUID="9920f3ed-7cc7-44c9-9677-3fff18412fbb" containerName="cleanup"
Nov 03 20:45:48.534427 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:48.534316    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9920f3ed-7cc7-44c9-9677-3fff18412fbb" containerName="cleanup"
Nov 03 20:45:48.534427 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:48.534357    5570 cpu_manager.go:414] "containerMap: removing container" podUID="81beac03-bfcc-4f4b-8ed8-3c6672dd4782" containerName="revoke-keys"
Nov 03 20:45:48.534427 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:48.534365    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81beac03-bfcc-4f4b-8ed8-3c6672dd4782" containerName="revoke-keys"
Nov 03 20:45:48.534588 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:48.534487    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="81beac03-bfcc-4f4b-8ed8-3c6672dd4782" containerName="revoke-keys"
Nov 03 20:45:48.538745 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:48.538720    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-229f0a-t5p4b"
Nov 03 20:45:48.552480 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:48.552451    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-229f0a-t5p4b"]
Nov 03 20:45:48.563826 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:48.563793    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:45:48.564063 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:48.564021    5570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 20:45:48.564147 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:48.564082    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l7s5w\""
Nov 03 20:45:48.567682 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:48.567642    5570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 20:45:48.669267 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:48.669229    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2dc7021-30c7-4b11-8108-8711ace2ea03-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-229f0a-t5p4b\" (UID: \"a2dc7021-30c7-4b11-8108-8711ace2ea03\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-229f0a-t5p4b"
Nov 03 20:45:48.669267 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:48.669273    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2dc7021-30c7-4b11-8108-8711ace2ea03-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-229f0a-t5p4b\" (UID: \"a2dc7021-30c7-4b11-8108-8711ace2ea03\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-229f0a-t5p4b"
Nov 03 20:45:48.669565 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:48.669375    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nm6zh\" (UniqueName: \"kubernetes.io/projected/a2dc7021-30c7-4b11-8108-8711ace2ea03-kube-api-access-nm6zh\") pod \"maas-api-revoke-keys-e2e-worker-single-main-229f0a-t5p4b\" (UID: \"a2dc7021-30c7-4b11-8108-8711ace2ea03\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-229f0a-t5p4b"
Nov 03 20:45:48.770963 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:48.770921    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2dc7021-30c7-4b11-8108-8711ace2ea03-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-229f0a-t5p4b\" (UID: \"a2dc7021-30c7-4b11-8108-8711ace2ea03\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-229f0a-t5p4b"
Nov 03 20:45:48.770963 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:48.770970    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2dc7021-30c7-4b11-8108-8711ace2ea03-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-229f0a-t5p4b\" (UID: \"a2dc7021-30c7-4b11-8108-8711ace2ea03\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-229f0a-t5p4b"
Nov 03 20:45:48.771191 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:48.771000    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nm6zh\" (UniqueName: \"kubernetes.io/projected/a2dc7021-30c7-4b11-8108-8711ace2ea03-kube-api-access-nm6zh\") pod \"maas-api-revoke-keys-e2e-worker-single-main-229f0a-t5p4b\" (UID: \"a2dc7021-30c7-4b11-8108-8711ace2ea03\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-229f0a-t5p4b"
Nov 03 20:45:48.771587 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:48.771565    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2dc7021-30c7-4b11-8108-8711ace2ea03-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-229f0a-t5p4b\" (UID: \"a2dc7021-30c7-4b11-8108-8711ace2ea03\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-229f0a-t5p4b"
Nov 03 20:45:48.773386 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:48.773359    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2dc7021-30c7-4b11-8108-8711ace2ea03-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-229f0a-t5p4b\" (UID: \"a2dc7021-30c7-4b11-8108-8711ace2ea03\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-229f0a-t5p4b"
Nov 03 20:45:48.791192 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:48.791107    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nm6zh\" (UniqueName: \"kubernetes.io/projected/a2dc7021-30c7-4b11-8108-8711ace2ea03-kube-api-access-nm6zh\") pod \"maas-api-revoke-keys-e2e-worker-single-main-229f0a-t5p4b\" (UID: \"a2dc7021-30c7-4b11-8108-8711ace2ea03\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-229f0a-t5p4b"
Nov 03 20:45:48.858444 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:48.858405    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-229f0a-t5p4b"
Nov 03 20:45:49.152975 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:49.152945    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-229f0a-t5p4b"]
Nov 03 20:45:49.154120 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:45:49.154083    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2dc7021_30c7_4b11_8108_8711ace2ea03.slice/crio-ff3c7029638c2e5e130f244353312db16897c8c091812079cb16a30b51c58b8e WatchSource:0}: Error finding container ff3c7029638c2e5e130f244353312db16897c8c091812079cb16a30b51c58b8e: Status 404 returned error can't find the container with id ff3c7029638c2e5e130f244353312db16897c8c091812079cb16a30b51c58b8e
Nov 03 20:45:49.554782 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:49.554736    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-229f0a-t5p4b" event={"ID":"a2dc7021-30c7-4b11-8108-8711ace2ea03","Type":"ContainerStarted","Data":"8611c520606be4969d35b1a72da4d6ca112b4e8d4f1777a323cd401810655b36"}
Nov 03 20:45:49.555198 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:49.554860    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-229f0a-t5p4b" event={"ID":"a2dc7021-30c7-4b11-8108-8711ace2ea03","Type":"ContainerStarted","Data":"ff3c7029638c2e5e130f244353312db16897c8c091812079cb16a30b51c58b8e"}
Nov 03 20:45:49.598164 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:49.598105    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-229f0a-t5p4b" podStartSLOduration=1.59809175 podStartE2EDuration="1.59809175s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:45:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:45:49.596756672 +0000 UTC m=+2483.749418561" watchObservedRunningTime="2026-11-03 20:45:49.59809175 +0000 UTC m=+2483.750753706"
Nov 03 20:45:49.784946 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:49.782059    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g69f8"]
Nov 03 20:45:50.563490 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:50.563449    5570 generic.go:356] "Generic (PLEG): container finished" podID="a2dc7021-30c7-4b11-8108-8711ace2ea03" containerID="8611c520606be4969d35b1a72da4d6ca112b4e8d4f1777a323cd401810655b36" exitCode=0
Nov 03 20:45:50.563982 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:50.563529    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-229f0a-t5p4b" event={"ID":"a2dc7021-30c7-4b11-8108-8711ace2ea03","Type":"ContainerDied","Data":"8611c520606be4969d35b1a72da4d6ca112b4e8d4f1777a323cd401810655b36"}
Nov 03 20:45:51.710232 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:51.710199    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-229f0a-t5p4b"
Nov 03 20:45:51.804262 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:51.804220    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a2dc7021-30c7-4b11-8108-8711ace2ea03-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2dc7021-30c7-4b11-8108-8711ace2ea03-maas-api-xks-ca\") pod \"a2dc7021-30c7-4b11-8108-8711ace2ea03\" (UID: \"a2dc7021-30c7-4b11-8108-8711ace2ea03\") "
Nov 03 20:45:51.804459 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:51.804324    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2dc7021-30c7-4b11-8108-8711ace2ea03-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2dc7021-30c7-4b11-8108-8711ace2ea03-maas-api-openshift-ca\") pod \"a2dc7021-30c7-4b11-8108-8711ace2ea03\" (UID: \"a2dc7021-30c7-4b11-8108-8711ace2ea03\") "
Nov 03 20:45:51.804459 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:51.804356    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2dc7021-30c7-4b11-8108-8711ace2ea03-kube-api-access-nm6zh\" (UniqueName: \"kubernetes.io/projected/a2dc7021-30c7-4b11-8108-8711ace2ea03-kube-api-access-nm6zh\") pod \"a2dc7021-30c7-4b11-8108-8711ace2ea03\" (UID: \"a2dc7021-30c7-4b11-8108-8711ace2ea03\") "
Nov 03 20:45:51.804792 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:51.804743    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2dc7021-30c7-4b11-8108-8711ace2ea03-maas-api-openshift-ca" pod "a2dc7021-30c7-4b11-8108-8711ace2ea03" (UID: "a2dc7021-30c7-4b11-8108-8711ace2ea03"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:45:51.806435 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:51.806406    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2dc7021-30c7-4b11-8108-8711ace2ea03-maas-api-xks-ca" pod "a2dc7021-30c7-4b11-8108-8711ace2ea03" (UID: "a2dc7021-30c7-4b11-8108-8711ace2ea03"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:45:51.806529 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:51.806453    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2dc7021-30c7-4b11-8108-8711ace2ea03-kube-api-access-nm6zh" pod "a2dc7021-30c7-4b11-8108-8711ace2ea03" (UID: "a2dc7021-30c7-4b11-8108-8711ace2ea03"). InnerVolumeSpecName "kube-api-access-nm6zh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:45:51.905460 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:51.905362    5570 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2dc7021-30c7-4b11-8108-8711ace2ea03-maas-api-openshift-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:45:51.905460 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:51.905398    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nm6zh\" (UniqueName: \"kubernetes.io/projected/a2dc7021-30c7-4b11-8108-8711ace2ea03-kube-api-access-nm6zh\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:45:51.905460 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:51.905408    5570 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2dc7021-30c7-4b11-8108-8711ace2ea03-maas-api-xks-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:45:52.576129 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:52.576096    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-229f0a-t5p4b"
Nov 03 20:45:52.576330 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:52.576129    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-229f0a-t5p4b" event={"ID":"a2dc7021-30c7-4b11-8108-8711ace2ea03","Type":"ContainerDied","Data":"ff3c7029638c2e5e130f244353312db16897c8c091812079cb16a30b51c58b8e"}
Nov 03 20:45:52.576330 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:52.576179    5570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ff3c7029638c2e5e130f244353312db16897c8c091812079cb16a30b51c58b8e"
Nov 03 20:45:57.027440 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:57.027401    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-bm5ql"]
Nov 03 20:45:57.040873 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:57.040832    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-bm5ql"]
Nov 03 20:45:57.229942 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:45:57.229903    5570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8bwhz" containerName="extractor"
Nov 03 20:45:57.964089 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:57.964052    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4e8bf4b-5796-4913-85f0-6997a7e0fcec" path="/var/lib/kubelet/pods/f4e8bf4b-5796-4913-85f0-6997a7e0fcec/volumes"
Nov 03 20:45:58.961524 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:58.961486    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-229f0a-7684d76d8znhfl"]
Nov 03 20:45:58.993278 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:45:58.993233    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-229f0a-5bf59kb62m"]
Nov 03 20:46:01.894930 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:46:01.894886    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-229f0a-t5p4b"]
Nov 03 20:46:01.918414 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:46:01.918375    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-229f0a-t5p4b"]
Nov 03 20:46:01.965214 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:46:01.965172    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2dc7021-30c7-4b11-8108-8711ace2ea03" path="/var/lib/kubelet/pods/a2dc7021-30c7-4b11-8108-8711ace2ea03/volumes"
Nov 03 20:46:03.729240 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:46:03.729201    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g69f8"]
Nov 03 20:46:03.962282 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:46:03.962188    5570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-229f0a-7684d76d8znhfl" podUID="af829965-4b63-42ac-82e8-f5e9c3e49bcc" containerName="payload-processing" containerID="cri-o://f752b46e884f2916c3fd122da9dbb1c0f2a41c7ca2e7f3f89c36715ad89807f4" gracePeriod=25
Nov 03 20:46:03.994333 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:46:03.994227    5570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-229f0a-5bf59kb62m" podUID="1308f0c5-ae02-4373-bf57-b557de9c0da8" containerName="payload-pre-processing" containerID="cri-o://3812ac83953641c6027333ec9d9b67e6506328070971ee56dcad5faa3d859973" gracePeriod=25
Nov 03 20:46:08.342635 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:46:08.342597    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-229f0a-openshift-default-d94c48585-9jfkr"]
Nov 03 20:46:08.343208 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:46:08.342894    5570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-229f0a-openshift-default-d94c48585-9jfkr" podUID="1ef1c1f9-e02e-41e8-ba84-1f5bf0f31c53" containerName="istio-proxy" containerID="cri-o://fdf362554e7d1a5be0ffd71d8ba5fc90d760767a51faa5b4ca1cf144f1bda66d" gracePeriod=30
Nov 03 20:46:09.530173 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:46:09.530139    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kdxp9"]
Nov 03 20:46:09.543306 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:46:09.543274    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kdxp9"]
Nov 03 20:46:09.581866 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:46:09.581830    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ds88f"]
Nov 03 20:46:09.582298 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:46:09.582286    5570 cpu_manager.go:414] "containerMap: removing container" podUID="a2dc7021-30c7-4b11-8108-8711ace2ea03" containerName="revoke-keys"
Nov 03 20:46:09.582342 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:46:09.582300    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2dc7021-30c7-4b11-8108-8711ace2ea03" containerName="revoke-keys"
Nov 03 20:46:09.582404 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:46:09.582395    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2dc7021-30c7-4b11-8108-8711ace2ea03" containerName="revoke-keys"
Nov 03 20:46:09.587131 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:46:09.587109    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ds88f"
Nov 03 20:46:09.596044 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:46:09.596018    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7303f2ad-e394-4f4c-8366-2b11b33ddc52-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-ds88f\" (UID: \"7303f2ad-e394-4f4c-8366-2b11b33ddc52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ds88f"
Nov 03 20:46:09.596163 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:46:09.596052    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzr6n\" (UniqueName: \"kubernetes.io/projected/7303f2ad-e394-4f4c-8366-2b11b33ddc52-kube-api-access-pzr6n\") pod \"maas-api-revoke-sub-3d4156f1-ds88f\" (UID: \"7303f2ad-e394-4f4c-8366-2b11b33ddc52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ds88f"
Nov 03 20:46:09.596163 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:46:09.596096    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7303f2ad-e394-4f4c-8366-2b11b33ddc52-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-ds88f\" (UID: \"7303f2ad-e394-4f4c-8366-2b11b33ddc52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ds88f"
Nov 03 20:46:09.602830 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:46:09.602798    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ds88f"]
Nov 03 20:46:09.612670 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:46:09.612575    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:46:09.612670 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:46:09.612644    5570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 20:46:09.612670 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:46:09.612651    5570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 20:46:09.613004 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:46:09.612682    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l7s5w\""
Nov 03 20:46:09.696949 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:46:09.696916    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7303f2ad-e394-4f4c-8366-2b11b33ddc52-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-ds88f\" (UID: \"7303f2ad-e394-4f4c-8366-2b11b33ddc52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ds88f"
Nov 03 20:46:09.697173 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:46:09.696954    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzr6n\" (UniqueName: \"kubernetes.io/projected/7303f2ad-e394-4f4c-8366-2b11b33ddc52-kube-api-access-pzr6n\") pod \"maas-api-revoke-sub-3d4156f1-ds88f\" (UID: \"7303f2ad-e394-4f4c-8366-2b11b33ddc52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ds88f"
Nov 03 20:46:09.697173 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:46:09.697012    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7303f2ad-e394-4f4c-8366-2b11b33ddc52-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-ds88f\" (UID: \"7303f2ad-e394-4f4c-8366-2b11b33ddc52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ds88f"
Nov 03 20:46:09.697594 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:46:09.697570    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7303f2ad-e394-4f4c-8366-2b11b33ddc52-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-ds88f\" (UID: \"7303f2ad-e394-4f4c-8366-2b11b33ddc52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ds88f"
Nov 03 20:46:09.699412 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:46:09.699393    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7303f2ad-e394-4f4c-8366-2b11b33ddc52-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-ds88f\" (UID: \"7303f2ad-e394-4f4c-8366-2b11b33ddc52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ds88f"
Nov 03 20:46:09.717029 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:46:09.716994    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzr6n\" (UniqueName: \"kubernetes.io/projected/7303f2ad-e394-4f4c-8366-2b11b33ddc52-kube-api-access-pzr6n\") pod \"maas-api-revoke-sub-3d4156f1-ds88f\" (UID: \"7303f2ad-e394-4f4c-8366-2b11b33ddc52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ds88f"
Nov 03 20:46:09.900915 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:46:09.900832    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ds88f"
Nov 03 20:46:09.970116 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:46:09.969639    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7697bff-4483-4412-b34a-2f8d1f57299b" path="/var/lib/kubelet/pods/f7697bff-4483-4412-b34a-2f8d1f57299b/volumes"
Nov 03 20:46:10.195895 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:46:10.195863    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ds88f"]
Nov 03 20:46:10.677256 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:46:10.677219    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ds88f" event={"ID":"7303f2ad-e394-4f4c-8366-2b11b33ddc52","Type":"ContainerStarted","Data":"5f593ca61f03988073772d109dab1e466545fe3f72c4d904fd37f9d26ae3283b"}
Nov 03 20:46:10.677685 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:46:10.677277    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ds88f" event={"ID":"7303f2ad-e394-4f4c-8366-2b11b33ddc52","Type":"ContainerStarted","Data":"851496ed206dbc05ec35771516729d902dd16f3d2712ae5ad3b22778e942d9f3"}
Nov 03 20:46:10.697295 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:46:10.697210    5570 patch_prober.go:29] interesting pod/e2e-worker-single-main-229f0a-openshift-default-d94c48585-9jfkr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused
Nov 03 20:46:10.697295 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:46:10.697271    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-229f0a-openshift-default-d94c48585-9jfkr" podUID="1ef1c1f9-e02e-41e8-ba84-1f5bf0f31c53" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 20:46:10.717554 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:46:10.717502    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ds88f" podStartSLOduration=1.71748235 podStartE2EDuration="1.71748235s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:46:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:46:10.716670238 +0000 UTC m=+2504.869332126" watchObservedRunningTime="2026-11-03 20:46:10.71748235 +0000 UTC m=+2504.870144236"
Nov 03 20:46:11.684438 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:46:11.684400    5570 generic.go:356] "Generic (PLEG): container finished" podID="7303f2ad-e394-4f4c-8366-2b11b33ddc52" containerID="5f593ca61f03988073772d109dab1e466545fe3f72c4d904fd37f9d26ae3283b" exitCode=0
Nov 03 20:46:11.684847 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:46:11.684477    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ds88f" event={"ID":"7303f2ad-e394-4f4c-8366-2b11b33ddc52","Type":"ContainerDied","Data":"5f593ca61f03988073772d109dab1e466545fe3f72c4d904fd37f9d26ae3283b"}
Nov 03 20:46:12.110112 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:46:12.110074    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-229f0a-5bf59kb62m" podUID="1308f0c5-ae02-4373-bf57-b557de9c0da8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 20:46:12.826338 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:46:12.826312    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ds88f"
Nov 03 20:46:12.934246 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:46:12.934209    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7303f2ad-e394-4f4c-8366-2b11b33ddc52-kube-api-access-pzr6n\" (UniqueName: \"kubernetes.io/projected/7303f2ad-e394-4f4c-8366-2b11b33ddc52-kube-api-access-pzr6n\") pod \"7303f2ad-e394-4f4c-8366-2b11b33ddc52\" (UID: \"7303f2ad-e394-4f4c-8366-2b11b33ddc52\") "
Nov 03 20:46:12.934463 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:46:12.934279    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7303f2ad-e394-4f4c-8366-2b11b33ddc52-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7303f2ad-e394-4f4c-8366-2b11b33ddc52-maas-api-xks-ca\") pod \"7303f2ad-e394-4f4c-8366-2b11b33ddc52\" (UID: \"7303f2ad-e394-4f4c-8366-2b11b33ddc52\") "
Nov 03 20:46:12.934463 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:46:12.934377    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7303f2ad-e394-4f4c-8366-2b11b33ddc52-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7303f2ad-e394-4f4c-8366-2b11b33ddc52-maas-api-openshift-ca\") pod \"7303f2ad-e394-4f4c-8366-2b11b33ddc52\" (UID: \"7303f2ad-e394-4f4c-8366-2b11b33ddc52\") "
Nov 03 20:46:12.934706 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:46:12.934680    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7303f2ad-e394-4f4c-8366-2b11b33ddc52-maas-api-openshift-ca" pod "7303f2ad-e394-4f4c-8366-2b11b33ddc52" (UID: "7303f2ad-e394-4f4c-8366-2b11b33ddc52"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:46:12.934831 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:46:12.934817    5570 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7303f2ad-e394-4f4c-8366-2b11b33ddc52-maas-api-openshift-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:46:12.936571 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:46:12.936540    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7303f2ad-e394-4f4c-8366-2b11b33ddc52-kube-api-access-pzr6n" pod "7303f2ad-e394-4f4c-8366-2b11b33ddc52" (UID: "7303f2ad-e394-4f4c-8366-2b11b33ddc52"). InnerVolumeSpecName "kube-api-access-pzr6n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:46:12.936660 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:46:12.936622    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7303f2ad-e394-4f4c-8366-2b11b33ddc52-maas-api-xks-ca" pod "7303f2ad-e394-4f4c-8366-2b11b33ddc52" (UID: "7303f2ad-e394-4f4c-8366-2b11b33ddc52"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:46:13.036326 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:46:13.036288    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzr6n\" (UniqueName: \"kubernetes.io/projected/7303f2ad-e394-4f4c-8366-2b11b33ddc52-kube-api-access-pzr6n\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:46:13.036326 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:46:13.036324    5570 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7303f2ad-e394-4f4c-8366-2b11b33ddc52-maas-api-xks-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:46:13.116425 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:46:13.116382    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-229f0a-7684d76d8znhfl" podUID="af829965-4b63-42ac-82e8-f5e9c3e49bcc" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 20:46:13.599671 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:46:13.599643    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-229f0a-openshift-default-d94c48585-9jfkr"
Nov 03 20:46:13.643108 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:46:13.643024    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ef1c1f9-e02e-41e8-ba84-1f5bf0f31c53-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ef1c1f9-e02e-41e8-ba84-1f5bf0f31c53-istio-data\") pod \"1ef1c1f9-e02e-41e8-ba84-1f5bf0f31c53\" (UID: \"1ef1c1f9-e02e-41e8-ba84-1f5bf0f31c53\") "
Nov 03 20:46:13.643108 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:46:13.643092    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1ef1c1f9-e02e-41e8-ba84-1f5bf0f31c53-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ef1c1f9-e02e-41e8-ba84-1f5bf0f31c53-istio-podinfo\") pod \"1ef1c1f9-e02e-41e8-ba84-1f5bf0f31c53\" (UID: \"1ef1c1f9-e02e-41e8-ba84-1f5bf0f31c53\") "
Nov 03 20:46:13.643340 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:46:13.643119    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ef1c1f9-e02e-41e8-ba84-1f5bf0f31c53-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ef1c1f9-e02e-41e8-ba84-1f5bf0f31c53-credential-socket\") pod \"1ef1c1f9-e02e-41e8-ba84-1f5bf0f31c53\" (UID: \"1ef1c1f9-e02e-41e8-ba84-1f5bf0f31c53\") "
Nov 03 20:46:13.643340 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:46:13.643139    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ef1c1f9-e02e-41e8-ba84-1f5bf0f31c53-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ef1c1f9-e02e-41e8-ba84-1f5bf0f31c53-istiod-ca-cert\") pod \"1ef1c1f9-e02e-41e8-ba84-1f5bf0f31c53\" (UID: \"1ef1c1f9-e02e-41e8-ba84-1f5bf0f31c53\") "
Nov 03 20:46:13.643340 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:46:13.643160    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ef1c1f9-e02e-41e8-ba84-1f5bf0f31c53-kube-api-access-tbpkw\" (UniqueName: \"kubernetes.io/projected/1ef1c1f9-e02e-41e8-ba84-1f5bf0f31c53-kube-api-access-tbpkw\") pod \"1ef1c1f9-e02e-41e8-ba84-1f5bf0f31c53\" (UID: \"1ef1c1f9-e02e-41e8-ba84-1f5bf0f31c53\") "
Nov 03 20:46:13.643340 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:46:13.643218    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ef1c1f9-e02e-41e8-ba84-1f5bf0f31c53-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ef1c1f9-e02e-41e8-ba84-1f5bf0f31c53-workload-socket\") pod \"1ef1c1f9-e02e-41e8-ba84-1f5bf0f31c53\" (UID: \"1ef1c1f9-e02e-41e8-ba84-1f5bf0f31c53\") "
Nov 03 20:46:13.643340 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:46:13.643287    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ef1c1f9-e02e-41e8-ba84-1f5bf0f31c53-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ef1c1f9-e02e-41e8-ba84-1f5bf0f31c53-workload-certs\") pod \"1ef1c1f9-e02e-41e8-ba84-1f5bf0f31c53\" (UID: \"1ef1c1f9-e02e-41e8-ba84-1f5bf0f31c53\") "
Nov 03 20:46:13.643340 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:46:13.643334    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ef1c1f9-e02e-41e8-ba84-1f5bf0f31c53-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ef1c1f9-e02e-41e8-ba84-1f5bf0f31c53-istio-envoy\") pod \"1ef1c1f9-e02e-41e8-ba84-1f5bf0f31c53\" (UID: \"1ef1c1f9-e02e-41e8-ba84-1f5bf0f31c53\") "
Nov 03 20:46:13.643640 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:46:13.643367    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ef1c1f9-e02e-41e8-ba84-1f5bf0f31c53-istio-data" pod "1ef1c1f9-e02e-41e8-ba84-1f5bf0f31c53" (UID: "1ef1c1f9-e02e-41e8-ba84-1f5bf0f31c53"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:46:13.643640 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:46:13.643378    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ef1c1f9-e02e-41e8-ba84-1f5bf0f31c53-istio-token\" (UniqueName: \"kubernetes.io/projected/1ef1c1f9-e02e-41e8-ba84-1f5bf0f31c53-istio-token\") pod \"1ef1c1f9-e02e-41e8-ba84-1f5bf0f31c53\" (UID: \"1ef1c1f9-e02e-41e8-ba84-1f5bf0f31c53\") "
Nov 03 20:46:13.643640 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:46:13.643475    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ef1c1f9-e02e-41e8-ba84-1f5bf0f31c53-credential-socket" pod "1ef1c1f9-e02e-41e8-ba84-1f5bf0f31c53" (UID: "1ef1c1f9-e02e-41e8-ba84-1f5bf0f31c53"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:46:13.643640 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:46:13.643603    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ef1c1f9-e02e-41e8-ba84-1f5bf0f31c53-istiod-ca-cert" pod "1ef1c1f9-e02e-41e8-ba84-1f5bf0f31c53" (UID: "1ef1c1f9-e02e-41e8-ba84-1f5bf0f31c53"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:46:13.643640 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:46:13.643627    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ef1c1f9-e02e-41e8-ba84-1f5bf0f31c53-workload-certs" pod "1ef1c1f9-e02e-41e8-ba84-1f5bf0f31c53" (UID: "1ef1c1f9-e02e-41e8-ba84-1f5bf0f31c53"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:46:13.643949 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:46:13.643757    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ef1c1f9-e02e-41e8-ba84-1f5bf0f31c53-workload-socket" pod "1ef1c1f9-e02e-41e8-ba84-1f5bf0f31c53" (UID: "1ef1c1f9-e02e-41e8-ba84-1f5bf0f31c53"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:46:13.644021 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:46:13.644008    5570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ef1c1f9-e02e-41e8-ba84-1f5bf0f31c53-workload-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:46:13.644076 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:46:13.644025    5570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ef1c1f9-e02e-41e8-ba84-1f5bf0f31c53-istio-data\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:46:13.644076 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:46:13.644035    5570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ef1c1f9-e02e-41e8-ba84-1f5bf0f31c53-credential-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:46:13.644076 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:46:13.644043    5570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ef1c1f9-e02e-41e8-ba84-1f5bf0f31c53-istiod-ca-cert\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:46:13.644076 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:46:13.644055    5570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ef1c1f9-e02e-41e8-ba84-1f5bf0f31c53-workload-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:46:13.645502 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:46:13.645473    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1ef1c1f9-e02e-41e8-ba84-1f5bf0f31c53-istio-podinfo" pod "1ef1c1f9-e02e-41e8-ba84-1f5bf0f31c53" (UID: "1ef1c1f9-e02e-41e8-ba84-1f5bf0f31c53"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:46:13.645613 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:46:13.645481    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ef1c1f9-e02e-41e8-ba84-1f5bf0f31c53-istio-token" pod "1ef1c1f9-e02e-41e8-ba84-1f5bf0f31c53" (UID: "1ef1c1f9-e02e-41e8-ba84-1f5bf0f31c53"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:46:13.645837 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:46:13.645821    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ef1c1f9-e02e-41e8-ba84-1f5bf0f31c53-istio-envoy" pod "1ef1c1f9-e02e-41e8-ba84-1f5bf0f31c53" (UID: "1ef1c1f9-e02e-41e8-ba84-1f5bf0f31c53"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:46:13.645901 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:46:13.645855    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ef1c1f9-e02e-41e8-ba84-1f5bf0f31c53-kube-api-access-tbpkw" pod "1ef1c1f9-e02e-41e8-ba84-1f5bf0f31c53" (UID: "1ef1c1f9-e02e-41e8-ba84-1f5bf0f31c53"). InnerVolumeSpecName "kube-api-access-tbpkw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:46:13.696029 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:46:13.695995    5570 generic.go:356] "Generic (PLEG): container finished" podID="1ef1c1f9-e02e-41e8-ba84-1f5bf0f31c53" containerID="fdf362554e7d1a5be0ffd71d8ba5fc90d760767a51faa5b4ca1cf144f1bda66d" exitCode=0
Nov 03 20:46:13.696240 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:46:13.696049    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-229f0a-openshift-default-d94c48585-9jfkr"
Nov 03 20:46:13.696240 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:46:13.696076    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-229f0a-openshift-default-d94c48585-9jfkr" event={"ID":"1ef1c1f9-e02e-41e8-ba84-1f5bf0f31c53","Type":"ContainerDied","Data":"fdf362554e7d1a5be0ffd71d8ba5fc90d760767a51faa5b4ca1cf144f1bda66d"}
Nov 03 20:46:13.696240 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:46:13.696135    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-229f0a-openshift-default-d94c48585-9jfkr" event={"ID":"1ef1c1f9-e02e-41e8-ba84-1f5bf0f31c53","Type":"ContainerDied","Data":"accbae5c201580d5cb444dd63c0de29273b6fb474aa8555e95ce48be44504fb1"}
Nov 03 20:46:13.696240 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:46:13.696166    5570 scope.go:122] "RemoveContainer" containerID="fdf362554e7d1a5be0ffd71d8ba5fc90d760767a51faa5b4ca1cf144f1bda66d"
Nov 03 20:46:13.697638 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:46:13.697617    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ds88f" event={"ID":"7303f2ad-e394-4f4c-8366-2b11b33ddc52","Type":"ContainerDied","Data":"851496ed206dbc05ec35771516729d902dd16f3d2712ae5ad3b22778e942d9f3"}
Nov 03 20:46:13.697742 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:46:13.697645    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ds88f"
Nov 03 20:46:13.697742 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:46:13.697659    5570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="851496ed206dbc05ec35771516729d902dd16f3d2712ae5ad3b22778e942d9f3"
Nov 03 20:46:13.706164 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:46:13.706144    5570 scope.go:122] "RemoveContainer" containerID="5c34a85e9a943eaed5cbb5da874bca7ef8764c88549063a88686cff90882ec34"
Nov 03 20:46:13.714995 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:46:13.714980    5570 scope.go:122] "RemoveContainer" containerID="fdf362554e7d1a5be0ffd71d8ba5fc90d760767a51faa5b4ca1cf144f1bda66d"
Nov 03 20:46:13.715245 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:46:13.715229    5570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdf362554e7d1a5be0ffd71d8ba5fc90d760767a51faa5b4ca1cf144f1bda66d\": container with ID starting with fdf362554e7d1a5be0ffd71d8ba5fc90d760767a51faa5b4ca1cf144f1bda66d not found: ID does not exist" containerID="fdf362554e7d1a5be0ffd71d8ba5fc90d760767a51faa5b4ca1cf144f1bda66d"
Nov 03 20:46:13.715294 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:46:13.715255    5570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdf362554e7d1a5be0ffd71d8ba5fc90d760767a51faa5b4ca1cf144f1bda66d"} err="failed to get container status \"fdf362554e7d1a5be0ffd71d8ba5fc90d760767a51faa5b4ca1cf144f1bda66d\": rpc error: code = NotFound desc = could not find container \"fdf362554e7d1a5be0ffd71d8ba5fc90d760767a51faa5b4ca1cf144f1bda66d\": container with ID starting with fdf362554e7d1a5be0ffd71d8ba5fc90d760767a51faa5b4ca1cf144f1bda66d not found: ID does not exist"
Nov 03 20:46:13.715294 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:46:13.715273    5570 scope.go:122] "RemoveContainer" containerID="5c34a85e9a943eaed5cbb5da874bca7ef8764c88549063a88686cff90882ec34"
Nov 03 20:46:13.715475 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:46:13.715460    5570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c34a85e9a943eaed5cbb5da874bca7ef8764c88549063a88686cff90882ec34\": container with ID starting with 5c34a85e9a943eaed5cbb5da874bca7ef8764c88549063a88686cff90882ec34 not found: ID does not exist" containerID="5c34a85e9a943eaed5cbb5da874bca7ef8764c88549063a88686cff90882ec34"
Nov 03 20:46:13.715516 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:46:13.715480    5570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c34a85e9a943eaed5cbb5da874bca7ef8764c88549063a88686cff90882ec34"} err="failed to get container status \"5c34a85e9a943eaed5cbb5da874bca7ef8764c88549063a88686cff90882ec34\": rpc error: code = NotFound desc = could not find container \"5c34a85e9a943eaed5cbb5da874bca7ef8764c88549063a88686cff90882ec34\": container with ID starting with 5c34a85e9a943eaed5cbb5da874bca7ef8764c88549063a88686cff90882ec34 not found: ID does not exist"
Nov 03 20:46:13.744975 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:46:13.744940    5570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ef1c1f9-e02e-41e8-ba84-1f5bf0f31c53-istio-envoy\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:46:13.744975 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:46:13.744969    5570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ef1c1f9-e02e-41e8-ba84-1f5bf0f31c53-istio-token\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:46:13.744975 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:46:13.744981    5570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ef1c1f9-e02e-41e8-ba84-1f5bf0f31c53-istio-podinfo\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:46:13.745219 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:46:13.744993    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbpkw\" (UniqueName: \"kubernetes.io/projected/1ef1c1f9-e02e-41e8-ba84-1f5bf0f31c53-kube-api-access-tbpkw\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:46:13.750108 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:46:13.750074    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-229f0a-openshift-default-d94c48585-9jfkr"]
Nov 03 20:46:13.756277 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:46:13.756250    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-229f0a-openshift-default-d94c48585-9jfkr"]
Nov 03 20:46:13.963328 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:46:13.963295    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ef1c1f9-e02e-41e8-ba84-1f5bf0f31c53" path="/var/lib/kubelet/pods/1ef1c1f9-e02e-41e8-ba84-1f5bf0f31c53/volumes"
Nov 03 20:46:21.850751 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:46:21.850716    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g69f8"]
Nov 03 20:46:22.110227 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:46:22.110132    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-229f0a-5bf59kb62m" podUID="1308f0c5-ae02-4373-bf57-b557de9c0da8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 20:46:23.123256 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:46:23.123209    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-229f0a-7684d76d8znhfl" podUID="af829965-4b63-42ac-82e8-f5e9c3e49bcc" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 20:46:23.736800 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:46:23.735994    5570 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-657vf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:46:23.736800 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:46:23.736076    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-657vf" podUID="f089dd20-d6a6-4c63-88a5-3fbe9a2ff315" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:46:24.042797 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:46:24.030469    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-8hxlb"]
Nov 03 20:46:24.077874 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:46:24.077833    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-8hxlb"]
Nov 03 20:46:24.364526 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:46:24.364496    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-229f0a-7684d76d8znhfl"
Nov 03 20:46:24.396326 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:46:24.396300    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-229f0a-5bf59kb62m"
Nov 03 20:46:24.536949 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:46:24.536859    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1308f0c5-ae02-4373-bf57-b557de9c0da8-kube-api-access-gh8rv\" (UniqueName: \"kubernetes.io/projected/1308f0c5-ae02-4373-bf57-b557de9c0da8-kube-api-access-gh8rv\") pod \"1308f0c5-ae02-4373-bf57-b557de9c0da8\" (UID: \"1308f0c5-ae02-4373-bf57-b557de9c0da8\") "
Nov 03 20:46:24.536949 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:46:24.536943    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af829965-4b63-42ac-82e8-f5e9c3e49bcc-kube-api-access-ffvlw\" (UniqueName: \"kubernetes.io/projected/af829965-4b63-42ac-82e8-f5e9c3e49bcc-kube-api-access-ffvlw\") pod \"af829965-4b63-42ac-82e8-f5e9c3e49bcc\" (UID: \"af829965-4b63-42ac-82e8-f5e9c3e49bcc\") "
Nov 03 20:46:24.537172 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:46:24.536969    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af829965-4b63-42ac-82e8-f5e9c3e49bcc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af829965-4b63-42ac-82e8-f5e9c3e49bcc-plugins-config-volume\") pod \"af829965-4b63-42ac-82e8-f5e9c3e49bcc\" (UID: \"af829965-4b63-42ac-82e8-f5e9c3e49bcc\") "
Nov 03 20:46:24.537172 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:46:24.537003    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1308f0c5-ae02-4373-bf57-b557de9c0da8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1308f0c5-ae02-4373-bf57-b557de9c0da8-plugins-config-volume\") pod \"1308f0c5-ae02-4373-bf57-b557de9c0da8\" (UID: \"1308f0c5-ae02-4373-bf57-b557de9c0da8\") "
Nov 03 20:46:24.537434 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:46:24.537404    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af829965-4b63-42ac-82e8-f5e9c3e49bcc-plugins-config-volume" pod "af829965-4b63-42ac-82e8-f5e9c3e49bcc" (UID: "af829965-4b63-42ac-82e8-f5e9c3e49bcc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:46:24.537569 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:46:24.537441    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1308f0c5-ae02-4373-bf57-b557de9c0da8-plugins-config-volume" pod "1308f0c5-ae02-4373-bf57-b557de9c0da8" (UID: "1308f0c5-ae02-4373-bf57-b557de9c0da8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:46:24.539090 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:46:24.539067    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1308f0c5-ae02-4373-bf57-b557de9c0da8-kube-api-access-gh8rv" pod "1308f0c5-ae02-4373-bf57-b557de9c0da8" (UID: "1308f0c5-ae02-4373-bf57-b557de9c0da8"). InnerVolumeSpecName "kube-api-access-gh8rv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:46:24.539194 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:46:24.539121    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af829965-4b63-42ac-82e8-f5e9c3e49bcc-kube-api-access-ffvlw" pod "af829965-4b63-42ac-82e8-f5e9c3e49bcc" (UID: "af829965-4b63-42ac-82e8-f5e9c3e49bcc"). InnerVolumeSpecName "kube-api-access-ffvlw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:46:24.638532 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:46:24.638484    5570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1308f0c5-ae02-4373-bf57-b557de9c0da8-plugins-config-volume\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:46:24.638532 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:46:24.638523    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gh8rv\" (UniqueName: \"kubernetes.io/projected/1308f0c5-ae02-4373-bf57-b557de9c0da8-kube-api-access-gh8rv\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:46:24.638532 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:46:24.638533    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ffvlw\" (UniqueName: \"kubernetes.io/projected/af829965-4b63-42ac-82e8-f5e9c3e49bcc-kube-api-access-ffvlw\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:46:24.638532 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:46:24.638543    5570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af829965-4b63-42ac-82e8-f5e9c3e49bcc-plugins-config-volume\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:46:24.786925 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:46:24.786882    5570 generic.go:356] "Generic (PLEG): container finished" podID="af829965-4b63-42ac-82e8-f5e9c3e49bcc" containerID="f752b46e884f2916c3fd122da9dbb1c0f2a41c7ca2e7f3f89c36715ad89807f4" exitCode=0
Nov 03 20:46:24.787164 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:46:24.786930    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-229f0a-7684d76d8znhfl"
Nov 03 20:46:24.787164 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:46:24.786962    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-229f0a-7684d76d8znhfl" event={"ID":"af829965-4b63-42ac-82e8-f5e9c3e49bcc","Type":"ContainerDied","Data":"f752b46e884f2916c3fd122da9dbb1c0f2a41c7ca2e7f3f89c36715ad89807f4"}
Nov 03 20:46:24.787164 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:46:24.787013    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-229f0a-7684d76d8znhfl" event={"ID":"af829965-4b63-42ac-82e8-f5e9c3e49bcc","Type":"ContainerDied","Data":"ca5dea995ba004e78aa6a23bdcc76cb7037d55132d0493746b3a77f4bbf3dd93"}
Nov 03 20:46:24.787164 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:46:24.787036    5570 scope.go:122] "RemoveContainer" containerID="f752b46e884f2916c3fd122da9dbb1c0f2a41c7ca2e7f3f89c36715ad89807f4"
Nov 03 20:46:24.788629 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:46:24.788600    5570 generic.go:356] "Generic (PLEG): container finished" podID="1308f0c5-ae02-4373-bf57-b557de9c0da8" containerID="3812ac83953641c6027333ec9d9b67e6506328070971ee56dcad5faa3d859973" exitCode=0
Nov 03 20:46:24.788757 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:46:24.788652    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-229f0a-5bf59kb62m"
Nov 03 20:46:24.788757 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:46:24.788659    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-229f0a-5bf59kb62m" event={"ID":"1308f0c5-ae02-4373-bf57-b557de9c0da8","Type":"ContainerDied","Data":"3812ac83953641c6027333ec9d9b67e6506328070971ee56dcad5faa3d859973"}
Nov 03 20:46:24.788757 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:46:24.788691    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-229f0a-5bf59kb62m" event={"ID":"1308f0c5-ae02-4373-bf57-b557de9c0da8","Type":"ContainerDied","Data":"9bf36fea3bbeb86396b1a917155833dad76d2ba2d6349130d724cd3d9688cac0"}
Nov 03 20:46:24.798692 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:46:24.798667    5570 scope.go:122] "RemoveContainer" containerID="f752b46e884f2916c3fd122da9dbb1c0f2a41c7ca2e7f3f89c36715ad89807f4"
Nov 03 20:46:24.799144 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:46:24.799118    5570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f752b46e884f2916c3fd122da9dbb1c0f2a41c7ca2e7f3f89c36715ad89807f4\": container with ID starting with f752b46e884f2916c3fd122da9dbb1c0f2a41c7ca2e7f3f89c36715ad89807f4 not found: ID does not exist" containerID="f752b46e884f2916c3fd122da9dbb1c0f2a41c7ca2e7f3f89c36715ad89807f4"
Nov 03 20:46:24.799216 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:46:24.799161    5570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f752b46e884f2916c3fd122da9dbb1c0f2a41c7ca2e7f3f89c36715ad89807f4"} err="failed to get container status \"f752b46e884f2916c3fd122da9dbb1c0f2a41c7ca2e7f3f89c36715ad89807f4\": rpc error: code = NotFound desc = could not find container \"f752b46e884f2916c3fd122da9dbb1c0f2a41c7ca2e7f3f89c36715ad89807f4\": container with ID starting with f752b46e884f2916c3fd122da9dbb1c0f2a41c7ca2e7f3f89c36715ad89807f4 not found: ID does not exist"
Nov 03 20:46:24.799216 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:46:24.799190    5570 scope.go:122] "RemoveContainer" containerID="3812ac83953641c6027333ec9d9b67e6506328070971ee56dcad5faa3d859973"
Nov 03 20:46:24.808336 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:46:24.808313    5570 scope.go:122] "RemoveContainer" containerID="3812ac83953641c6027333ec9d9b67e6506328070971ee56dcad5faa3d859973"
Nov 03 20:46:24.808638 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:46:24.808616    5570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3812ac83953641c6027333ec9d9b67e6506328070971ee56dcad5faa3d859973\": container with ID starting with 3812ac83953641c6027333ec9d9b67e6506328070971ee56dcad5faa3d859973 not found: ID does not exist" containerID="3812ac83953641c6027333ec9d9b67e6506328070971ee56dcad5faa3d859973"
Nov 03 20:46:24.808706 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:46:24.808651    5570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3812ac83953641c6027333ec9d9b67e6506328070971ee56dcad5faa3d859973"} err="failed to get container status \"3812ac83953641c6027333ec9d9b67e6506328070971ee56dcad5faa3d859973\": rpc error: code = NotFound desc = could not find container \"3812ac83953641c6027333ec9d9b67e6506328070971ee56dcad5faa3d859973\": container with ID starting with 3812ac83953641c6027333ec9d9b67e6506328070971ee56dcad5faa3d859973 not found: ID does not exist"
Nov 03 20:46:24.841263 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:46:24.841227    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-229f0a-5bf59kb62m"]
Nov 03 20:46:24.844130 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:46:24.844099    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-229f0a-5bf59kb62m"]
Nov 03 20:46:24.891008 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:46:24.889831    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-229f0a-7684d76d8znhfl"]
Nov 03 20:46:24.894070 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:46:24.893994    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-229f0a-7684d76d8znhfl"]
Nov 03 20:46:25.964101 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:46:25.964066    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1308f0c5-ae02-4373-bf57-b557de9c0da8" path="/var/lib/kubelet/pods/1308f0c5-ae02-4373-bf57-b557de9c0da8/volumes"
Nov 03 20:46:25.964495 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:46:25.964450    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af829965-4b63-42ac-82e8-f5e9c3e49bcc" path="/var/lib/kubelet/pods/af829965-4b63-42ac-82e8-f5e9c3e49bcc/volumes"
Nov 03 20:46:25.964831 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:46:25.964817    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8b8db97-5583-40e0-90b5-8076a3e052d0" path="/var/lib/kubelet/pods/d8b8db97-5583-40e0-90b5-8076a3e052d0/volumes"
Nov 03 20:46:27.229500 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:46:27.229454    5570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8bwhz" containerName="extractor"
Nov 03 20:46:27.729548 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:46:27.729509    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g69f8"]
Nov 03 20:46:32.238699 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:46:32.238659    5570 scope.go:122] "RemoveContainer" containerID="04cc4d4785163288501bd4e4d3c1be0be2cbe27dbd1711d005a320154383795c"
Nov 03 20:46:35.025336 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:46:35.025291    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-br276"]
Nov 03 20:46:35.038513 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:46:35.038487    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-br276"]
Nov 03 20:46:35.964966 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:46:35.964929    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2375597e-e94e-433b-9b1b-d48abe92e870" path="/var/lib/kubelet/pods/2375597e-e94e-433b-9b1b-d48abe92e870/volumes"
Nov 03 20:46:56.027008 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:46:56.026932    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-kwvfl"]
Nov 03 20:46:56.039558 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:46:56.039530    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-kwvfl"]
Nov 03 20:46:57.226624 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:46:57.226587    5570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8bwhz" containerName="extractor"
Nov 03 20:46:57.962716 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:46:57.962676    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1f35880-1a34-4214-a893-b1e1ba5f27c4" path="/var/lib/kubelet/pods/d1f35880-1a34-4214-a893-b1e1ba5f27c4/volumes"
Nov 03 20:47:18.028683 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:47:18.028647    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-p4kmk"]
Nov 03 20:47:18.042029 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:47:18.041992    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-p4kmk"]
Nov 03 20:47:19.963488 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:47:19.963451    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d6c1202-d2b4-4066-b6b9-b8ba9d7e3421" path="/var/lib/kubelet/pods/4d6c1202-d2b4-4066-b6b9-b8ba9d7e3421/volumes"
Nov 03 20:47:27.227200 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:47:27.227155    5570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8bwhz" containerName="extractor"
Nov 03 20:47:32.273739 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:47:32.273708    5570 scope.go:122] "RemoveContainer" containerID="26303f6ea19af99d93b739eb6517a0b50fe499d28e10528e5ee1d8af34b77645"
Nov 03 20:47:32.282351 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:47:32.282324    5570 scope.go:122] "RemoveContainer" containerID="874afa765839756afeb7c6a531db3ad2dbebce38b1c07f17d716078b7cf0322d"
Nov 03 20:47:32.289962 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:47:32.289941    5570 scope.go:122] "RemoveContainer" containerID="2d24bb8477d0c4f0a6985b8b19df168880d0b7349042a43a2599e441ac23eb01"
Nov 03 20:47:32.297562 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:47:32.297543    5570 scope.go:122] "RemoveContainer" containerID="65d1b6094e465f0942980664226750d8ee6880970a51f6bd4708581ae855ce30"
Nov 03 20:47:38.071097 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:47:38.071067    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-2ds6v"]
Nov 03 20:47:38.084055 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:47:38.084026    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-2ds6v"]
Nov 03 20:47:39.963337 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:47:39.963303    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e19047f4-09c4-4a2d-b576-6988366782d8" path="/var/lib/kubelet/pods/e19047f4-09c4-4a2d-b576-6988366782d8/volumes"
Nov 03 20:47:57.070326 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:47:57.070289    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-2kn85"]
Nov 03 20:47:57.070746 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:47:57.070730    5570 cpu_manager.go:414] "containerMap: removing container" podUID="7303f2ad-e394-4f4c-8366-2b11b33ddc52" containerName="revoke-keys"
Nov 03 20:47:57.070815 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:47:57.070747    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7303f2ad-e394-4f4c-8366-2b11b33ddc52" containerName="revoke-keys"
Nov 03 20:47:57.070815 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:47:57.070765    5570 cpu_manager.go:414] "containerMap: removing container" podUID="af829965-4b63-42ac-82e8-f5e9c3e49bcc" containerName="payload-processing"
Nov 03 20:47:57.070815 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:47:57.070782    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af829965-4b63-42ac-82e8-f5e9c3e49bcc" containerName="payload-processing"
Nov 03 20:47:57.070815 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:47:57.070806    5570 cpu_manager.go:414] "containerMap: removing container" podUID="1ef1c1f9-e02e-41e8-ba84-1f5bf0f31c53" containerName="istio-proxy"
Nov 03 20:47:57.070815 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:47:57.070811    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ef1c1f9-e02e-41e8-ba84-1f5bf0f31c53" containerName="istio-proxy"
Nov 03 20:47:57.070977 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:47:57.070822    5570 cpu_manager.go:414] "containerMap: removing container" podUID="1308f0c5-ae02-4373-bf57-b557de9c0da8" containerName="payload-pre-processing"
Nov 03 20:47:57.070977 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:47:57.070828    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1308f0c5-ae02-4373-bf57-b557de9c0da8" containerName="payload-pre-processing"
Nov 03 20:47:57.070977 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:47:57.070838    5570 cpu_manager.go:414] "containerMap: removing container" podUID="1ef1c1f9-e02e-41e8-ba84-1f5bf0f31c53" containerName="istio-proxy"
Nov 03 20:47:57.070977 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:47:57.070843    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ef1c1f9-e02e-41e8-ba84-1f5bf0f31c53" containerName="istio-proxy"
Nov 03 20:47:57.070977 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:47:57.070911    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ef1c1f9-e02e-41e8-ba84-1f5bf0f31c53" containerName="istio-proxy"
Nov 03 20:47:57.070977 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:47:57.070923    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="7303f2ad-e394-4f4c-8366-2b11b33ddc52" containerName="revoke-keys"
Nov 03 20:47:57.070977 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:47:57.070933    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="af829965-4b63-42ac-82e8-f5e9c3e49bcc" containerName="payload-processing"
Nov 03 20:47:57.070977 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:47:57.070946    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ef1c1f9-e02e-41e8-ba84-1f5bf0f31c53" containerName="istio-proxy"
Nov 03 20:47:57.070977 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:47:57.070953    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="1308f0c5-ae02-4373-bf57-b557de9c0da8" containerName="payload-pre-processing"
Nov 03 20:47:57.074032 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:47:57.074016    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-2kn85"
Nov 03 20:47:57.089738 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:47:57.089709    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-2kn85"]
Nov 03 20:47:57.101184 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:47:57.101152    5570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Nov 03 20:47:57.101184 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:47:57.101156    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Nov 03 20:47:57.140649 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:47:57.140620    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4f5c\" (UniqueName: \"kubernetes.io/projected/f288d261-5a87-46be-b67f-89370c5a4a48-kube-api-access-r4f5c\") pod \"kuadrant-console-plugin-66c4fb575d-2kn85\" (UID: \"f288d261-5a87-46be-b67f-89370c5a4a48\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-2kn85"
Nov 03 20:47:57.140836 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:47:57.140658    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f288d261-5a87-46be-b67f-89370c5a4a48-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-2kn85\" (UID: \"f288d261-5a87-46be-b67f-89370c5a4a48\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-2kn85"
Nov 03 20:47:57.140884 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:47:57.140849    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/f288d261-5a87-46be-b67f-89370c5a4a48-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-2kn85\" (UID: \"f288d261-5a87-46be-b67f-89370c5a4a48\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-2kn85"
Nov 03 20:47:57.226009 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:47:57.225976    5570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8bwhz" containerName="extractor"
Nov 03 20:47:57.241897 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:47:57.241859    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/f288d261-5a87-46be-b67f-89370c5a4a48-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-2kn85\" (UID: \"f288d261-5a87-46be-b67f-89370c5a4a48\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-2kn85"
Nov 03 20:47:57.242219 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:47:57.241933    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4f5c\" (UniqueName: \"kubernetes.io/projected/f288d261-5a87-46be-b67f-89370c5a4a48-kube-api-access-r4f5c\") pod \"kuadrant-console-plugin-66c4fb575d-2kn85\" (UID: \"f288d261-5a87-46be-b67f-89370c5a4a48\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-2kn85"
Nov 03 20:47:57.242219 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:47:57.241978    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f288d261-5a87-46be-b67f-89370c5a4a48-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-2kn85\" (UID: \"f288d261-5a87-46be-b67f-89370c5a4a48\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-2kn85"
Nov 03 20:47:57.255715 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:47:57.242769    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f288d261-5a87-46be-b67f-89370c5a4a48-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-2kn85\" (UID: \"f288d261-5a87-46be-b67f-89370c5a4a48\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-2kn85"
Nov 03 20:47:57.262068 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:47:57.261038    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g69f8"]
Nov 03 20:47:57.267953 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:47:57.267921    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/f288d261-5a87-46be-b67f-89370c5a4a48-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-2kn85\" (UID: \"f288d261-5a87-46be-b67f-89370c5a4a48\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-2kn85"
Nov 03 20:47:57.279009 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:47:57.278985    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4f5c\" (UniqueName: \"kubernetes.io/projected/f288d261-5a87-46be-b67f-89370c5a4a48-kube-api-access-r4f5c\") pod \"kuadrant-console-plugin-66c4fb575d-2kn85\" (UID: \"f288d261-5a87-46be-b67f-89370c5a4a48\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-2kn85"
Nov 03 20:47:57.385315 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:47:57.385248    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-2kn85"
Nov 03 20:47:57.689161 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:47:57.689118    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-2kn85"]
Nov 03 20:47:57.690722 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:47:57.690679    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf288d261_5a87_46be_b67f_89370c5a4a48.slice/crio-005e399ddd4fe55f25cc29a6e9d8a1d443c7735d77500250d7425af0f0f1d56b WatchSource:0}: Error finding container 005e399ddd4fe55f25cc29a6e9d8a1d443c7735d77500250d7425af0f0f1d56b: Status 404 returned error can't find the container with id 005e399ddd4fe55f25cc29a6e9d8a1d443c7735d77500250d7425af0f0f1d56b
Nov 03 20:47:57.692471 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:47:57.692446    5570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 20:47:58.028401 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:47:58.028095    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-fjm82"]
Nov 03 20:47:58.041237 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:47:58.041197    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-fjm82"]
Nov 03 20:47:58.318679 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:47:58.318575    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-2kn85" event={"ID":"f288d261-5a87-46be-b67f-89370c5a4a48","Type":"ContainerStarted","Data":"005e399ddd4fe55f25cc29a6e9d8a1d443c7735d77500250d7425af0f0f1d56b"}
Nov 03 20:48:00.081799 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:48:00.059686    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9bc5d581-0ac5-4623-b41a-0e1b81e964f3" path="/var/lib/kubelet/pods/9bc5d581-0ac5-4623-b41a-0e1b81e964f3/volumes"
Nov 03 20:48:00.416394 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:48:00.416311    5570 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-78txb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:48:00.416714 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:48:00.416657    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-78txb" podUID="b65b5a79-c181-45ec-9cae-ebb5e5bbd865" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:48:01.478661 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:48:01.478627    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-8wdh8"]
Nov 03 20:48:01.482681 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:48:01.482660    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-8wdh8"
Nov 03 20:48:01.511877 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:48:01.511766    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-8wdh8"]
Nov 03 20:48:01.523237 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:48:01.523181    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:48:01.523237 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:48:01.523181    5570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 20:48:01.523449 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:48:01.523247    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l7s5w\""
Nov 03 20:48:01.523511 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:48:01.523490    5570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 20:48:01.607498 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:48:01.607467    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44794ebc-c5e7-47fc-a0d3-0cd1d9e63e53-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-8wdh8\" (UID: \"44794ebc-c5e7-47fc-a0d3-0cd1d9e63e53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-8wdh8"
Nov 03 20:48:01.607682 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:48:01.607660    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44794ebc-c5e7-47fc-a0d3-0cd1d9e63e53-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-8wdh8\" (UID: \"44794ebc-c5e7-47fc-a0d3-0cd1d9e63e53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-8wdh8"
Nov 03 20:48:01.607743 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:48:01.607717    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qm2jb\" (UniqueName: \"kubernetes.io/projected/44794ebc-c5e7-47fc-a0d3-0cd1d9e63e53-kube-api-access-qm2jb\") pod \"maas-api-revoke-sub-d576e68b-8wdh8\" (UID: \"44794ebc-c5e7-47fc-a0d3-0cd1d9e63e53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-8wdh8"
Nov 03 20:48:01.709096 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:48:01.709045    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44794ebc-c5e7-47fc-a0d3-0cd1d9e63e53-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-8wdh8\" (UID: \"44794ebc-c5e7-47fc-a0d3-0cd1d9e63e53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-8wdh8"
Nov 03 20:48:01.709299 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:48:01.709138    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qm2jb\" (UniqueName: \"kubernetes.io/projected/44794ebc-c5e7-47fc-a0d3-0cd1d9e63e53-kube-api-access-qm2jb\") pod \"maas-api-revoke-sub-d576e68b-8wdh8\" (UID: \"44794ebc-c5e7-47fc-a0d3-0cd1d9e63e53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-8wdh8"
Nov 03 20:48:01.709299 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:48:01.709189    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44794ebc-c5e7-47fc-a0d3-0cd1d9e63e53-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-8wdh8\" (UID: \"44794ebc-c5e7-47fc-a0d3-0cd1d9e63e53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-8wdh8"
Nov 03 20:48:01.710150 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:48:01.710124    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44794ebc-c5e7-47fc-a0d3-0cd1d9e63e53-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-8wdh8\" (UID: \"44794ebc-c5e7-47fc-a0d3-0cd1d9e63e53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-8wdh8"
Nov 03 20:48:01.711959 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:48:01.711925    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44794ebc-c5e7-47fc-a0d3-0cd1d9e63e53-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-8wdh8\" (UID: \"44794ebc-c5e7-47fc-a0d3-0cd1d9e63e53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-8wdh8"
Nov 03 20:48:01.731087 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:48:01.731020    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qm2jb\" (UniqueName: \"kubernetes.io/projected/44794ebc-c5e7-47fc-a0d3-0cd1d9e63e53-kube-api-access-qm2jb\") pod \"maas-api-revoke-sub-d576e68b-8wdh8\" (UID: \"44794ebc-c5e7-47fc-a0d3-0cd1d9e63e53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-8wdh8"
Nov 03 20:48:01.798633 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:48:01.798595    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-8wdh8"
Nov 03 20:48:02.110700 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:48:02.110669    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-8wdh8"]
Nov 03 20:48:02.112072 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:48:02.112045    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44794ebc_c5e7_47fc_a0d3_0cd1d9e63e53.slice/crio-b820dd59421c5f74f414f89a2b294b49b53162448132534b1228a7d679b1e205 WatchSource:0}: Error finding container b820dd59421c5f74f414f89a2b294b49b53162448132534b1228a7d679b1e205: Status 404 returned error can't find the container with id b820dd59421c5f74f414f89a2b294b49b53162448132534b1228a7d679b1e205
Nov 03 20:48:02.457986 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:48:02.457949    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-8wdh8" event={"ID":"44794ebc-c5e7-47fc-a0d3-0cd1d9e63e53","Type":"ContainerStarted","Data":"b820dd59421c5f74f414f89a2b294b49b53162448132534b1228a7d679b1e205"}
Nov 03 20:48:03.467006 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:48:03.466964    5570 generic.go:356] "Generic (PLEG): container finished" podID="44794ebc-c5e7-47fc-a0d3-0cd1d9e63e53" containerID="3b4111ba8ef93d9aea95435b6361cd7b2ac57f775782dd8d0d42ad3d5ea1777e" exitCode=0
Nov 03 20:48:03.467471 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:48:03.467017    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-8wdh8" event={"ID":"44794ebc-c5e7-47fc-a0d3-0cd1d9e63e53","Type":"ContainerDied","Data":"3b4111ba8ef93d9aea95435b6361cd7b2ac57f775782dd8d0d42ad3d5ea1777e"}
Nov 03 20:48:04.473513 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:48:04.473472    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-2kn85" event={"ID":"f288d261-5a87-46be-b67f-89370c5a4a48","Type":"ContainerStarted","Data":"d675eff8d4cc1cdccef2c9e517ba6327c733ee6890eda6f9500dc7ac56622c8f"}
Nov 03 20:48:04.532221 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:48:04.532160    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-2kn85" podStartSLOduration=1.054447724 podStartE2EDuration="7.53214195s" totalImagesPullingTime="6.477694226s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:47:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:48:04.528682655 +0000 UTC m=+2618.681344542" watchObservedRunningTime="2026-11-03 20:48:04.53214195 +0000 UTC m=+2618.684803838"
Nov 03 20:48:04.609645 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:48:04.609614    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-8wdh8"
Nov 03 20:48:04.742218 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:48:04.742125    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/44794ebc-c5e7-47fc-a0d3-0cd1d9e63e53-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44794ebc-c5e7-47fc-a0d3-0cd1d9e63e53-maas-api-xks-ca\") pod \"44794ebc-c5e7-47fc-a0d3-0cd1d9e63e53\" (UID: \"44794ebc-c5e7-47fc-a0d3-0cd1d9e63e53\") "
Nov 03 20:48:04.742218 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:48:04.742214    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44794ebc-c5e7-47fc-a0d3-0cd1d9e63e53-kube-api-access-qm2jb\" (UniqueName: \"kubernetes.io/projected/44794ebc-c5e7-47fc-a0d3-0cd1d9e63e53-kube-api-access-qm2jb\") pod \"44794ebc-c5e7-47fc-a0d3-0cd1d9e63e53\" (UID: \"44794ebc-c5e7-47fc-a0d3-0cd1d9e63e53\") "
Nov 03 20:48:04.742467 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:48:04.742316    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44794ebc-c5e7-47fc-a0d3-0cd1d9e63e53-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44794ebc-c5e7-47fc-a0d3-0cd1d9e63e53-maas-api-openshift-ca\") pod \"44794ebc-c5e7-47fc-a0d3-0cd1d9e63e53\" (UID: \"44794ebc-c5e7-47fc-a0d3-0cd1d9e63e53\") "
Nov 03 20:48:04.742714 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:48:04.742681    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44794ebc-c5e7-47fc-a0d3-0cd1d9e63e53-maas-api-openshift-ca" pod "44794ebc-c5e7-47fc-a0d3-0cd1d9e63e53" (UID: "44794ebc-c5e7-47fc-a0d3-0cd1d9e63e53"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:48:04.744324 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:48:04.744304    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44794ebc-c5e7-47fc-a0d3-0cd1d9e63e53-maas-api-xks-ca" pod "44794ebc-c5e7-47fc-a0d3-0cd1d9e63e53" (UID: "44794ebc-c5e7-47fc-a0d3-0cd1d9e63e53"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:48:04.744393 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:48:04.744360    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44794ebc-c5e7-47fc-a0d3-0cd1d9e63e53-kube-api-access-qm2jb" pod "44794ebc-c5e7-47fc-a0d3-0cd1d9e63e53" (UID: "44794ebc-c5e7-47fc-a0d3-0cd1d9e63e53"). InnerVolumeSpecName "kube-api-access-qm2jb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:48:04.843480 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:48:04.843451    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qm2jb\" (UniqueName: \"kubernetes.io/projected/44794ebc-c5e7-47fc-a0d3-0cd1d9e63e53-kube-api-access-qm2jb\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:48:04.843480 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:48:04.843477    5570 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44794ebc-c5e7-47fc-a0d3-0cd1d9e63e53-maas-api-openshift-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:48:04.843480 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:48:04.843487    5570 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44794ebc-c5e7-47fc-a0d3-0cd1d9e63e53-maas-api-xks-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:48:05.480074 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:48:05.480041    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-8wdh8"
Nov 03 20:48:05.480568 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:48:05.480080    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-8wdh8" event={"ID":"44794ebc-c5e7-47fc-a0d3-0cd1d9e63e53","Type":"ContainerDied","Data":"b820dd59421c5f74f414f89a2b294b49b53162448132534b1228a7d679b1e205"}
Nov 03 20:48:05.480568 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:48:05.480155    5570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b820dd59421c5f74f414f89a2b294b49b53162448132534b1228a7d679b1e205"
Nov 03 20:48:12.740570 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:48:12.740532    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g69f8"]
Nov 03 20:48:17.904988 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:48:17.904951    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g69f8"]
Nov 03 20:48:27.227277 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:48:27.227246    5570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8bwhz" containerName="extractor"
Nov 03 20:48:32.329593 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:48:32.329561    5570 scope.go:122] "RemoveContainer" containerID="6025faab011372ec2db92604e8c2aaa50d6d3c8653c916742a761bd62d7a5c5d"
Nov 03 20:48:32.338436 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:48:32.338413    5570 scope.go:122] "RemoveContainer" containerID="66c06a9cd82318108ea0c87d760f7f57ab3bf5f4dae231006e13ac8622f4f7d6"
Nov 03 20:48:33.742284 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:48:33.742250    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g69f8"]
Nov 03 20:48:34.623446 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:48:34.623416    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-0d89d1-openshift-default-79f4655d69477wq"]
Nov 03 20:48:34.624016 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:48:34.623992    5570 cpu_manager.go:414] "containerMap: removing container" podUID="44794ebc-c5e7-47fc-a0d3-0cd1d9e63e53" containerName="revoke-keys"
Nov 03 20:48:34.624016 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:48:34.624017    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44794ebc-c5e7-47fc-a0d3-0cd1d9e63e53" containerName="revoke-keys"
Nov 03 20:48:34.624214 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:48:34.624154    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="44794ebc-c5e7-47fc-a0d3-0cd1d9e63e53" containerName="revoke-keys"
Nov 03 20:48:34.770294 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:48:34.768069    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-0d89d1-openshift-default-79f4655d69477wq"]
Nov 03 20:48:34.770294 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:48:34.768160    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-0d89d1-openshift-default-79f4655d69477wq"
Nov 03 20:48:34.787331 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:48:34.787300    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-0d89d1-openshift-default-dockercfg-v4rsg\""
Nov 03 20:48:34.886813 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:48:34.886703    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f89ad58b-cab6-4558-aa07-24370249825a-workload-socket\") pod \"e2e-worker-single-main-0d89d1-openshift-default-79f4655d69477wq\" (UID: \"f89ad58b-cab6-4558-aa07-24370249825a\") " pod="openshift-ingress/e2e-worker-single-main-0d89d1-openshift-default-79f4655d69477wq"
Nov 03 20:48:34.887608 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:48:34.887053    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f89ad58b-cab6-4558-aa07-24370249825a-istiod-ca-cert\") pod \"e2e-worker-single-main-0d89d1-openshift-default-79f4655d69477wq\" (UID: \"f89ad58b-cab6-4558-aa07-24370249825a\") " pod="openshift-ingress/e2e-worker-single-main-0d89d1-openshift-default-79f4655d69477wq"
Nov 03 20:48:34.887608 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:48:34.887096    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f89ad58b-cab6-4558-aa07-24370249825a-workload-certs\") pod \"e2e-worker-single-main-0d89d1-openshift-default-79f4655d69477wq\" (UID: \"f89ad58b-cab6-4558-aa07-24370249825a\") " pod="openshift-ingress/e2e-worker-single-main-0d89d1-openshift-default-79f4655d69477wq"
Nov 03 20:48:34.887608 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:48:34.887196    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f89ad58b-cab6-4558-aa07-24370249825a-istio-token\") pod \"e2e-worker-single-main-0d89d1-openshift-default-79f4655d69477wq\" (UID: \"f89ad58b-cab6-4558-aa07-24370249825a\") " pod="openshift-ingress/e2e-worker-single-main-0d89d1-openshift-default-79f4655d69477wq"
Nov 03 20:48:34.887608 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:48:34.887242    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7g8vn\" (UniqueName: \"kubernetes.io/projected/f89ad58b-cab6-4558-aa07-24370249825a-kube-api-access-7g8vn\") pod \"e2e-worker-single-main-0d89d1-openshift-default-79f4655d69477wq\" (UID: \"f89ad58b-cab6-4558-aa07-24370249825a\") " pod="openshift-ingress/e2e-worker-single-main-0d89d1-openshift-default-79f4655d69477wq"
Nov 03 20:48:34.887608 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:48:34.887271    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f89ad58b-cab6-4558-aa07-24370249825a-istio-envoy\") pod \"e2e-worker-single-main-0d89d1-openshift-default-79f4655d69477wq\" (UID: \"f89ad58b-cab6-4558-aa07-24370249825a\") " pod="openshift-ingress/e2e-worker-single-main-0d89d1-openshift-default-79f4655d69477wq"
Nov 03 20:48:34.887608 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:48:34.887307    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f89ad58b-cab6-4558-aa07-24370249825a-istio-data\") pod \"e2e-worker-single-main-0d89d1-openshift-default-79f4655d69477wq\" (UID: \"f89ad58b-cab6-4558-aa07-24370249825a\") " pod="openshift-ingress/e2e-worker-single-main-0d89d1-openshift-default-79f4655d69477wq"
Nov 03 20:48:34.887608 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:48:34.887386    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f89ad58b-cab6-4558-aa07-24370249825a-istio-podinfo\") pod \"e2e-worker-single-main-0d89d1-openshift-default-79f4655d69477wq\" (UID: \"f89ad58b-cab6-4558-aa07-24370249825a\") " pod="openshift-ingress/e2e-worker-single-main-0d89d1-openshift-default-79f4655d69477wq"
Nov 03 20:48:34.887608 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:48:34.887426    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f89ad58b-cab6-4558-aa07-24370249825a-credential-socket\") pod \"e2e-worker-single-main-0d89d1-openshift-default-79f4655d69477wq\" (UID: \"f89ad58b-cab6-4558-aa07-24370249825a\") " pod="openshift-ingress/e2e-worker-single-main-0d89d1-openshift-default-79f4655d69477wq"
Nov 03 20:48:34.988466 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:48:34.988430    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f89ad58b-cab6-4558-aa07-24370249825a-istio-token\") pod \"e2e-worker-single-main-0d89d1-openshift-default-79f4655d69477wq\" (UID: \"f89ad58b-cab6-4558-aa07-24370249825a\") " pod="openshift-ingress/e2e-worker-single-main-0d89d1-openshift-default-79f4655d69477wq"
Nov 03 20:48:34.988656 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:48:34.988475    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7g8vn\" (UniqueName: \"kubernetes.io/projected/f89ad58b-cab6-4558-aa07-24370249825a-kube-api-access-7g8vn\") pod \"e2e-worker-single-main-0d89d1-openshift-default-79f4655d69477wq\" (UID: \"f89ad58b-cab6-4558-aa07-24370249825a\") " pod="openshift-ingress/e2e-worker-single-main-0d89d1-openshift-default-79f4655d69477wq"
Nov 03 20:48:34.988656 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:48:34.988496    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f89ad58b-cab6-4558-aa07-24370249825a-istio-envoy\") pod \"e2e-worker-single-main-0d89d1-openshift-default-79f4655d69477wq\" (UID: \"f89ad58b-cab6-4558-aa07-24370249825a\") " pod="openshift-ingress/e2e-worker-single-main-0d89d1-openshift-default-79f4655d69477wq"
Nov 03 20:48:34.988656 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:48:34.988527    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f89ad58b-cab6-4558-aa07-24370249825a-istio-data\") pod \"e2e-worker-single-main-0d89d1-openshift-default-79f4655d69477wq\" (UID: \"f89ad58b-cab6-4558-aa07-24370249825a\") " pod="openshift-ingress/e2e-worker-single-main-0d89d1-openshift-default-79f4655d69477wq"
Nov 03 20:48:34.988656 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:48:34.988573    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f89ad58b-cab6-4558-aa07-24370249825a-istio-podinfo\") pod \"e2e-worker-single-main-0d89d1-openshift-default-79f4655d69477wq\" (UID: \"f89ad58b-cab6-4558-aa07-24370249825a\") " pod="openshift-ingress/e2e-worker-single-main-0d89d1-openshift-default-79f4655d69477wq"
Nov 03 20:48:34.988656 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:48:34.988610    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f89ad58b-cab6-4558-aa07-24370249825a-credential-socket\") pod \"e2e-worker-single-main-0d89d1-openshift-default-79f4655d69477wq\" (UID: \"f89ad58b-cab6-4558-aa07-24370249825a\") " pod="openshift-ingress/e2e-worker-single-main-0d89d1-openshift-default-79f4655d69477wq"
Nov 03 20:48:34.988656 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:48:34.988646    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f89ad58b-cab6-4558-aa07-24370249825a-workload-socket\") pod \"e2e-worker-single-main-0d89d1-openshift-default-79f4655d69477wq\" (UID: \"f89ad58b-cab6-4558-aa07-24370249825a\") " pod="openshift-ingress/e2e-worker-single-main-0d89d1-openshift-default-79f4655d69477wq"
Nov 03 20:48:34.989054 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:48:34.988678    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f89ad58b-cab6-4558-aa07-24370249825a-istiod-ca-cert\") pod \"e2e-worker-single-main-0d89d1-openshift-default-79f4655d69477wq\" (UID: \"f89ad58b-cab6-4558-aa07-24370249825a\") " pod="openshift-ingress/e2e-worker-single-main-0d89d1-openshift-default-79f4655d69477wq"
Nov 03 20:48:34.989054 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:48:34.988704    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f89ad58b-cab6-4558-aa07-24370249825a-workload-certs\") pod \"e2e-worker-single-main-0d89d1-openshift-default-79f4655d69477wq\" (UID: \"f89ad58b-cab6-4558-aa07-24370249825a\") " pod="openshift-ingress/e2e-worker-single-main-0d89d1-openshift-default-79f4655d69477wq"
Nov 03 20:48:34.989187 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:48:34.989094    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f89ad58b-cab6-4558-aa07-24370249825a-credential-socket\") pod \"e2e-worker-single-main-0d89d1-openshift-default-79f4655d69477wq\" (UID: \"f89ad58b-cab6-4558-aa07-24370249825a\") " pod="openshift-ingress/e2e-worker-single-main-0d89d1-openshift-default-79f4655d69477wq"
Nov 03 20:48:34.989187 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:48:34.989155    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f89ad58b-cab6-4558-aa07-24370249825a-workload-socket\") pod \"e2e-worker-single-main-0d89d1-openshift-default-79f4655d69477wq\" (UID: \"f89ad58b-cab6-4558-aa07-24370249825a\") " pod="openshift-ingress/e2e-worker-single-main-0d89d1-openshift-default-79f4655d69477wq"
Nov 03 20:48:34.989308 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:48:34.989208    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f89ad58b-cab6-4558-aa07-24370249825a-workload-certs\") pod \"e2e-worker-single-main-0d89d1-openshift-default-79f4655d69477wq\" (UID: \"f89ad58b-cab6-4558-aa07-24370249825a\") " pod="openshift-ingress/e2e-worker-single-main-0d89d1-openshift-default-79f4655d69477wq"
Nov 03 20:48:34.989308 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:48:34.989265    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f89ad58b-cab6-4558-aa07-24370249825a-istio-data\") pod \"e2e-worker-single-main-0d89d1-openshift-default-79f4655d69477wq\" (UID: \"f89ad58b-cab6-4558-aa07-24370249825a\") " pod="openshift-ingress/e2e-worker-single-main-0d89d1-openshift-default-79f4655d69477wq"
Nov 03 20:48:34.989435 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:48:34.989408    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f89ad58b-cab6-4558-aa07-24370249825a-istiod-ca-cert\") pod \"e2e-worker-single-main-0d89d1-openshift-default-79f4655d69477wq\" (UID: \"f89ad58b-cab6-4558-aa07-24370249825a\") " pod="openshift-ingress/e2e-worker-single-main-0d89d1-openshift-default-79f4655d69477wq"
Nov 03 20:48:34.991036 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:48:34.991011    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f89ad58b-cab6-4558-aa07-24370249825a-istio-envoy\") pod \"e2e-worker-single-main-0d89d1-openshift-default-79f4655d69477wq\" (UID: \"f89ad58b-cab6-4558-aa07-24370249825a\") " pod="openshift-ingress/e2e-worker-single-main-0d89d1-openshift-default-79f4655d69477wq"
Nov 03 20:48:34.991196 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:48:34.991179    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f89ad58b-cab6-4558-aa07-24370249825a-istio-podinfo\") pod \"e2e-worker-single-main-0d89d1-openshift-default-79f4655d69477wq\" (UID: \"f89ad58b-cab6-4558-aa07-24370249825a\") " pod="openshift-ingress/e2e-worker-single-main-0d89d1-openshift-default-79f4655d69477wq"
Nov 03 20:48:35.009676 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:48:35.009656    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7g8vn\" (UniqueName: \"kubernetes.io/projected/f89ad58b-cab6-4558-aa07-24370249825a-kube-api-access-7g8vn\") pod \"e2e-worker-single-main-0d89d1-openshift-default-79f4655d69477wq\" (UID: \"f89ad58b-cab6-4558-aa07-24370249825a\") " pod="openshift-ingress/e2e-worker-single-main-0d89d1-openshift-default-79f4655d69477wq"
Nov 03 20:48:35.009956 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:48:35.009936    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f89ad58b-cab6-4558-aa07-24370249825a-istio-token\") pod \"e2e-worker-single-main-0d89d1-openshift-default-79f4655d69477wq\" (UID: \"f89ad58b-cab6-4558-aa07-24370249825a\") " pod="openshift-ingress/e2e-worker-single-main-0d89d1-openshift-default-79f4655d69477wq"
Nov 03 20:48:35.116090 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:48:35.116061    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-0d89d1-openshift-default-79f4655d69477wq"
Nov 03 20:48:35.427151 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:48:35.427125    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-0d89d1-openshift-default-79f4655d69477wq"]
Nov 03 20:48:35.429522 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:48:35.429496    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf89ad58b_cab6_4558_aa07_24370249825a.slice/crio-3438749eb1a356a1896bd33ddde0f211a8d917523ffe1d0ec9d1c2e0c484d426 WatchSource:0}: Error finding container 3438749eb1a356a1896bd33ddde0f211a8d917523ffe1d0ec9d1c2e0c484d426: Status 404 returned error can't find the container with id 3438749eb1a356a1896bd33ddde0f211a8d917523ffe1d0ec9d1c2e0c484d426
Nov 03 20:48:35.431561 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:48:35.431530    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:48:35.431654 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:48:35.431610    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:48:35.431654 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:48:35.431636    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:48:35.732355 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:48:35.732317    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-0d89d1-openshift-default-79f4655d69477wq" event={"ID":"f89ad58b-cab6-4558-aa07-24370249825a","Type":"ContainerStarted","Data":"c99ab1a36dfc5e8e2ef30dee156ccb0b745f347fc1a8e936a546da805e8f0898"}
Nov 03 20:48:35.732561 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:48:35.732371    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-0d89d1-openshift-default-79f4655d69477wq" event={"ID":"f89ad58b-cab6-4558-aa07-24370249825a","Type":"ContainerStarted","Data":"3438749eb1a356a1896bd33ddde0f211a8d917523ffe1d0ec9d1c2e0c484d426"}
Nov 03 20:48:35.778437 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:48:35.778382    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-0d89d1-openshift-default-79f4655d69477wq" podStartSLOduration=1.7783678 podStartE2EDuration="1.7783678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:48:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:48:35.777285961 +0000 UTC m=+2649.929947848" watchObservedRunningTime="2026-11-03 20:48:35.7783678 +0000 UTC m=+2649.931029685"
Nov 03 20:48:36.116886 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:48:36.116809    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-0d89d1-openshift-default-79f4655d69477wq"
Nov 03 20:48:36.121653 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:48:36.121627    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-0d89d1-openshift-default-79f4655d69477wq"
Nov 03 20:48:36.737591 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:48:36.737558    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-0d89d1-openshift-default-79f4655d69477wq"
Nov 03 20:48:36.738426 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:48:36.738401    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-0d89d1-openshift-default-79f4655d69477wq"
Nov 03 20:48:39.938546 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:48:39.938515    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-0d89d1-58cbc9888472n8"]
Nov 03 20:48:39.941442 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:48:39.941419    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-0d89d1-58cbc9888472n8"
Nov 03 20:48:39.964414 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:48:39.964385    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-0d89d1-58cbc9888472n8"]
Nov 03 20:48:39.970671 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:48:39.970652    5570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-0d89d1\""
Nov 03 20:48:40.002195 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:48:40.002170    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-0d89d1-75ff75nj52"]
Nov 03 20:48:40.008201 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:48:40.008174    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0d89d1-75ff75nj52"
Nov 03 20:48:40.023736 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:48:40.023711    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-0d89d1-75ff75nj52"]
Nov 03 20:48:40.063800 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:48:40.053757    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-3c501ae5-dockercfg-b9v72\""
Nov 03 20:48:40.097008 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:48:40.096968    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r75nw\" (UniqueName: \"kubernetes.io/projected/33f2f1cf-adcb-4fb6-b1aa-c5c4e3268ac5-kube-api-access-r75nw\") pod \"payload-processing-e2e-worker-single-main-0d89d1-58cbc9888472n8\" (UID: \"33f2f1cf-adcb-4fb6-b1aa-c5c4e3268ac5\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-0d89d1-58cbc9888472n8"
Nov 03 20:48:40.097306 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:48:40.097282    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33f2f1cf-adcb-4fb6-b1aa-c5c4e3268ac5-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-0d89d1-58cbc9888472n8\" (UID: \"33f2f1cf-adcb-4fb6-b1aa-c5c4e3268ac5\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-0d89d1-58cbc9888472n8"
Nov 03 20:48:40.097676 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:48:40.097650    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xfx5\" (UniqueName: \"kubernetes.io/projected/f17a6904-d1c6-4e83-9b02-58ad3a1a619a-kube-api-access-7xfx5\") pod \"payload-pre-processing-e2e-worker-single-main-0d89d1-75ff75nj52\" (UID: \"f17a6904-d1c6-4e83-9b02-58ad3a1a619a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0d89d1-75ff75nj52"
Nov 03 20:48:40.097960 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:48:40.097938    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f17a6904-d1c6-4e83-9b02-58ad3a1a619a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-0d89d1-75ff75nj52\" (UID: \"f17a6904-d1c6-4e83-9b02-58ad3a1a619a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0d89d1-75ff75nj52"
Nov 03 20:48:40.198873 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:48:40.198767    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f17a6904-d1c6-4e83-9b02-58ad3a1a619a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-0d89d1-75ff75nj52\" (UID: \"f17a6904-d1c6-4e83-9b02-58ad3a1a619a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0d89d1-75ff75nj52"
Nov 03 20:48:40.198873 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:48:40.198857    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r75nw\" (UniqueName: \"kubernetes.io/projected/33f2f1cf-adcb-4fb6-b1aa-c5c4e3268ac5-kube-api-access-r75nw\") pod \"payload-processing-e2e-worker-single-main-0d89d1-58cbc9888472n8\" (UID: \"33f2f1cf-adcb-4fb6-b1aa-c5c4e3268ac5\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-0d89d1-58cbc9888472n8"
Nov 03 20:48:40.199068 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:48:40.198885    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33f2f1cf-adcb-4fb6-b1aa-c5c4e3268ac5-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-0d89d1-58cbc9888472n8\" (UID: \"33f2f1cf-adcb-4fb6-b1aa-c5c4e3268ac5\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-0d89d1-58cbc9888472n8"
Nov 03 20:48:40.199068 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:48:40.198929    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xfx5\" (UniqueName: \"kubernetes.io/projected/f17a6904-d1c6-4e83-9b02-58ad3a1a619a-kube-api-access-7xfx5\") pod \"payload-pre-processing-e2e-worker-single-main-0d89d1-75ff75nj52\" (UID: \"f17a6904-d1c6-4e83-9b02-58ad3a1a619a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0d89d1-75ff75nj52"
Nov 03 20:48:40.199804 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:48:40.199766    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f17a6904-d1c6-4e83-9b02-58ad3a1a619a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-0d89d1-75ff75nj52\" (UID: \"f17a6904-d1c6-4e83-9b02-58ad3a1a619a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0d89d1-75ff75nj52"
Nov 03 20:48:40.200522 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:48:40.200504    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33f2f1cf-adcb-4fb6-b1aa-c5c4e3268ac5-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-0d89d1-58cbc9888472n8\" (UID: \"33f2f1cf-adcb-4fb6-b1aa-c5c4e3268ac5\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-0d89d1-58cbc9888472n8"
Nov 03 20:48:40.227604 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:48:40.227575    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r75nw\" (UniqueName: \"kubernetes.io/projected/33f2f1cf-adcb-4fb6-b1aa-c5c4e3268ac5-kube-api-access-r75nw\") pod \"payload-processing-e2e-worker-single-main-0d89d1-58cbc9888472n8\" (UID: \"33f2f1cf-adcb-4fb6-b1aa-c5c4e3268ac5\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-0d89d1-58cbc9888472n8"
Nov 03 20:48:40.229532 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:48:40.229500    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xfx5\" (UniqueName: \"kubernetes.io/projected/f17a6904-d1c6-4e83-9b02-58ad3a1a619a-kube-api-access-7xfx5\") pod \"payload-pre-processing-e2e-worker-single-main-0d89d1-75ff75nj52\" (UID: \"f17a6904-d1c6-4e83-9b02-58ad3a1a619a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0d89d1-75ff75nj52"
Nov 03 20:48:40.253934 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:48:40.253909    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-0d89d1-58cbc9888472n8"
Nov 03 20:48:40.322566 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:48:40.322511    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0d89d1-75ff75nj52"
Nov 03 20:48:40.544584 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:48:40.544555    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-0d89d1-58cbc9888472n8"]
Nov 03 20:48:40.544957 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:48:40.544934    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33f2f1cf_adcb_4fb6_b1aa_c5c4e3268ac5.slice/crio-be34c112ee7080ec9cbc5bc373bdf0eced495a65e328fe0cd9f444bb48ba00ea WatchSource:0}: Error finding container be34c112ee7080ec9cbc5bc373bdf0eced495a65e328fe0cd9f444bb48ba00ea: Status 404 returned error can't find the container with id be34c112ee7080ec9cbc5bc373bdf0eced495a65e328fe0cd9f444bb48ba00ea
Nov 03 20:48:40.620965 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:48:40.620907    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-0d89d1-75ff75nj52"]
Nov 03 20:48:40.621100 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:48:40.621067    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf17a6904_d1c6_4e83_9b02_58ad3a1a619a.slice/crio-9bd65d34910c9e824ff3935e5394b3b2056ece324bad51756ff41999597c0903 WatchSource:0}: Error finding container 9bd65d34910c9e824ff3935e5394b3b2056ece324bad51756ff41999597c0903: Status 404 returned error can't find the container with id 9bd65d34910c9e824ff3935e5394b3b2056ece324bad51756ff41999597c0903
Nov 03 20:48:40.761313 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:48:40.761267    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-0d89d1-58cbc9888472n8" event={"ID":"33f2f1cf-adcb-4fb6-b1aa-c5c4e3268ac5","Type":"ContainerStarted","Data":"211c27d176435fd0d564b11c667084225229c6a83d2245de4ed4d27740ed7668"}
Nov 03 20:48:40.761507 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:48:40.761329    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-0d89d1-58cbc9888472n8" event={"ID":"33f2f1cf-adcb-4fb6-b1aa-c5c4e3268ac5","Type":"ContainerStarted","Data":"be34c112ee7080ec9cbc5bc373bdf0eced495a65e328fe0cd9f444bb48ba00ea"}
Nov 03 20:48:40.761507 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:48:40.761458    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-0d89d1-58cbc9888472n8"
Nov 03 20:48:40.762828 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:48:40.762804    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0d89d1-75ff75nj52" event={"ID":"f17a6904-d1c6-4e83-9b02-58ad3a1a619a","Type":"ContainerStarted","Data":"25d2630950ed5301175d36b8af705e163c3a34e938fead480d6be05d1fe820e1"}
Nov 03 20:48:40.762937 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:48:40.762840    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0d89d1-75ff75nj52" event={"ID":"f17a6904-d1c6-4e83-9b02-58ad3a1a619a","Type":"ContainerStarted","Data":"9bd65d34910c9e824ff3935e5394b3b2056ece324bad51756ff41999597c0903"}
Nov 03 20:48:40.762937 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:48:40.762889    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0d89d1-75ff75nj52"
Nov 03 20:48:40.803193 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:48:40.803103    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-0d89d1-58cbc9888472n8" podStartSLOduration=1.8030891549999999 podStartE2EDuration="1.803089155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:48:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:48:40.801985842 +0000 UTC m=+2654.954647725" watchObservedRunningTime="2026-11-03 20:48:40.803089155 +0000 UTC m=+2654.955751080"
Nov 03 20:48:40.841726 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:48:40.841676    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0d89d1-75ff75nj52" podStartSLOduration=1.841663542 podStartE2EDuration="1.841663542s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:48:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:48:40.839838121 +0000 UTC m=+2654.992500004" watchObservedRunningTime="2026-11-03 20:48:40.841663542 +0000 UTC m=+2654.994325502"
Nov 03 20:48:43.340176 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:48:43.340133    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g69f8"]
Nov 03 20:48:51.771600 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:48:51.771570    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0d89d1-75ff75nj52"
Nov 03 20:48:51.772044 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:48:51.771643    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-0d89d1-58cbc9888472n8"
Nov 03 20:48:52.778949 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:48:52.778909    5570 patch_prober.go:29] interesting pod/e2e-worker-single-main-0d89d1-openshift-default-79f4655d69477wq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.214:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:48:52.779365 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:48:52.778995    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-0d89d1-openshift-default-79f4655d69477wq" podUID="f89ad58b-cab6-4558-aa07-24370249825a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.214:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:48:53.836720 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:48:53.836686    5570 generic.go:356] "Generic (PLEG): container finished" podID="f89ad58b-cab6-4558-aa07-24370249825a" containerID="c99ab1a36dfc5e8e2ef30dee156ccb0b745f347fc1a8e936a546da805e8f0898" exitCode=0
Nov 03 20:48:53.837325 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:48:53.836765    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-0d89d1-openshift-default-79f4655d69477wq" event={"ID":"f89ad58b-cab6-4558-aa07-24370249825a","Type":"ContainerDied","Data":"c99ab1a36dfc5e8e2ef30dee156ccb0b745f347fc1a8e936a546da805e8f0898"}
Nov 03 20:48:53.837409 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:48:53.837359    5570 scope.go:122] "RemoveContainer" containerID="c99ab1a36dfc5e8e2ef30dee156ccb0b745f347fc1a8e936a546da805e8f0898"
Nov 03 20:48:53.838590 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:48:53.838561    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:48:53.838659 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:48:53.838631    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:48:53.838695 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:48:53.838661    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:48:54.868080 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:48:54.868040    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-0d89d1-openshift-default-79f4655d69477wq" event={"ID":"f89ad58b-cab6-4558-aa07-24370249825a","Type":"ContainerStarted","Data":"629bd85b89942e12c878358081fee731f8bdfca0a88bf84f051a7b1432a3566b"}
Nov 03 20:48:55.116511 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:48:55.116471    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-0d89d1-openshift-default-79f4655d69477wq"
Nov 03 20:48:55.116726 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:48:55.116702    5570 patch_prober.go:29] interesting pod/e2e-worker-single-main-0d89d1-openshift-default-79f4655d69477wq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused" start-of-body=
Nov 03 20:48:55.116804 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:48:55.116761    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-0d89d1-openshift-default-79f4655d69477wq" podUID="f89ad58b-cab6-4558-aa07-24370249825a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused"
Nov 03 20:48:56.116844 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:48:56.116809    5570 patch_prober.go:29] interesting pod/e2e-worker-single-main-0d89d1-openshift-default-79f4655d69477wq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused" start-of-body=
Nov 03 20:48:56.117262 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:48:56.116869    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-0d89d1-openshift-default-79f4655d69477wq" podUID="f89ad58b-cab6-4558-aa07-24370249825a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused"
Nov 03 20:48:57.116953 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:48:57.116917    5570 patch_prober.go:29] interesting pod/e2e-worker-single-main-0d89d1-openshift-default-79f4655d69477wq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused" start-of-body=
Nov 03 20:48:57.117387 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:48:57.116971    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-0d89d1-openshift-default-79f4655d69477wq" podUID="f89ad58b-cab6-4558-aa07-24370249825a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused"
Nov 03 20:48:57.226622 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:48:57.226589    5570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8bwhz" containerName="extractor"
Nov 03 20:48:57.666905 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:48:57.666834    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g69f8"]
Nov 03 20:48:58.116677 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:48:58.116635    5570 patch_prober.go:29] interesting pod/e2e-worker-single-main-0d89d1-openshift-default-79f4655d69477wq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused" start-of-body=
Nov 03 20:48:58.116931 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:48:58.116708    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-0d89d1-openshift-default-79f4655d69477wq" podUID="f89ad58b-cab6-4558-aa07-24370249825a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused"
Nov 03 20:48:59.117322 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:48:59.117286    5570 patch_prober.go:29] interesting pod/e2e-worker-single-main-0d89d1-openshift-default-79f4655d69477wq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused" start-of-body=
Nov 03 20:48:59.117733 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:48:59.117339    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-0d89d1-openshift-default-79f4655d69477wq" podUID="f89ad58b-cab6-4558-aa07-24370249825a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused"
Nov 03 20:49:00.197294 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:49:00.197257    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-0d89d1-openshift-default-79f4655d69477wq"
Nov 03 20:49:00.198080 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:49:00.198055    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-0d89d1-openshift-default-79f4655d69477wq"
Nov 03 20:49:00.198895 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:49:00.198873    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-0d89d1-openshift-default-79f4655d69477wq"
Nov 03 20:49:04.040818 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:49:04.040733    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-pbxdh"]
Nov 03 20:49:04.043789 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:49:04.043748    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-pbxdh"]
Nov 03 20:49:05.964161 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:49:05.964127    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e07a89e4-f07f-4a03-ba49-f310f25343a0" path="/var/lib/kubelet/pods/e07a89e4-f07f-4a03-ba49-f310f25343a0/volumes"
Nov 03 20:49:14.270634 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:49:14.270589    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g69f8"]
Nov 03 20:49:15.025445 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:49:15.025397    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-qldvw"]
Nov 03 20:49:15.038515 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:49:15.038486    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-qldvw"]
Nov 03 20:49:15.962662 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:49:15.962625    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9fa4c49-b441-489d-b635-b483a1eddeb0" path="/var/lib/kubelet/pods/c9fa4c49-b441-489d-b635-b483a1eddeb0/volumes"
Nov 03 20:49:27.227706 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:49:27.227673    5570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8bwhz" containerName="extractor"
Nov 03 20:49:27.274103 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:49:27.274075    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f2d30bd-v7wbx"]
Nov 03 20:49:27.277340 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:49:27.277323    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f2d30bd-v7wbx"
Nov 03 20:49:27.294125 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:49:27.294097    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f2d30bd-v7wbx"]
Nov 03 20:49:27.305089 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:49:27.305064    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:49:27.305230 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:49:27.305089    5570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 20:49:27.305230 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:49:27.305138    5570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 20:49:27.306905 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:49:27.306884    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l7s5w\""
Nov 03 20:49:27.360433 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:49:27.360402    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87f3acb3-331f-40a9-a7bd-f2d5b4ad3fc4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3f2d30bd-v7wbx\" (UID: \"87f3acb3-331f-40a9-a7bd-f2d5b4ad3fc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f2d30bd-v7wbx"
Nov 03 20:49:27.360573 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:49:27.360556    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87f3acb3-331f-40a9-a7bd-f2d5b4ad3fc4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3f2d30bd-v7wbx\" (UID: \"87f3acb3-331f-40a9-a7bd-f2d5b4ad3fc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f2d30bd-v7wbx"
Nov 03 20:49:27.360624 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:49:27.360588    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cr2w\" (UniqueName: \"kubernetes.io/projected/87f3acb3-331f-40a9-a7bd-f2d5b4ad3fc4-kube-api-access-9cr2w\") pod \"maas-api-revoke-sub-3f2d30bd-v7wbx\" (UID: \"87f3acb3-331f-40a9-a7bd-f2d5b4ad3fc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f2d30bd-v7wbx"
Nov 03 20:49:27.461706 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:49:27.461674    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87f3acb3-331f-40a9-a7bd-f2d5b4ad3fc4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3f2d30bd-v7wbx\" (UID: \"87f3acb3-331f-40a9-a7bd-f2d5b4ad3fc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f2d30bd-v7wbx"
Nov 03 20:49:27.461706 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:49:27.461704    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cr2w\" (UniqueName: \"kubernetes.io/projected/87f3acb3-331f-40a9-a7bd-f2d5b4ad3fc4-kube-api-access-9cr2w\") pod \"maas-api-revoke-sub-3f2d30bd-v7wbx\" (UID: \"87f3acb3-331f-40a9-a7bd-f2d5b4ad3fc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f2d30bd-v7wbx"
Nov 03 20:49:27.461958 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:49:27.461782    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87f3acb3-331f-40a9-a7bd-f2d5b4ad3fc4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3f2d30bd-v7wbx\" (UID: \"87f3acb3-331f-40a9-a7bd-f2d5b4ad3fc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f2d30bd-v7wbx"
Nov 03 20:49:27.462333 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:49:27.462305    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87f3acb3-331f-40a9-a7bd-f2d5b4ad3fc4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3f2d30bd-v7wbx\" (UID: \"87f3acb3-331f-40a9-a7bd-f2d5b4ad3fc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f2d30bd-v7wbx"
Nov 03 20:49:27.464029 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:49:27.464005    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87f3acb3-331f-40a9-a7bd-f2d5b4ad3fc4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3f2d30bd-v7wbx\" (UID: \"87f3acb3-331f-40a9-a7bd-f2d5b4ad3fc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f2d30bd-v7wbx"
Nov 03 20:49:27.481528 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:49:27.481473    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cr2w\" (UniqueName: \"kubernetes.io/projected/87f3acb3-331f-40a9-a7bd-f2d5b4ad3fc4-kube-api-access-9cr2w\") pod \"maas-api-revoke-sub-3f2d30bd-v7wbx\" (UID: \"87f3acb3-331f-40a9-a7bd-f2d5b4ad3fc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f2d30bd-v7wbx"
Nov 03 20:49:27.590533 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:49:27.590501    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f2d30bd-v7wbx"
Nov 03 20:49:27.888412 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:49:27.888387    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f2d30bd-v7wbx"]
Nov 03 20:49:27.889494 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:49:27.889468    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87f3acb3_331f_40a9_a7bd_f2d5b4ad3fc4.slice/crio-389f2827fc527b556f784884c12c9fdc2af42357771f6d21d5691a1c50024a2b WatchSource:0}: Error finding container 389f2827fc527b556f784884c12c9fdc2af42357771f6d21d5691a1c50024a2b: Status 404 returned error can't find the container with id 389f2827fc527b556f784884c12c9fdc2af42357771f6d21d5691a1c50024a2b
Nov 03 20:49:28.045580 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:49:28.045550    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f2d30bd-v7wbx" event={"ID":"87f3acb3-331f-40a9-a7bd-f2d5b4ad3fc4","Type":"ContainerStarted","Data":"389f2827fc527b556f784884c12c9fdc2af42357771f6d21d5691a1c50024a2b"}
Nov 03 20:49:29.052556 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:49:29.052472    5570 generic.go:356] "Generic (PLEG): container finished" podID="87f3acb3-331f-40a9-a7bd-f2d5b4ad3fc4" containerID="b5d714f5f9dc7b9de25cb9e37efb202473433323679acdf66cbb2ac8594aa980" exitCode=0
Nov 03 20:49:29.052960 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:49:29.052559    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f2d30bd-v7wbx" event={"ID":"87f3acb3-331f-40a9-a7bd-f2d5b4ad3fc4","Type":"ContainerDied","Data":"b5d714f5f9dc7b9de25cb9e37efb202473433323679acdf66cbb2ac8594aa980"}
Nov 03 20:49:30.198105 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:49:30.198078    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f2d30bd-v7wbx"
Nov 03 20:49:30.289370 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:49:30.289336    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87f3acb3-331f-40a9-a7bd-f2d5b4ad3fc4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87f3acb3-331f-40a9-a7bd-f2d5b4ad3fc4-maas-api-openshift-ca\") pod \"87f3acb3-331f-40a9-a7bd-f2d5b4ad3fc4\" (UID: \"87f3acb3-331f-40a9-a7bd-f2d5b4ad3fc4\") "
Nov 03 20:49:30.289546 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:49:30.289406    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87f3acb3-331f-40a9-a7bd-f2d5b4ad3fc4-kube-api-access-9cr2w\" (UniqueName: \"kubernetes.io/projected/87f3acb3-331f-40a9-a7bd-f2d5b4ad3fc4-kube-api-access-9cr2w\") pod \"87f3acb3-331f-40a9-a7bd-f2d5b4ad3fc4\" (UID: \"87f3acb3-331f-40a9-a7bd-f2d5b4ad3fc4\") "
Nov 03 20:49:30.289546 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:49:30.289449    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/87f3acb3-331f-40a9-a7bd-f2d5b4ad3fc4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87f3acb3-331f-40a9-a7bd-f2d5b4ad3fc4-maas-api-xks-ca\") pod \"87f3acb3-331f-40a9-a7bd-f2d5b4ad3fc4\" (UID: \"87f3acb3-331f-40a9-a7bd-f2d5b4ad3fc4\") "
Nov 03 20:49:30.289706 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:49:30.289682    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87f3acb3-331f-40a9-a7bd-f2d5b4ad3fc4-maas-api-openshift-ca" pod "87f3acb3-331f-40a9-a7bd-f2d5b4ad3fc4" (UID: "87f3acb3-331f-40a9-a7bd-f2d5b4ad3fc4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:49:30.291488 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:49:30.291463    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87f3acb3-331f-40a9-a7bd-f2d5b4ad3fc4-maas-api-xks-ca" pod "87f3acb3-331f-40a9-a7bd-f2d5b4ad3fc4" (UID: "87f3acb3-331f-40a9-a7bd-f2d5b4ad3fc4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:49:30.291606 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:49:30.291554    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87f3acb3-331f-40a9-a7bd-f2d5b4ad3fc4-kube-api-access-9cr2w" pod "87f3acb3-331f-40a9-a7bd-f2d5b4ad3fc4" (UID: "87f3acb3-331f-40a9-a7bd-f2d5b4ad3fc4"). InnerVolumeSpecName "kube-api-access-9cr2w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:49:30.391017 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:49:30.390950    5570 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87f3acb3-331f-40a9-a7bd-f2d5b4ad3fc4-maas-api-openshift-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:49:30.391017 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:49:30.390984    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9cr2w\" (UniqueName: \"kubernetes.io/projected/87f3acb3-331f-40a9-a7bd-f2d5b4ad3fc4-kube-api-access-9cr2w\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:49:30.391017 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:49:30.391004    5570 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87f3acb3-331f-40a9-a7bd-f2d5b4ad3fc4-maas-api-xks-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:49:31.063415 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:49:31.063384    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f2d30bd-v7wbx"
Nov 03 20:49:31.063602 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:49:31.063411    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f2d30bd-v7wbx" event={"ID":"87f3acb3-331f-40a9-a7bd-f2d5b4ad3fc4","Type":"ContainerDied","Data":"389f2827fc527b556f784884c12c9fdc2af42357771f6d21d5691a1c50024a2b"}
Nov 03 20:49:31.063602 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:49:31.063462    5570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="389f2827fc527b556f784884c12c9fdc2af42357771f6d21d5691a1c50024a2b"
Nov 03 20:49:32.362192 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:49:32.362165    5570 scope.go:122] "RemoveContainer" containerID="a94392be6662a165b2ec36410dfb91d76f3b80c6a8f08efb99ddba52df5ada1a"
Nov 03 20:49:32.374316 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:49:32.374292    5570 scope.go:122] "RemoveContainer" containerID="f1c9d6f66d29f0c3c930fb4dec17b43a9739ee52ae8954d58967fd6f30f1db56"
Nov 03 20:49:32.382463 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:49:32.382443    5570 scope.go:122] "RemoveContainer" containerID="278fc687282538009a3be94bba855d1f3814b710572018392f1fe809703c6a96"
Nov 03 20:49:32.390179 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:49:32.390156    5570 scope.go:122] "RemoveContainer" containerID="db16a74203a9d0b80d98387b7dd504dd91b24824dff2ae882fda264c4a09475d"
Nov 03 20:49:38.317299 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:49:38.317212    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0d89d1-swmwv"]
Nov 03 20:49:38.317912 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:49:38.317885    5570 cpu_manager.go:414] "containerMap: removing container" podUID="87f3acb3-331f-40a9-a7bd-f2d5b4ad3fc4" containerName="revoke-keys"
Nov 03 20:49:38.317912 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:49:38.317907    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87f3acb3-331f-40a9-a7bd-f2d5b4ad3fc4" containerName="revoke-keys"
Nov 03 20:49:38.318063 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:49:38.318027    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="87f3acb3-331f-40a9-a7bd-f2d5b4ad3fc4" containerName="revoke-keys"
Nov 03 20:49:38.322791 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:49:38.322750    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0d89d1-swmwv"
Nov 03 20:49:38.336744 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:49:38.336717    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0d89d1-swmwv"]
Nov 03 20:49:38.347516 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:49:38.347492    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:49:38.347688 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:49:38.347493    5570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 20:49:38.347688 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:49:38.347503    5570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 20:49:38.352196 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:49:38.352180    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l7s5w\""
Nov 03 20:49:38.382861 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:49:38.382836    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf1c2473-a812-46bd-8bae-d736b25e002c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-0d89d1-swmwv\" (UID: \"cf1c2473-a812-46bd-8bae-d736b25e002c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0d89d1-swmwv"
Nov 03 20:49:38.382990 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:49:38.382878    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75lbw\" (UniqueName: \"kubernetes.io/projected/cf1c2473-a812-46bd-8bae-d736b25e002c-kube-api-access-75lbw\") pod \"maas-api-revoke-keys-e2e-worker-single-main-0d89d1-swmwv\" (UID: \"cf1c2473-a812-46bd-8bae-d736b25e002c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0d89d1-swmwv"
Nov 03 20:49:38.382990 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:49:38.382912    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf1c2473-a812-46bd-8bae-d736b25e002c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-0d89d1-swmwv\" (UID: \"cf1c2473-a812-46bd-8bae-d736b25e002c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0d89d1-swmwv"
Nov 03 20:49:38.483675 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:49:38.483638    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75lbw\" (UniqueName: \"kubernetes.io/projected/cf1c2473-a812-46bd-8bae-d736b25e002c-kube-api-access-75lbw\") pod \"maas-api-revoke-keys-e2e-worker-single-main-0d89d1-swmwv\" (UID: \"cf1c2473-a812-46bd-8bae-d736b25e002c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0d89d1-swmwv"
Nov 03 20:49:38.483852 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:49:38.483690    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf1c2473-a812-46bd-8bae-d736b25e002c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-0d89d1-swmwv\" (UID: \"cf1c2473-a812-46bd-8bae-d736b25e002c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0d89d1-swmwv"
Nov 03 20:49:38.483852 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:49:38.483810    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf1c2473-a812-46bd-8bae-d736b25e002c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-0d89d1-swmwv\" (UID: \"cf1c2473-a812-46bd-8bae-d736b25e002c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0d89d1-swmwv"
Nov 03 20:49:38.484387 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:49:38.484354    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf1c2473-a812-46bd-8bae-d736b25e002c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-0d89d1-swmwv\" (UID: \"cf1c2473-a812-46bd-8bae-d736b25e002c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0d89d1-swmwv"
Nov 03 20:49:38.486132 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:49:38.486111    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf1c2473-a812-46bd-8bae-d736b25e002c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-0d89d1-swmwv\" (UID: \"cf1c2473-a812-46bd-8bae-d736b25e002c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0d89d1-swmwv"
Nov 03 20:49:38.504041 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:49:38.504019    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75lbw\" (UniqueName: \"kubernetes.io/projected/cf1c2473-a812-46bd-8bae-d736b25e002c-kube-api-access-75lbw\") pod \"maas-api-revoke-keys-e2e-worker-single-main-0d89d1-swmwv\" (UID: \"cf1c2473-a812-46bd-8bae-d736b25e002c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0d89d1-swmwv"
Nov 03 20:49:38.637498 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:49:38.637425    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0d89d1-swmwv"
Nov 03 20:49:38.739286 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:49:38.739246    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g69f8"]
Nov 03 20:49:38.935309 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:49:38.935280    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0d89d1-swmwv"]
Nov 03 20:49:38.935874 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:49:38.935842    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf1c2473_a812_46bd_8bae_d736b25e002c.slice/crio-b3567cc548e2593f9701157cc3bf7770ff17613c61b683f202dcd84af3a8bb4b WatchSource:0}: Error finding container b3567cc548e2593f9701157cc3bf7770ff17613c61b683f202dcd84af3a8bb4b: Status 404 returned error can't find the container with id b3567cc548e2593f9701157cc3bf7770ff17613c61b683f202dcd84af3a8bb4b
Nov 03 20:49:39.107046 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:49:39.107008    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0d89d1-swmwv" event={"ID":"cf1c2473-a812-46bd-8bae-d736b25e002c","Type":"ContainerStarted","Data":"b3567cc548e2593f9701157cc3bf7770ff17613c61b683f202dcd84af3a8bb4b"}
Nov 03 20:49:40.113798 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:49:40.113696    5570 generic.go:356] "Generic (PLEG): container finished" podID="cf1c2473-a812-46bd-8bae-d736b25e002c" containerID="d2542e1e84125cfe5c2172dd3c025299ed6d67955ab79c3866c39db032016fc8" exitCode=0
Nov 03 20:49:40.113798 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:49:40.113766    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0d89d1-swmwv" event={"ID":"cf1c2473-a812-46bd-8bae-d736b25e002c","Type":"ContainerDied","Data":"d2542e1e84125cfe5c2172dd3c025299ed6d67955ab79c3866c39db032016fc8"}
Nov 03 20:49:41.270456 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:49:41.270431    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0d89d1-swmwv"
Nov 03 20:49:41.312305 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:49:41.312273    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf1c2473-a812-46bd-8bae-d736b25e002c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf1c2473-a812-46bd-8bae-d736b25e002c-maas-api-xks-ca\") pod \"cf1c2473-a812-46bd-8bae-d736b25e002c\" (UID: \"cf1c2473-a812-46bd-8bae-d736b25e002c\") "
Nov 03 20:49:41.312444 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:49:41.312330    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf1c2473-a812-46bd-8bae-d736b25e002c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf1c2473-a812-46bd-8bae-d736b25e002c-maas-api-openshift-ca\") pod \"cf1c2473-a812-46bd-8bae-d736b25e002c\" (UID: \"cf1c2473-a812-46bd-8bae-d736b25e002c\") "
Nov 03 20:49:41.312444 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:49:41.312427    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf1c2473-a812-46bd-8bae-d736b25e002c-kube-api-access-75lbw\" (UniqueName: \"kubernetes.io/projected/cf1c2473-a812-46bd-8bae-d736b25e002c-kube-api-access-75lbw\") pod \"cf1c2473-a812-46bd-8bae-d736b25e002c\" (UID: \"cf1c2473-a812-46bd-8bae-d736b25e002c\") "
Nov 03 20:49:41.312735 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:49:41.312707    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf1c2473-a812-46bd-8bae-d736b25e002c-maas-api-openshift-ca" pod "cf1c2473-a812-46bd-8bae-d736b25e002c" (UID: "cf1c2473-a812-46bd-8bae-d736b25e002c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:49:41.314468 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:49:41.314443    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf1c2473-a812-46bd-8bae-d736b25e002c-kube-api-access-75lbw" pod "cf1c2473-a812-46bd-8bae-d736b25e002c" (UID: "cf1c2473-a812-46bd-8bae-d736b25e002c"). InnerVolumeSpecName "kube-api-access-75lbw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:49:41.314567 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:49:41.314542    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf1c2473-a812-46bd-8bae-d736b25e002c-maas-api-xks-ca" pod "cf1c2473-a812-46bd-8bae-d736b25e002c" (UID: "cf1c2473-a812-46bd-8bae-d736b25e002c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:49:41.413950 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:49:41.413890    5570 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf1c2473-a812-46bd-8bae-d736b25e002c-maas-api-xks-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:49:41.413950 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:49:41.413914    5570 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf1c2473-a812-46bd-8bae-d736b25e002c-maas-api-openshift-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:49:41.413950 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:49:41.413923    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75lbw\" (UniqueName: \"kubernetes.io/projected/cf1c2473-a812-46bd-8bae-d736b25e002c-kube-api-access-75lbw\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:49:42.130494 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:49:42.130424    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0d89d1-swmwv" event={"ID":"cf1c2473-a812-46bd-8bae-d736b25e002c","Type":"ContainerDied","Data":"b3567cc548e2593f9701157cc3bf7770ff17613c61b683f202dcd84af3a8bb4b"}
Nov 03 20:49:42.130694 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:49:42.130547    5570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b3567cc548e2593f9701157cc3bf7770ff17613c61b683f202dcd84af3a8bb4b"
Nov 03 20:49:42.130694 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:49:42.130554    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0d89d1-swmwv"
Nov 03 20:49:48.757816 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:49:48.757767    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-0d89d1-58cbc9888472n8"]
Nov 03 20:49:48.788931 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:49:48.788892    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-0d89d1-75ff75nj52"]
Nov 03 20:49:51.628454 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:49:51.628290    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0d89d1-swmwv"]
Nov 03 20:49:51.649936 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:49:51.649894    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0d89d1-swmwv"]
Nov 03 20:49:51.963743 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:49:51.963709    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf1c2473-a812-46bd-8bae-d736b25e002c" path="/var/lib/kubelet/pods/cf1c2473-a812-46bd-8bae-d736b25e002c/volumes"
Nov 03 20:49:53.347795 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:49:53.346084    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g69f8"]
Nov 03 20:49:53.758879 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:49:53.758841    5570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-0d89d1-58cbc9888472n8" podUID="33f2f1cf-adcb-4fb6-b1aa-c5c4e3268ac5" containerName="payload-processing" containerID="cri-o://211c27d176435fd0d564b11c667084225229c6a83d2245de4ed4d27740ed7668" gracePeriod=25
Nov 03 20:49:53.789677 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:49:53.789642    5570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0d89d1-75ff75nj52" podUID="f17a6904-d1c6-4e83-9b02-58ad3a1a619a" containerName="payload-pre-processing" containerID="cri-o://25d2630950ed5301175d36b8af705e163c3a34e938fead480d6be05d1fe820e1" gracePeriod=25
Nov 03 20:49:54.037015 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:49:54.036981    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-0d89d1-58cbc9888472n8"
Nov 03 20:49:54.077330 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:49:54.077298    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33f2f1cf-adcb-4fb6-b1aa-c5c4e3268ac5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33f2f1cf-adcb-4fb6-b1aa-c5c4e3268ac5-plugins-config-volume\") pod \"33f2f1cf-adcb-4fb6-b1aa-c5c4e3268ac5\" (UID: \"33f2f1cf-adcb-4fb6-b1aa-c5c4e3268ac5\") "
Nov 03 20:49:54.077447 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:49:54.077409    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33f2f1cf-adcb-4fb6-b1aa-c5c4e3268ac5-kube-api-access-r75nw\" (UniqueName: \"kubernetes.io/projected/33f2f1cf-adcb-4fb6-b1aa-c5c4e3268ac5-kube-api-access-r75nw\") pod \"33f2f1cf-adcb-4fb6-b1aa-c5c4e3268ac5\" (UID: \"33f2f1cf-adcb-4fb6-b1aa-c5c4e3268ac5\") "
Nov 03 20:49:54.077699 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:49:54.077671    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33f2f1cf-adcb-4fb6-b1aa-c5c4e3268ac5-plugins-config-volume" pod "33f2f1cf-adcb-4fb6-b1aa-c5c4e3268ac5" (UID: "33f2f1cf-adcb-4fb6-b1aa-c5c4e3268ac5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:49:54.079621 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:49:54.079598    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33f2f1cf-adcb-4fb6-b1aa-c5c4e3268ac5-kube-api-access-r75nw" pod "33f2f1cf-adcb-4fb6-b1aa-c5c4e3268ac5" (UID: "33f2f1cf-adcb-4fb6-b1aa-c5c4e3268ac5"). InnerVolumeSpecName "kube-api-access-r75nw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:49:54.080054 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:49:54.080030    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0d89d1-75ff75nj52"
Nov 03 20:49:54.178597 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:49:54.178568    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f17a6904-d1c6-4e83-9b02-58ad3a1a619a-kube-api-access-7xfx5\" (UniqueName: \"kubernetes.io/projected/f17a6904-d1c6-4e83-9b02-58ad3a1a619a-kube-api-access-7xfx5\") pod \"f17a6904-d1c6-4e83-9b02-58ad3a1a619a\" (UID: \"f17a6904-d1c6-4e83-9b02-58ad3a1a619a\") "
Nov 03 20:49:54.178759 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:49:54.178616    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f17a6904-d1c6-4e83-9b02-58ad3a1a619a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f17a6904-d1c6-4e83-9b02-58ad3a1a619a-plugins-config-volume\") pod \"f17a6904-d1c6-4e83-9b02-58ad3a1a619a\" (UID: \"f17a6904-d1c6-4e83-9b02-58ad3a1a619a\") "
Nov 03 20:49:54.178948 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:49:54.178933    5570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33f2f1cf-adcb-4fb6-b1aa-c5c4e3268ac5-plugins-config-volume\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:49:54.178995 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:49:54.178949    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r75nw\" (UniqueName: \"kubernetes.io/projected/33f2f1cf-adcb-4fb6-b1aa-c5c4e3268ac5-kube-api-access-r75nw\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:49:54.179030 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:49:54.179012    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f17a6904-d1c6-4e83-9b02-58ad3a1a619a-plugins-config-volume" pod "f17a6904-d1c6-4e83-9b02-58ad3a1a619a" (UID: "f17a6904-d1c6-4e83-9b02-58ad3a1a619a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:49:54.180678 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:49:54.180655    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f17a6904-d1c6-4e83-9b02-58ad3a1a619a-kube-api-access-7xfx5" pod "f17a6904-d1c6-4e83-9b02-58ad3a1a619a" (UID: "f17a6904-d1c6-4e83-9b02-58ad3a1a619a"). InnerVolumeSpecName "kube-api-access-7xfx5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:49:54.209583 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:49:54.209554    5570 generic.go:356] "Generic (PLEG): container finished" podID="33f2f1cf-adcb-4fb6-b1aa-c5c4e3268ac5" containerID="211c27d176435fd0d564b11c667084225229c6a83d2245de4ed4d27740ed7668" exitCode=0
Nov 03 20:49:54.209725 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:49:54.209608    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-0d89d1-58cbc9888472n8"
Nov 03 20:49:54.209725 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:49:54.209636    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-0d89d1-58cbc9888472n8" event={"ID":"33f2f1cf-adcb-4fb6-b1aa-c5c4e3268ac5","Type":"ContainerDied","Data":"211c27d176435fd0d564b11c667084225229c6a83d2245de4ed4d27740ed7668"}
Nov 03 20:49:54.209842 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:49:54.209724    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-0d89d1-58cbc9888472n8" event={"ID":"33f2f1cf-adcb-4fb6-b1aa-c5c4e3268ac5","Type":"ContainerDied","Data":"be34c112ee7080ec9cbc5bc373bdf0eced495a65e328fe0cd9f444bb48ba00ea"}
Nov 03 20:49:54.209842 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:49:54.209758    5570 scope.go:122] "RemoveContainer" containerID="211c27d176435fd0d564b11c667084225229c6a83d2245de4ed4d27740ed7668"
Nov 03 20:49:54.211141 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:49:54.211114    5570 generic.go:356] "Generic (PLEG): container finished" podID="f17a6904-d1c6-4e83-9b02-58ad3a1a619a" containerID="25d2630950ed5301175d36b8af705e163c3a34e938fead480d6be05d1fe820e1" exitCode=0
Nov 03 20:49:54.211274 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:49:54.211172    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0d89d1-75ff75nj52"
Nov 03 20:49:54.211274 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:49:54.211203    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0d89d1-75ff75nj52" event={"ID":"f17a6904-d1c6-4e83-9b02-58ad3a1a619a","Type":"ContainerDied","Data":"25d2630950ed5301175d36b8af705e163c3a34e938fead480d6be05d1fe820e1"}
Nov 03 20:49:54.211274 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:49:54.211252    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-0d89d1-75ff75nj52" event={"ID":"f17a6904-d1c6-4e83-9b02-58ad3a1a619a","Type":"ContainerDied","Data":"9bd65d34910c9e824ff3935e5394b3b2056ece324bad51756ff41999597c0903"}
Nov 03 20:49:54.220316 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:49:54.220296    5570 scope.go:122] "RemoveContainer" containerID="211c27d176435fd0d564b11c667084225229c6a83d2245de4ed4d27740ed7668"
Nov 03 20:49:54.220591 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:49:54.220571    5570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"211c27d176435fd0d564b11c667084225229c6a83d2245de4ed4d27740ed7668\": container with ID starting with 211c27d176435fd0d564b11c667084225229c6a83d2245de4ed4d27740ed7668 not found: ID does not exist" containerID="211c27d176435fd0d564b11c667084225229c6a83d2245de4ed4d27740ed7668"
Nov 03 20:49:54.220651 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:49:54.220598    5570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"211c27d176435fd0d564b11c667084225229c6a83d2245de4ed4d27740ed7668"} err="failed to get container status \"211c27d176435fd0d564b11c667084225229c6a83d2245de4ed4d27740ed7668\": rpc error: code = NotFound desc = could not find container \"211c27d176435fd0d564b11c667084225229c6a83d2245de4ed4d27740ed7668\": container with ID starting with 211c27d176435fd0d564b11c667084225229c6a83d2245de4ed4d27740ed7668 not found: ID does not exist"
Nov 03 20:49:54.220651 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:49:54.220615    5570 scope.go:122] "RemoveContainer" containerID="25d2630950ed5301175d36b8af705e163c3a34e938fead480d6be05d1fe820e1"
Nov 03 20:49:54.228830 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:49:54.228811    5570 scope.go:122] "RemoveContainer" containerID="25d2630950ed5301175d36b8af705e163c3a34e938fead480d6be05d1fe820e1"
Nov 03 20:49:54.229074 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:49:54.229056    5570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25d2630950ed5301175d36b8af705e163c3a34e938fead480d6be05d1fe820e1\": container with ID starting with 25d2630950ed5301175d36b8af705e163c3a34e938fead480d6be05d1fe820e1 not found: ID does not exist" containerID="25d2630950ed5301175d36b8af705e163c3a34e938fead480d6be05d1fe820e1"
Nov 03 20:49:54.229139 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:49:54.229079    5570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25d2630950ed5301175d36b8af705e163c3a34e938fead480d6be05d1fe820e1"} err="failed to get container status \"25d2630950ed5301175d36b8af705e163c3a34e938fead480d6be05d1fe820e1\": rpc error: code = NotFound desc = could not find container \"25d2630950ed5301175d36b8af705e163c3a34e938fead480d6be05d1fe820e1\": container with ID starting with 25d2630950ed5301175d36b8af705e163c3a34e938fead480d6be05d1fe820e1 not found: ID does not exist"
Nov 03 20:49:54.265958 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:49:54.265906    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-0d89d1-58cbc9888472n8"]
Nov 03 20:49:54.276079 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:49:54.276055    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-0d89d1-58cbc9888472n8"]
Nov 03 20:49:54.279544 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:49:54.279525    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xfx5\" (UniqueName: \"kubernetes.io/projected/f17a6904-d1c6-4e83-9b02-58ad3a1a619a-kube-api-access-7xfx5\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:49:54.279605 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:49:54.279546    5570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f17a6904-d1c6-4e83-9b02-58ad3a1a619a-plugins-config-volume\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:49:54.328926 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:49:54.328897    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-0d89d1-75ff75nj52"]
Nov 03 20:49:54.334726 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:49:54.334700    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-0d89d1-75ff75nj52"]
Nov 03 20:49:55.964444 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:49:55.964413    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33f2f1cf-adcb-4fb6-b1aa-c5c4e3268ac5" path="/var/lib/kubelet/pods/33f2f1cf-adcb-4fb6-b1aa-c5c4e3268ac5/volumes"
Nov 03 20:49:55.964845 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:49:55.964790    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f17a6904-d1c6-4e83-9b02-58ad3a1a619a" path="/var/lib/kubelet/pods/f17a6904-d1c6-4e83-9b02-58ad3a1a619a/volumes"
Nov 03 20:49:57.226866 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:49:57.226834    5570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8bwhz" containerName="extractor"
Nov 03 20:49:58.117203 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:49:58.117171    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-0d89d1-openshift-default-79f4655d69477wq"]
Nov 03 20:49:58.117428 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:49:58.117396    5570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-0d89d1-openshift-default-79f4655d69477wq" podUID="f89ad58b-cab6-4558-aa07-24370249825a" containerName="istio-proxy" containerID="cri-o://629bd85b89942e12c878358081fee731f8bdfca0a88bf84f051a7b1432a3566b" gracePeriod=30
Nov 03 20:50:00.200715 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:50:00.200675    5570 patch_prober.go:29] interesting pod/e2e-worker-single-main-0d89d1-openshift-default-79f4655d69477wq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 20:50:00.201348 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:50:00.200738    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-0d89d1-openshift-default-79f4655d69477wq" podUID="f89ad58b-cab6-4558-aa07-24370249825a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 20:50:03.279717 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:50:03.279684    5570 generic.go:356] "Generic (PLEG): container finished" podID="f89ad58b-cab6-4558-aa07-24370249825a" containerID="629bd85b89942e12c878358081fee731f8bdfca0a88bf84f051a7b1432a3566b" exitCode=0
Nov 03 20:50:03.280152 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:50:03.279753    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-0d89d1-openshift-default-79f4655d69477wq" event={"ID":"f89ad58b-cab6-4558-aa07-24370249825a","Type":"ContainerDied","Data":"629bd85b89942e12c878358081fee731f8bdfca0a88bf84f051a7b1432a3566b"}
Nov 03 20:50:03.280152 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:50:03.279829    5570 scope.go:122] "RemoveContainer" containerID="c99ab1a36dfc5e8e2ef30dee156ccb0b745f347fc1a8e936a546da805e8f0898"
Nov 03 20:50:03.374996 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:50:03.374975    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-0d89d1-openshift-default-79f4655d69477wq"
Nov 03 20:50:03.491350 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:50:03.491325    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f89ad58b-cab6-4558-aa07-24370249825a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f89ad58b-cab6-4558-aa07-24370249825a-istio-envoy\") pod \"f89ad58b-cab6-4558-aa07-24370249825a\" (UID: \"f89ad58b-cab6-4558-aa07-24370249825a\") "
Nov 03 20:50:03.491350 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:50:03.491358    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f89ad58b-cab6-4558-aa07-24370249825a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f89ad58b-cab6-4558-aa07-24370249825a-istio-data\") pod \"f89ad58b-cab6-4558-aa07-24370249825a\" (UID: \"f89ad58b-cab6-4558-aa07-24370249825a\") "
Nov 03 20:50:03.491557 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:50:03.491393    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f89ad58b-cab6-4558-aa07-24370249825a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f89ad58b-cab6-4558-aa07-24370249825a-workload-socket\") pod \"f89ad58b-cab6-4558-aa07-24370249825a\" (UID: \"f89ad58b-cab6-4558-aa07-24370249825a\") "
Nov 03 20:50:03.491557 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:50:03.491420    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f89ad58b-cab6-4558-aa07-24370249825a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f89ad58b-cab6-4558-aa07-24370249825a-istiod-ca-cert\") pod \"f89ad58b-cab6-4558-aa07-24370249825a\" (UID: \"f89ad58b-cab6-4558-aa07-24370249825a\") "
Nov 03 20:50:03.491557 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:50:03.491461    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f89ad58b-cab6-4558-aa07-24370249825a-kube-api-access-7g8vn\" (UniqueName: \"kubernetes.io/projected/f89ad58b-cab6-4558-aa07-24370249825a-kube-api-access-7g8vn\") pod \"f89ad58b-cab6-4558-aa07-24370249825a\" (UID: \"f89ad58b-cab6-4558-aa07-24370249825a\") "
Nov 03 20:50:03.491557 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:50:03.491515    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f89ad58b-cab6-4558-aa07-24370249825a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f89ad58b-cab6-4558-aa07-24370249825a-istio-podinfo\") pod \"f89ad58b-cab6-4558-aa07-24370249825a\" (UID: \"f89ad58b-cab6-4558-aa07-24370249825a\") "
Nov 03 20:50:03.491978 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:50:03.491565    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f89ad58b-cab6-4558-aa07-24370249825a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f89ad58b-cab6-4558-aa07-24370249825a-workload-certs\") pod \"f89ad58b-cab6-4558-aa07-24370249825a\" (UID: \"f89ad58b-cab6-4558-aa07-24370249825a\") "
Nov 03 20:50:03.492134 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:50:03.492005    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f89ad58b-cab6-4558-aa07-24370249825a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f89ad58b-cab6-4558-aa07-24370249825a-credential-socket\") pod \"f89ad58b-cab6-4558-aa07-24370249825a\" (UID: \"f89ad58b-cab6-4558-aa07-24370249825a\") "
Nov 03 20:50:03.492134 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:50:03.492038    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f89ad58b-cab6-4558-aa07-24370249825a-istio-token\" (UniqueName: \"kubernetes.io/projected/f89ad58b-cab6-4558-aa07-24370249825a-istio-token\") pod \"f89ad58b-cab6-4558-aa07-24370249825a\" (UID: \"f89ad58b-cab6-4558-aa07-24370249825a\") "
Nov 03 20:50:03.492134 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:50:03.491660    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f89ad58b-cab6-4558-aa07-24370249825a-istio-data" pod "f89ad58b-cab6-4558-aa07-24370249825a" (UID: "f89ad58b-cab6-4558-aa07-24370249825a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:50:03.492134 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:50:03.491706    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f89ad58b-cab6-4558-aa07-24370249825a-workload-socket" pod "f89ad58b-cab6-4558-aa07-24370249825a" (UID: "f89ad58b-cab6-4558-aa07-24370249825a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:50:03.492134 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:50:03.491857    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f89ad58b-cab6-4558-aa07-24370249825a-istiod-ca-cert" pod "f89ad58b-cab6-4558-aa07-24370249825a" (UID: "f89ad58b-cab6-4558-aa07-24370249825a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:50:03.492134 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:50:03.491953    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f89ad58b-cab6-4558-aa07-24370249825a-workload-certs" pod "f89ad58b-cab6-4558-aa07-24370249825a" (UID: "f89ad58b-cab6-4558-aa07-24370249825a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:50:03.492511 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:50:03.492241    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f89ad58b-cab6-4558-aa07-24370249825a-credential-socket" pod "f89ad58b-cab6-4558-aa07-24370249825a" (UID: "f89ad58b-cab6-4558-aa07-24370249825a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:50:03.492511 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:50:03.492434    5570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f89ad58b-cab6-4558-aa07-24370249825a-workload-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:50:03.492511 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:50:03.492457    5570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f89ad58b-cab6-4558-aa07-24370249825a-credential-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:50:03.492511 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:50:03.492470    5570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f89ad58b-cab6-4558-aa07-24370249825a-istio-data\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:50:03.492511 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:50:03.492485    5570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f89ad58b-cab6-4558-aa07-24370249825a-workload-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:50:03.492511 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:50:03.492500    5570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f89ad58b-cab6-4558-aa07-24370249825a-istiod-ca-cert\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:50:03.493655 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:50:03.493631    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f89ad58b-cab6-4558-aa07-24370249825a-istio-envoy" pod "f89ad58b-cab6-4558-aa07-24370249825a" (UID: "f89ad58b-cab6-4558-aa07-24370249825a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:50:03.493799 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:50:03.493759    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f89ad58b-cab6-4558-aa07-24370249825a-istio-podinfo" pod "f89ad58b-cab6-4558-aa07-24370249825a" (UID: "f89ad58b-cab6-4558-aa07-24370249825a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:50:03.493903 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:50:03.493884    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f89ad58b-cab6-4558-aa07-24370249825a-kube-api-access-7g8vn" pod "f89ad58b-cab6-4558-aa07-24370249825a" (UID: "f89ad58b-cab6-4558-aa07-24370249825a"). InnerVolumeSpecName "kube-api-access-7g8vn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:50:03.494176 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:50:03.494161    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f89ad58b-cab6-4558-aa07-24370249825a-istio-token" pod "f89ad58b-cab6-4558-aa07-24370249825a" (UID: "f89ad58b-cab6-4558-aa07-24370249825a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:50:03.593882 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:50:03.593841    5570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f89ad58b-cab6-4558-aa07-24370249825a-istio-podinfo\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:50:03.593882 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:50:03.593873    5570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f89ad58b-cab6-4558-aa07-24370249825a-istio-token\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:50:03.593882 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:50:03.593884    5570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f89ad58b-cab6-4558-aa07-24370249825a-istio-envoy\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:50:03.594307 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:50:03.593896    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7g8vn\" (UniqueName: \"kubernetes.io/projected/f89ad58b-cab6-4558-aa07-24370249825a-kube-api-access-7g8vn\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:50:03.643407 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:50:03.643373    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g69f8"]
Nov 03 20:50:04.289419 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:50:04.289326    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-0d89d1-openshift-default-79f4655d69477wq" event={"ID":"f89ad58b-cab6-4558-aa07-24370249825a","Type":"ContainerDied","Data":"3438749eb1a356a1896bd33ddde0f211a8d917523ffe1d0ec9d1c2e0c484d426"}
Nov 03 20:50:04.289419 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:50:04.289397    5570 scope.go:122] "RemoveContainer" containerID="629bd85b89942e12c878358081fee731f8bdfca0a88bf84f051a7b1432a3566b"
Nov 03 20:50:04.290012 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:50:04.289508    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-0d89d1-openshift-default-79f4655d69477wq"
Nov 03 20:50:04.362278 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:50:04.362235    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-0d89d1-openshift-default-79f4655d69477wq"]
Nov 03 20:50:04.366512 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:50:04.366481    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-0d89d1-openshift-default-79f4655d69477wq"]
Nov 03 20:50:06.020801 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:50:06.013060    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f89ad58b-cab6-4558-aa07-24370249825a" path="/var/lib/kubelet/pods/f89ad58b-cab6-4558-aa07-24370249825a/volumes"
Nov 03 20:50:21.346303 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:50:21.346267    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g69f8"]
Nov 03 20:50:27.226579 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:50:27.226540    5570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8bwhz" containerName="extractor"
Nov 03 20:50:32.422543 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:50:32.422510    5570 scope.go:122] "RemoveContainer" containerID="634b2cb11b71e6851773e59803ad8e2dbfa102ab389c5fce3c723daacee82667"
Nov 03 20:50:32.431154 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:50:32.431132    5570 scope.go:122] "RemoveContainer" containerID="ea563f33d3b9a256ba021a4c8c61099219fd7b2b4a1d62f055c81084c6b32e4f"
Nov 03 20:50:32.440890 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:50:32.439989    5570 scope.go:122] "RemoveContainer" containerID="6e4efd932684956a5baa16178e0fed43d20483d3c9b924f8be5307c6c710adb4"
Nov 03 20:50:40.674640 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:50:40.674609    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-d2fmp"]
Nov 03 20:50:40.675069 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:50:40.675054    5570 cpu_manager.go:414] "containerMap: removing container" podUID="f89ad58b-cab6-4558-aa07-24370249825a" containerName="istio-proxy"
Nov 03 20:50:40.675116 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:50:40.675071    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f89ad58b-cab6-4558-aa07-24370249825a" containerName="istio-proxy"
Nov 03 20:50:40.675116 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:50:40.675106    5570 cpu_manager.go:414] "containerMap: removing container" podUID="cf1c2473-a812-46bd-8bae-d736b25e002c" containerName="revoke-keys"
Nov 03 20:50:40.675116 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:50:40.675112    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf1c2473-a812-46bd-8bae-d736b25e002c" containerName="revoke-keys"
Nov 03 20:50:40.675216 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:50:40.675119    5570 cpu_manager.go:414] "containerMap: removing container" podUID="33f2f1cf-adcb-4fb6-b1aa-c5c4e3268ac5" containerName="payload-processing"
Nov 03 20:50:40.675216 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:50:40.675124    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33f2f1cf-adcb-4fb6-b1aa-c5c4e3268ac5" containerName="payload-processing"
Nov 03 20:50:40.675216 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:50:40.675135    5570 cpu_manager.go:414] "containerMap: removing container" podUID="f17a6904-d1c6-4e83-9b02-58ad3a1a619a" containerName="payload-pre-processing"
Nov 03 20:50:40.675216 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:50:40.675140    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f17a6904-d1c6-4e83-9b02-58ad3a1a619a" containerName="payload-pre-processing"
Nov 03 20:50:40.675216 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:50:40.675206    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf1c2473-a812-46bd-8bae-d736b25e002c" containerName="revoke-keys"
Nov 03 20:50:40.675216 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:50:40.675213    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="33f2f1cf-adcb-4fb6-b1aa-c5c4e3268ac5" containerName="payload-processing"
Nov 03 20:50:40.675405 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:50:40.675224    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="f89ad58b-cab6-4558-aa07-24370249825a" containerName="istio-proxy"
Nov 03 20:50:40.675405 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:50:40.675232    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="f17a6904-d1c6-4e83-9b02-58ad3a1a619a" containerName="payload-pre-processing"
Nov 03 20:50:40.680002 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:50:40.679981    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-d2fmp"
Nov 03 20:50:40.694696 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:50:40.694675    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-d2fmp"]
Nov 03 20:50:40.706096 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:50:40.706069    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:50:40.706096 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:50:40.706072    5570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 20:50:40.706276 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:50:40.706074    5570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 20:50:40.709169 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:50:40.709151    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l7s5w\""
Nov 03 20:50:40.750692 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:50:40.750667    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vx5h\" (UniqueName: \"kubernetes.io/projected/56240608-f9e3-4e52-8a4d-5fcfefbebda8-kube-api-access-6vx5h\") pod \"maas-api-revoke-sub-452b6d7b-d2fmp\" (UID: \"56240608-f9e3-4e52-8a4d-5fcfefbebda8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-d2fmp"
Nov 03 20:50:40.750841 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:50:40.750703    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56240608-f9e3-4e52-8a4d-5fcfefbebda8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-d2fmp\" (UID: \"56240608-f9e3-4e52-8a4d-5fcfefbebda8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-d2fmp"
Nov 03 20:50:40.750841 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:50:40.750726    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56240608-f9e3-4e52-8a4d-5fcfefbebda8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-d2fmp\" (UID: \"56240608-f9e3-4e52-8a4d-5fcfefbebda8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-d2fmp"
Nov 03 20:50:40.852126 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:50:40.852101    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vx5h\" (UniqueName: \"kubernetes.io/projected/56240608-f9e3-4e52-8a4d-5fcfefbebda8-kube-api-access-6vx5h\") pod \"maas-api-revoke-sub-452b6d7b-d2fmp\" (UID: \"56240608-f9e3-4e52-8a4d-5fcfefbebda8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-d2fmp"
Nov 03 20:50:40.852245 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:50:40.852136    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56240608-f9e3-4e52-8a4d-5fcfefbebda8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-d2fmp\" (UID: \"56240608-f9e3-4e52-8a4d-5fcfefbebda8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-d2fmp"
Nov 03 20:50:40.852245 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:50:40.852171    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56240608-f9e3-4e52-8a4d-5fcfefbebda8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-d2fmp\" (UID: \"56240608-f9e3-4e52-8a4d-5fcfefbebda8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-d2fmp"
Nov 03 20:50:40.852825 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:50:40.852769    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56240608-f9e3-4e52-8a4d-5fcfefbebda8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-d2fmp\" (UID: \"56240608-f9e3-4e52-8a4d-5fcfefbebda8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-d2fmp"
Nov 03 20:50:40.854445 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:50:40.854426    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56240608-f9e3-4e52-8a4d-5fcfefbebda8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-d2fmp\" (UID: \"56240608-f9e3-4e52-8a4d-5fcfefbebda8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-d2fmp"
Nov 03 20:50:40.872880 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:50:40.872858    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vx5h\" (UniqueName: \"kubernetes.io/projected/56240608-f9e3-4e52-8a4d-5fcfefbebda8-kube-api-access-6vx5h\") pod \"maas-api-revoke-sub-452b6d7b-d2fmp\" (UID: \"56240608-f9e3-4e52-8a4d-5fcfefbebda8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-d2fmp"
Nov 03 20:50:40.993175 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:50:40.993151    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-d2fmp"
Nov 03 20:50:41.026117 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:50:41.026087    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd9d743c-zb28r"]
Nov 03 20:50:41.028187 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:50:41.028162    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd9d743c-zb28r"]
Nov 03 20:50:41.291420 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:50:41.291396    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-d2fmp"]
Nov 03 20:50:41.293011 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:50:41.292982    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56240608_f9e3_4e52_8a4d_5fcfefbebda8.slice/crio-81163d8ee55056d142718c297597797dd7d5b09b10dc713d474a4e5c88e8c343 WatchSource:0}: Error finding container 81163d8ee55056d142718c297597797dd7d5b09b10dc713d474a4e5c88e8c343: Status 404 returned error can't find the container with id 81163d8ee55056d142718c297597797dd7d5b09b10dc713d474a4e5c88e8c343
Nov 03 20:50:41.623462 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:50:41.623427    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-d2fmp" event={"ID":"56240608-f9e3-4e52-8a4d-5fcfefbebda8","Type":"ContainerStarted","Data":"bc093fef1a201af7ac1598c0023022844d1319ea61daee70113aeecb883b0408"}
Nov 03 20:50:41.623602 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:50:41.623481    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-d2fmp" event={"ID":"56240608-f9e3-4e52-8a4d-5fcfefbebda8","Type":"ContainerStarted","Data":"81163d8ee55056d142718c297597797dd7d5b09b10dc713d474a4e5c88e8c343"}
Nov 03 20:50:41.664709 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:50:41.664662    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-d2fmp" podStartSLOduration=1.664649585 podStartE2EDuration="1.664649585s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:50:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:50:41.663293906 +0000 UTC m=+2775.815955805" watchObservedRunningTime="2026-11-03 20:50:41.664649585 +0000 UTC m=+2775.817311472"
Nov 03 20:50:41.964413 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:50:41.964382    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81beac03-bfcc-4f4b-8ed8-3c6672dd4782" path="/var/lib/kubelet/pods/81beac03-bfcc-4f4b-8ed8-3c6672dd4782/volumes"
Nov 03 20:50:42.043114 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:50:42.043078    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g69f8"]
Nov 03 20:50:42.633073 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:50:42.633030    5570 generic.go:356] "Generic (PLEG): container finished" podID="56240608-f9e3-4e52-8a4d-5fcfefbebda8" containerID="bc093fef1a201af7ac1598c0023022844d1319ea61daee70113aeecb883b0408" exitCode=0
Nov 03 20:50:42.633295 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:50:42.633126    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-d2fmp" event={"ID":"56240608-f9e3-4e52-8a4d-5fcfefbebda8","Type":"ContainerDied","Data":"bc093fef1a201af7ac1598c0023022844d1319ea61daee70113aeecb883b0408"}
Nov 03 20:50:44.419524 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:50:44.417215    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-d2fmp"
Nov 03 20:50:44.562539 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:50:44.562505    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56240608-f9e3-4e52-8a4d-5fcfefbebda8-kube-api-access-6vx5h\" (UniqueName: \"kubernetes.io/projected/56240608-f9e3-4e52-8a4d-5fcfefbebda8-kube-api-access-6vx5h\") pod \"56240608-f9e3-4e52-8a4d-5fcfefbebda8\" (UID: \"56240608-f9e3-4e52-8a4d-5fcfefbebda8\") "
Nov 03 20:50:44.563292 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:50:44.563269    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56240608-f9e3-4e52-8a4d-5fcfefbebda8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56240608-f9e3-4e52-8a4d-5fcfefbebda8-maas-api-openshift-ca\") pod \"56240608-f9e3-4e52-8a4d-5fcfefbebda8\" (UID: \"56240608-f9e3-4e52-8a4d-5fcfefbebda8\") "
Nov 03 20:50:44.596139 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:50:44.581253    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/56240608-f9e3-4e52-8a4d-5fcfefbebda8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56240608-f9e3-4e52-8a4d-5fcfefbebda8-maas-api-xks-ca\") pod \"56240608-f9e3-4e52-8a4d-5fcfefbebda8\" (UID: \"56240608-f9e3-4e52-8a4d-5fcfefbebda8\") "
Nov 03 20:50:44.596139 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:50:44.580977    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56240608-f9e3-4e52-8a4d-5fcfefbebda8-maas-api-openshift-ca" pod "56240608-f9e3-4e52-8a4d-5fcfefbebda8" (UID: "56240608-f9e3-4e52-8a4d-5fcfefbebda8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:50:44.596139 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:50:44.588293    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56240608-f9e3-4e52-8a4d-5fcfefbebda8-kube-api-access-6vx5h" pod "56240608-f9e3-4e52-8a4d-5fcfefbebda8" (UID: "56240608-f9e3-4e52-8a4d-5fcfefbebda8"). InnerVolumeSpecName "kube-api-access-6vx5h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:50:44.597911 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:50:44.597882    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56240608-f9e3-4e52-8a4d-5fcfefbebda8-maas-api-xks-ca" pod "56240608-f9e3-4e52-8a4d-5fcfefbebda8" (UID: "56240608-f9e3-4e52-8a4d-5fcfefbebda8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:50:44.722816 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:50:44.712506    5570 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56240608-f9e3-4e52-8a4d-5fcfefbebda8-maas-api-openshift-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:50:44.723518 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:50:44.723493    5570 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56240608-f9e3-4e52-8a4d-5fcfefbebda8-maas-api-xks-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:50:44.723685 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:50:44.723669    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vx5h\" (UniqueName: \"kubernetes.io/projected/56240608-f9e3-4e52-8a4d-5fcfefbebda8-kube-api-access-6vx5h\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:50:44.737799 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:50:44.736337    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-d2fmp" event={"ID":"56240608-f9e3-4e52-8a4d-5fcfefbebda8","Type":"ContainerDied","Data":"81163d8ee55056d142718c297597797dd7d5b09b10dc713d474a4e5c88e8c343"}
Nov 03 20:50:44.737799 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:50:44.736419    5570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="81163d8ee55056d142718c297597797dd7d5b09b10dc713d474a4e5c88e8c343"
Nov 03 20:50:44.737799 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:50:44.736517    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-d2fmp"
Nov 03 20:50:45.432721 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:50:45.432041    5570 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-78txb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:50:45.432721 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:50:45.432175    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-78txb" podUID="b65b5a79-c181-45ec-9cae-ebb5e5bbd865" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:50:45.526206 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:50:45.525874    5570 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-wpxcz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:50:45.526206 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:50:45.525934    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-wpxcz" podUID="36477cba-9827-41ba-997d-d3d646ef4904" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:50:45.623976 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:50:45.623929    5570 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-msmxq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:50:45.624271 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:50:45.624229    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-msmxq" podUID="9ebdd5ef-b321-4dfa-8c3e-1ad7c249a0ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:50:53.846714 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:50:53.846673    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g69f8"]
Nov 03 20:50:57.263559 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:50:57.263518    5570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8bwhz" containerName="extractor"
Nov 03 20:50:57.627404 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:50:57.627323    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-svzb6"]
Nov 03 20:50:57.628134 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:50:57.628108    5570 cpu_manager.go:414] "containerMap: removing container" podUID="56240608-f9e3-4e52-8a4d-5fcfefbebda8" containerName="revoke-keys"
Nov 03 20:50:57.628195 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:50:57.628139    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56240608-f9e3-4e52-8a4d-5fcfefbebda8" containerName="revoke-keys"
Nov 03 20:50:57.628195 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:50:57.628171    5570 cpu_manager.go:414] "containerMap: removing container" podUID="f89ad58b-cab6-4558-aa07-24370249825a" containerName="istio-proxy"
Nov 03 20:50:57.628195 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:50:57.628179    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f89ad58b-cab6-4558-aa07-24370249825a" containerName="istio-proxy"
Nov 03 20:50:57.628350 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:50:57.628334    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="f89ad58b-cab6-4558-aa07-24370249825a" containerName="istio-proxy"
Nov 03 20:50:57.628386 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:50:57.628353    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="56240608-f9e3-4e52-8a4d-5fcfefbebda8" containerName="revoke-keys"
Nov 03 20:50:57.632103 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:50:57.632077    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-svzb6"
Nov 03 20:50:57.650176 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:50:57.650151    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-svzb6"]
Nov 03 20:50:57.656587 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:50:57.656560    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:50:57.661835 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:50:57.661814    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l7s5w\""
Nov 03 20:50:57.661957 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:50:57.661843    5570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 20:50:57.662051 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:50:57.662032    5570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 20:50:57.692688 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:50:57.692661    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxk94\" (UniqueName: \"kubernetes.io/projected/acf7732e-1f8f-4bb5-a04d-5168f3a21bb9-kube-api-access-wxk94\") pod \"maas-api-revoke-sub-3e31a506-svzb6\" (UID: \"acf7732e-1f8f-4bb5-a04d-5168f3a21bb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-svzb6"
Nov 03 20:50:57.692834 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:50:57.692697    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acf7732e-1f8f-4bb5-a04d-5168f3a21bb9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-svzb6\" (UID: \"acf7732e-1f8f-4bb5-a04d-5168f3a21bb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-svzb6"
Nov 03 20:50:57.692834 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:50:57.692808    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acf7732e-1f8f-4bb5-a04d-5168f3a21bb9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-svzb6\" (UID: \"acf7732e-1f8f-4bb5-a04d-5168f3a21bb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-svzb6"
Nov 03 20:50:57.793403 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:50:57.793370    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxk94\" (UniqueName: \"kubernetes.io/projected/acf7732e-1f8f-4bb5-a04d-5168f3a21bb9-kube-api-access-wxk94\") pod \"maas-api-revoke-sub-3e31a506-svzb6\" (UID: \"acf7732e-1f8f-4bb5-a04d-5168f3a21bb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-svzb6"
Nov 03 20:50:57.793555 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:50:57.793409    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acf7732e-1f8f-4bb5-a04d-5168f3a21bb9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-svzb6\" (UID: \"acf7732e-1f8f-4bb5-a04d-5168f3a21bb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-svzb6"
Nov 03 20:50:57.793616 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:50:57.793593    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acf7732e-1f8f-4bb5-a04d-5168f3a21bb9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-svzb6\" (UID: \"acf7732e-1f8f-4bb5-a04d-5168f3a21bb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-svzb6"
Nov 03 20:50:57.794111 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:50:57.794092    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acf7732e-1f8f-4bb5-a04d-5168f3a21bb9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-svzb6\" (UID: \"acf7732e-1f8f-4bb5-a04d-5168f3a21bb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-svzb6"
Nov 03 20:50:57.795848 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:50:57.795823    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acf7732e-1f8f-4bb5-a04d-5168f3a21bb9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-svzb6\" (UID: \"acf7732e-1f8f-4bb5-a04d-5168f3a21bb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-svzb6"
Nov 03 20:50:57.813929 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:50:57.813901    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxk94\" (UniqueName: \"kubernetes.io/projected/acf7732e-1f8f-4bb5-a04d-5168f3a21bb9-kube-api-access-wxk94\") pod \"maas-api-revoke-sub-3e31a506-svzb6\" (UID: \"acf7732e-1f8f-4bb5-a04d-5168f3a21bb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-svzb6"
Nov 03 20:50:57.956791 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:50:57.956739    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-svzb6"
Nov 03 20:50:58.260809 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:50:58.260757    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-svzb6"]
Nov 03 20:50:58.261895 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:50:58.261864    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacf7732e_1f8f_4bb5_a04d_5168f3a21bb9.slice/crio-540f35b37c154f06565e59036cc888bbd0bec61dfd9807c20f99ffed5c87af57 WatchSource:0}: Error finding container 540f35b37c154f06565e59036cc888bbd0bec61dfd9807c20f99ffed5c87af57: Status 404 returned error can't find the container with id 540f35b37c154f06565e59036cc888bbd0bec61dfd9807c20f99ffed5c87af57
Nov 03 20:50:58.848952 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:50:58.848854    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-svzb6" event={"ID":"acf7732e-1f8f-4bb5-a04d-5168f3a21bb9","Type":"ContainerStarted","Data":"177533e9e7e2a055099a8e08174c9255b62ad140e3148b66c808c57f589ae7bb"}
Nov 03 20:50:58.848952 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:50:58.848926    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-svzb6" event={"ID":"acf7732e-1f8f-4bb5-a04d-5168f3a21bb9","Type":"ContainerStarted","Data":"540f35b37c154f06565e59036cc888bbd0bec61dfd9807c20f99ffed5c87af57"}
Nov 03 20:50:58.890743 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:50:58.890682    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-svzb6" podStartSLOduration=1.890663983 podStartE2EDuration="1.890663983s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:50:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:50:58.887586444 +0000 UTC m=+2793.040248330" watchObservedRunningTime="2026-11-03 20:50:58.890663983 +0000 UTC m=+2793.043325871"
Nov 03 20:50:59.477839 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:50:59.473595    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g69f8"]
Nov 03 20:50:59.865802 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:50:59.865317    5570 generic.go:356] "Generic (PLEG): container finished" podID="acf7732e-1f8f-4bb5-a04d-5168f3a21bb9" containerID="177533e9e7e2a055099a8e08174c9255b62ad140e3148b66c808c57f589ae7bb" exitCode=0
Nov 03 20:50:59.865802 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:50:59.865466    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-svzb6" event={"ID":"acf7732e-1f8f-4bb5-a04d-5168f3a21bb9","Type":"ContainerDied","Data":"177533e9e7e2a055099a8e08174c9255b62ad140e3148b66c808c57f589ae7bb"}
Nov 03 20:51:00.422267 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:00.422211    5570 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-78txb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:51:00.422593 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:00.422547    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-78txb" podUID="b65b5a79-c181-45ec-9cae-ebb5e5bbd865" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:51:00.519965 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:00.519897    5570 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-wpxcz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:51:00.520265 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:00.520219    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-wpxcz" podUID="36477cba-9827-41ba-997d-d3d646ef4904" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:51:00.631993 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:00.631899    5570 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-msmxq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:51:00.632701 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:00.632239    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-msmxq" podUID="9ebdd5ef-b321-4dfa-8c3e-1ad7c249a0ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:51:01.024384 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:01.024358    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-svzb6"
Nov 03 20:51:01.172689 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:01.172620    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/acf7732e-1f8f-4bb5-a04d-5168f3a21bb9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acf7732e-1f8f-4bb5-a04d-5168f3a21bb9-maas-api-openshift-ca\") pod \"acf7732e-1f8f-4bb5-a04d-5168f3a21bb9\" (UID: \"acf7732e-1f8f-4bb5-a04d-5168f3a21bb9\") "
Nov 03 20:51:01.172689 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:01.172680    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/acf7732e-1f8f-4bb5-a04d-5168f3a21bb9-kube-api-access-wxk94\" (UniqueName: \"kubernetes.io/projected/acf7732e-1f8f-4bb5-a04d-5168f3a21bb9-kube-api-access-wxk94\") pod \"acf7732e-1f8f-4bb5-a04d-5168f3a21bb9\" (UID: \"acf7732e-1f8f-4bb5-a04d-5168f3a21bb9\") "
Nov 03 20:51:01.172899 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:01.172708    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/acf7732e-1f8f-4bb5-a04d-5168f3a21bb9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acf7732e-1f8f-4bb5-a04d-5168f3a21bb9-maas-api-xks-ca\") pod \"acf7732e-1f8f-4bb5-a04d-5168f3a21bb9\" (UID: \"acf7732e-1f8f-4bb5-a04d-5168f3a21bb9\") "
Nov 03 20:51:01.173007 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:01.172974    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acf7732e-1f8f-4bb5-a04d-5168f3a21bb9-maas-api-openshift-ca" pod "acf7732e-1f8f-4bb5-a04d-5168f3a21bb9" (UID: "acf7732e-1f8f-4bb5-a04d-5168f3a21bb9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:51:01.173147 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:01.173078    5570 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acf7732e-1f8f-4bb5-a04d-5168f3a21bb9-maas-api-openshift-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:51:01.174731 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:01.174713    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/acf7732e-1f8f-4bb5-a04d-5168f3a21bb9-maas-api-xks-ca" pod "acf7732e-1f8f-4bb5-a04d-5168f3a21bb9" (UID: "acf7732e-1f8f-4bb5-a04d-5168f3a21bb9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:51:01.174883 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:01.174867    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acf7732e-1f8f-4bb5-a04d-5168f3a21bb9-kube-api-access-wxk94" pod "acf7732e-1f8f-4bb5-a04d-5168f3a21bb9" (UID: "acf7732e-1f8f-4bb5-a04d-5168f3a21bb9"). InnerVolumeSpecName "kube-api-access-wxk94". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:51:01.273936 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:01.273909    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxk94\" (UniqueName: \"kubernetes.io/projected/acf7732e-1f8f-4bb5-a04d-5168f3a21bb9-kube-api-access-wxk94\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:51:01.273936 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:01.273931    5570 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acf7732e-1f8f-4bb5-a04d-5168f3a21bb9-maas-api-xks-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:51:01.877972 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:01.877949    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-svzb6"
Nov 03 20:51:01.878139 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:01.877946    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-svzb6" event={"ID":"acf7732e-1f8f-4bb5-a04d-5168f3a21bb9","Type":"ContainerDied","Data":"540f35b37c154f06565e59036cc888bbd0bec61dfd9807c20f99ffed5c87af57"}
Nov 03 20:51:01.878139 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:01.878086    5570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="540f35b37c154f06565e59036cc888bbd0bec61dfd9807c20f99ffed5c87af57"
Nov 03 20:51:08.830051 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:08.830017    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g69f8"]
Nov 03 20:51:13.026726 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:13.026692    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ds88f"]
Nov 03 20:51:13.038924 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:13.038891    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ds88f"]
Nov 03 20:51:13.750662 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:13.750622    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g69f8"]
Nov 03 20:51:13.963538 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:13.963509    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7303f2ad-e394-4f4c-8366-2b11b33ddc52" path="/var/lib/kubelet/pods/7303f2ad-e394-4f4c-8366-2b11b33ddc52/volumes"
Nov 03 20:51:19.875327 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:19.875293    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f163065b-wfvlt"]
Nov 03 20:51:19.875931 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:19.875891    5570 cpu_manager.go:414] "containerMap: removing container" podUID="acf7732e-1f8f-4bb5-a04d-5168f3a21bb9" containerName="revoke-keys"
Nov 03 20:51:19.876047 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:19.875932    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="acf7732e-1f8f-4bb5-a04d-5168f3a21bb9" containerName="revoke-keys"
Nov 03 20:51:19.876115 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:19.876044    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="acf7732e-1f8f-4bb5-a04d-5168f3a21bb9" containerName="revoke-keys"
Nov 03 20:51:19.880374 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:19.880351    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f163065b-wfvlt"
Nov 03 20:51:19.896080 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:19.896054    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f163065b-wfvlt"]
Nov 03 20:51:19.907589 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:19.907558    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:51:19.907693 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:19.907590    5570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 20:51:19.907693 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:19.907592    5570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 20:51:19.909474 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:19.909460    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l7s5w\""
Nov 03 20:51:19.929071 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:19.929050    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9944db13-8572-49f1-9e75-5c3a31d3f040-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f163065b-wfvlt\" (UID: \"9944db13-8572-49f1-9e75-5c3a31d3f040\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f163065b-wfvlt"
Nov 03 20:51:19.929211 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:19.929105    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tn4v8\" (UniqueName: \"kubernetes.io/projected/9944db13-8572-49f1-9e75-5c3a31d3f040-kube-api-access-tn4v8\") pod \"maas-api-revoke-sub-f163065b-wfvlt\" (UID: \"9944db13-8572-49f1-9e75-5c3a31d3f040\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f163065b-wfvlt"
Nov 03 20:51:19.929275 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:19.929260    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9944db13-8572-49f1-9e75-5c3a31d3f040-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f163065b-wfvlt\" (UID: \"9944db13-8572-49f1-9e75-5c3a31d3f040\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f163065b-wfvlt"
Nov 03 20:51:20.030513 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:20.030490    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9944db13-8572-49f1-9e75-5c3a31d3f040-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f163065b-wfvlt\" (UID: \"9944db13-8572-49f1-9e75-5c3a31d3f040\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f163065b-wfvlt"
Nov 03 20:51:20.030633 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:20.030544    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9944db13-8572-49f1-9e75-5c3a31d3f040-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f163065b-wfvlt\" (UID: \"9944db13-8572-49f1-9e75-5c3a31d3f040\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f163065b-wfvlt"
Nov 03 20:51:20.030633 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:20.030605    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tn4v8\" (UniqueName: \"kubernetes.io/projected/9944db13-8572-49f1-9e75-5c3a31d3f040-kube-api-access-tn4v8\") pod \"maas-api-revoke-sub-f163065b-wfvlt\" (UID: \"9944db13-8572-49f1-9e75-5c3a31d3f040\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f163065b-wfvlt"
Nov 03 20:51:20.031240 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:20.031217    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9944db13-8572-49f1-9e75-5c3a31d3f040-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f163065b-wfvlt\" (UID: \"9944db13-8572-49f1-9e75-5c3a31d3f040\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f163065b-wfvlt"
Nov 03 20:51:20.032911 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:20.032889    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9944db13-8572-49f1-9e75-5c3a31d3f040-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f163065b-wfvlt\" (UID: \"9944db13-8572-49f1-9e75-5c3a31d3f040\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f163065b-wfvlt"
Nov 03 20:51:20.050394 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:20.050370    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tn4v8\" (UniqueName: \"kubernetes.io/projected/9944db13-8572-49f1-9e75-5c3a31d3f040-kube-api-access-tn4v8\") pod \"maas-api-revoke-sub-f163065b-wfvlt\" (UID: \"9944db13-8572-49f1-9e75-5c3a31d3f040\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f163065b-wfvlt"
Nov 03 20:51:20.193768 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:20.193749    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f163065b-wfvlt"
Nov 03 20:51:20.484180 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:20.484152    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f163065b-wfvlt"]
Nov 03 20:51:20.485704 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:51:20.485673    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9944db13_8572_49f1_9e75_5c3a31d3f040.slice/crio-cfbbf23d4204a61f22426ce2de44fb3d920a98944859c74a8c6543d16820ad56 WatchSource:0}: Error finding container cfbbf23d4204a61f22426ce2de44fb3d920a98944859c74a8c6543d16820ad56: Status 404 returned error can't find the container with id cfbbf23d4204a61f22426ce2de44fb3d920a98944859c74a8c6543d16820ad56
Nov 03 20:51:21.043093 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:21.043019    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f163065b-wfvlt" event={"ID":"9944db13-8572-49f1-9e75-5c3a31d3f040","Type":"ContainerStarted","Data":"13bb865a6218f93bff6f3ea21004248a12a035f9bcfef72acabe68e18cb9ad93"}
Nov 03 20:51:21.043093 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:21.043079    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f163065b-wfvlt" event={"ID":"9944db13-8572-49f1-9e75-5c3a31d3f040","Type":"ContainerStarted","Data":"cfbbf23d4204a61f22426ce2de44fb3d920a98944859c74a8c6543d16820ad56"}
Nov 03 20:51:21.084949 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:21.084905    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f163065b-wfvlt" podStartSLOduration=2.084890649 podStartE2EDuration="2.084890649s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:51:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:51:21.08327284 +0000 UTC m=+2815.235934725" watchObservedRunningTime="2026-11-03 20:51:21.084890649 +0000 UTC m=+2815.237552535"
Nov 03 20:51:22.050097 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:22.050063    5570 generic.go:356] "Generic (PLEG): container finished" podID="9944db13-8572-49f1-9e75-5c3a31d3f040" containerID="13bb865a6218f93bff6f3ea21004248a12a035f9bcfef72acabe68e18cb9ad93" exitCode=0
Nov 03 20:51:22.050514 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:22.050155    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f163065b-wfvlt" event={"ID":"9944db13-8572-49f1-9e75-5c3a31d3f040","Type":"ContainerDied","Data":"13bb865a6218f93bff6f3ea21004248a12a035f9bcfef72acabe68e18cb9ad93"}
Nov 03 20:51:23.196685 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:23.196661    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f163065b-wfvlt"
Nov 03 20:51:23.258763 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:23.258737    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9944db13-8572-49f1-9e75-5c3a31d3f040-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9944db13-8572-49f1-9e75-5c3a31d3f040-maas-api-xks-ca\") pod \"9944db13-8572-49f1-9e75-5c3a31d3f040\" (UID: \"9944db13-8572-49f1-9e75-5c3a31d3f040\") "
Nov 03 20:51:23.258935 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:23.258849    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9944db13-8572-49f1-9e75-5c3a31d3f040-kube-api-access-tn4v8\" (UniqueName: \"kubernetes.io/projected/9944db13-8572-49f1-9e75-5c3a31d3f040-kube-api-access-tn4v8\") pod \"9944db13-8572-49f1-9e75-5c3a31d3f040\" (UID: \"9944db13-8572-49f1-9e75-5c3a31d3f040\") "
Nov 03 20:51:23.258935 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:23.258918    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9944db13-8572-49f1-9e75-5c3a31d3f040-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9944db13-8572-49f1-9e75-5c3a31d3f040-maas-api-openshift-ca\") pod \"9944db13-8572-49f1-9e75-5c3a31d3f040\" (UID: \"9944db13-8572-49f1-9e75-5c3a31d3f040\") "
Nov 03 20:51:23.259298 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:23.259266    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9944db13-8572-49f1-9e75-5c3a31d3f040-maas-api-openshift-ca" pod "9944db13-8572-49f1-9e75-5c3a31d3f040" (UID: "9944db13-8572-49f1-9e75-5c3a31d3f040"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:51:23.260825 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:23.260798    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9944db13-8572-49f1-9e75-5c3a31d3f040-maas-api-xks-ca" pod "9944db13-8572-49f1-9e75-5c3a31d3f040" (UID: "9944db13-8572-49f1-9e75-5c3a31d3f040"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:51:23.260992 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:23.260970    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9944db13-8572-49f1-9e75-5c3a31d3f040-kube-api-access-tn4v8" pod "9944db13-8572-49f1-9e75-5c3a31d3f040" (UID: "9944db13-8572-49f1-9e75-5c3a31d3f040"). InnerVolumeSpecName "kube-api-access-tn4v8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:51:23.360128 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:23.360069    5570 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9944db13-8572-49f1-9e75-5c3a31d3f040-maas-api-openshift-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:51:23.360128 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:23.360102    5570 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9944db13-8572-49f1-9e75-5c3a31d3f040-maas-api-xks-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:51:23.360128 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:23.360118    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tn4v8\" (UniqueName: \"kubernetes.io/projected/9944db13-8572-49f1-9e75-5c3a31d3f040-kube-api-access-tn4v8\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:51:24.062310 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:24.062276    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f163065b-wfvlt" event={"ID":"9944db13-8572-49f1-9e75-5c3a31d3f040","Type":"ContainerDied","Data":"cfbbf23d4204a61f22426ce2de44fb3d920a98944859c74a8c6543d16820ad56"}
Nov 03 20:51:24.062448 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:24.062325    5570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cfbbf23d4204a61f22426ce2de44fb3d920a98944859c74a8c6543d16820ad56"
Nov 03 20:51:24.062448 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:24.062291    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f163065b-wfvlt"
Nov 03 20:51:27.227348 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:51:27.227305    5570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8bwhz" containerName="extractor"
Nov 03 20:51:31.144256 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:31.144217    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g69f8"]
Nov 03 20:51:32.111985 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:32.111947    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-57c467f9cd-5llh8"]
Nov 03 20:51:32.112625 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:32.112601    5570 cpu_manager.go:414] "containerMap: removing container" podUID="9944db13-8572-49f1-9e75-5c3a31d3f040" containerName="revoke-keys"
Nov 03 20:51:32.112625 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:32.112624    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9944db13-8572-49f1-9e75-5c3a31d3f040" containerName="revoke-keys"
Nov 03 20:51:32.112863 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:32.112751    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="9944db13-8572-49f1-9e75-5c3a31d3f040" containerName="revoke-keys"
Nov 03 20:51:32.116179 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:32.116160    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-57c467f9cd-5llh8"
Nov 03 20:51:32.132723 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:32.132697    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-57c467f9cd-5llh8"]
Nov 03 20:51:32.143805 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:32.143756    5570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 20:51:32.143996 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:32.143982    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-49bxb\""
Nov 03 20:51:32.147239 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:32.147215    5570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 20:51:32.147617 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:32.147246    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 03 20:51:32.147617 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:32.147294    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Nov 03 20:51:32.243562 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:32.243525    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/849d6b39-d800-4658-be04-10929ab6e50b-maas-api-tls\") pod \"maas-api-57c467f9cd-5llh8\" (UID: \"849d6b39-d800-4658-be04-10929ab6e50b\") " pod="odh-ai-gateway-infra/maas-api-57c467f9cd-5llh8"
Nov 03 20:51:32.243562 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:32.243561    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/849d6b39-d800-4658-be04-10929ab6e50b-metrics-certs\") pod \"maas-api-57c467f9cd-5llh8\" (UID: \"849d6b39-d800-4658-be04-10929ab6e50b\") " pod="odh-ai-gateway-infra/maas-api-57c467f9cd-5llh8"
Nov 03 20:51:32.243818 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:32.243608    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfqnj\" (UniqueName: \"kubernetes.io/projected/849d6b39-d800-4658-be04-10929ab6e50b-kube-api-access-zfqnj\") pod \"maas-api-57c467f9cd-5llh8\" (UID: \"849d6b39-d800-4658-be04-10929ab6e50b\") " pod="odh-ai-gateway-infra/maas-api-57c467f9cd-5llh8"
Nov 03 20:51:32.243818 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:32.243713    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/849d6b39-d800-4658-be04-10929ab6e50b-openshift-service-ca\") pod \"maas-api-57c467f9cd-5llh8\" (UID: \"849d6b39-d800-4658-be04-10929ab6e50b\") " pod="odh-ai-gateway-infra/maas-api-57c467f9cd-5llh8"
Nov 03 20:51:32.345319 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:32.345289    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/849d6b39-d800-4658-be04-10929ab6e50b-maas-api-tls\") pod \"maas-api-57c467f9cd-5llh8\" (UID: \"849d6b39-d800-4658-be04-10929ab6e50b\") " pod="odh-ai-gateway-infra/maas-api-57c467f9cd-5llh8"
Nov 03 20:51:32.345319 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:32.345320    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/849d6b39-d800-4658-be04-10929ab6e50b-metrics-certs\") pod \"maas-api-57c467f9cd-5llh8\" (UID: \"849d6b39-d800-4658-be04-10929ab6e50b\") " pod="odh-ai-gateway-infra/maas-api-57c467f9cd-5llh8"
Nov 03 20:51:32.345538 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:32.345362    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfqnj\" (UniqueName: \"kubernetes.io/projected/849d6b39-d800-4658-be04-10929ab6e50b-kube-api-access-zfqnj\") pod \"maas-api-57c467f9cd-5llh8\" (UID: \"849d6b39-d800-4658-be04-10929ab6e50b\") " pod="odh-ai-gateway-infra/maas-api-57c467f9cd-5llh8"
Nov 03 20:51:32.345538 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:32.345380    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/849d6b39-d800-4658-be04-10929ab6e50b-openshift-service-ca\") pod \"maas-api-57c467f9cd-5llh8\" (UID: \"849d6b39-d800-4658-be04-10929ab6e50b\") " pod="odh-ai-gateway-infra/maas-api-57c467f9cd-5llh8"
Nov 03 20:51:32.346035 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:32.346012    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/849d6b39-d800-4658-be04-10929ab6e50b-openshift-service-ca\") pod \"maas-api-57c467f9cd-5llh8\" (UID: \"849d6b39-d800-4658-be04-10929ab6e50b\") " pod="odh-ai-gateway-infra/maas-api-57c467f9cd-5llh8"
Nov 03 20:51:32.347696 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:32.347669    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/849d6b39-d800-4658-be04-10929ab6e50b-maas-api-tls\") pod \"maas-api-57c467f9cd-5llh8\" (UID: \"849d6b39-d800-4658-be04-10929ab6e50b\") " pod="odh-ai-gateway-infra/maas-api-57c467f9cd-5llh8"
Nov 03 20:51:32.347800 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:32.347679    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/849d6b39-d800-4658-be04-10929ab6e50b-metrics-certs\") pod \"maas-api-57c467f9cd-5llh8\" (UID: \"849d6b39-d800-4658-be04-10929ab6e50b\") " pod="odh-ai-gateway-infra/maas-api-57c467f9cd-5llh8"
Nov 03 20:51:32.372106 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:32.372031    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfqnj\" (UniqueName: \"kubernetes.io/projected/849d6b39-d800-4658-be04-10929ab6e50b-kube-api-access-zfqnj\") pod \"maas-api-57c467f9cd-5llh8\" (UID: \"849d6b39-d800-4658-be04-10929ab6e50b\") " pod="odh-ai-gateway-infra/maas-api-57c467f9cd-5llh8"
Nov 03 20:51:32.432338 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:32.432313    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-57c467f9cd-5llh8"
Nov 03 20:51:32.728109 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:32.728081    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-57c467f9cd-5llh8"]
Nov 03 20:51:32.729558 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:51:32.729523    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod849d6b39_d800_4658_be04_10929ab6e50b.slice/crio-d0aaf5ef5fa52ca2c35ab428ded11a1f0be59f1c4767b4fdd7eab6b9b2159e6a WatchSource:0}: Error finding container d0aaf5ef5fa52ca2c35ab428ded11a1f0be59f1c4767b4fdd7eab6b9b2159e6a: Status 404 returned error can't find the container with id d0aaf5ef5fa52ca2c35ab428ded11a1f0be59f1c4767b4fdd7eab6b9b2159e6a
Nov 03 20:51:33.121734 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:33.121706    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-57c467f9cd-5llh8" event={"ID":"849d6b39-d800-4658-be04-10929ab6e50b","Type":"ContainerStarted","Data":"b3c87049f78615fd86f0fb9b27c43e38034fb6d23cbf0883df8bd34f1701b3cf"}
Nov 03 20:51:33.121850 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:33.121764    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-57c467f9cd-5llh8" event={"ID":"849d6b39-d800-4658-be04-10929ab6e50b","Type":"ContainerStarted","Data":"d0aaf5ef5fa52ca2c35ab428ded11a1f0be59f1c4767b4fdd7eab6b9b2159e6a"}
Nov 03 20:51:33.121914 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:33.121900    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-57c467f9cd-5llh8"
Nov 03 20:51:33.168870 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:33.168824    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-57c467f9cd-5llh8" podStartSLOduration=0.834644893 podStartE2EDuration="1.168805479s" totalImagesPullingTime="334.160586ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:51:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:51:33.168113969 +0000 UTC m=+2827.320775856" watchObservedRunningTime="2026-11-03 20:51:33.168805479 +0000 UTC m=+2827.321467383"
Nov 03 20:51:39.133387 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:39.133357    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-57c467f9cd-5llh8"
Nov 03 20:51:39.211058 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:39.211026    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-e2966c90-openshift-default-7cf67d4585-6nznw"]
Nov 03 20:51:39.215098 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:39.215072    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e2966c90-openshift-default-7cf67d4585-6nznw"
Nov 03 20:51:39.235969 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:39.235250    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-e2966c90-openshift-default-dockercfg-mgmcm\""
Nov 03 20:51:39.238152 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:39.238128    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-e2966c90-openshift-default-7cf67d4585-6nznw"]
Nov 03 20:51:39.310108 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:39.310062    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b86bb594-b896-443b-aeaf-a734cf38cea0-istio-podinfo\") pod \"e2e-ait-e2966c90-openshift-default-7cf67d4585-6nznw\" (UID: \"b86bb594-b896-443b-aeaf-a734cf38cea0\") " pod="openshift-ingress/e2e-ait-e2966c90-openshift-default-7cf67d4585-6nznw"
Nov 03 20:51:39.310108 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:39.310111    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b86bb594-b896-443b-aeaf-a734cf38cea0-istio-token\") pod \"e2e-ait-e2966c90-openshift-default-7cf67d4585-6nznw\" (UID: \"b86bb594-b896-443b-aeaf-a734cf38cea0\") " pod="openshift-ingress/e2e-ait-e2966c90-openshift-default-7cf67d4585-6nznw"
Nov 03 20:51:39.310387 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:39.310144    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b86bb594-b896-443b-aeaf-a734cf38cea0-workload-certs\") pod \"e2e-ait-e2966c90-openshift-default-7cf67d4585-6nznw\" (UID: \"b86bb594-b896-443b-aeaf-a734cf38cea0\") " pod="openshift-ingress/e2e-ait-e2966c90-openshift-default-7cf67d4585-6nznw"
Nov 03 20:51:39.310387 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:39.310179    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b86bb594-b896-443b-aeaf-a734cf38cea0-istio-data\") pod \"e2e-ait-e2966c90-openshift-default-7cf67d4585-6nznw\" (UID: \"b86bb594-b896-443b-aeaf-a734cf38cea0\") " pod="openshift-ingress/e2e-ait-e2966c90-openshift-default-7cf67d4585-6nznw"
Nov 03 20:51:39.310387 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:39.310223    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b86bb594-b896-443b-aeaf-a734cf38cea0-credential-socket\") pod \"e2e-ait-e2966c90-openshift-default-7cf67d4585-6nznw\" (UID: \"b86bb594-b896-443b-aeaf-a734cf38cea0\") " pod="openshift-ingress/e2e-ait-e2966c90-openshift-default-7cf67d4585-6nznw"
Nov 03 20:51:39.310387 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:39.310280    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b86bb594-b896-443b-aeaf-a734cf38cea0-workload-socket\") pod \"e2e-ait-e2966c90-openshift-default-7cf67d4585-6nznw\" (UID: \"b86bb594-b896-443b-aeaf-a734cf38cea0\") " pod="openshift-ingress/e2e-ait-e2966c90-openshift-default-7cf67d4585-6nznw"
Nov 03 20:51:39.310387 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:39.310304    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b86bb594-b896-443b-aeaf-a734cf38cea0-istiod-ca-cert\") pod \"e2e-ait-e2966c90-openshift-default-7cf67d4585-6nznw\" (UID: \"b86bb594-b896-443b-aeaf-a734cf38cea0\") " pod="openshift-ingress/e2e-ait-e2966c90-openshift-default-7cf67d4585-6nznw"
Nov 03 20:51:39.310387 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:39.310349    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b86bb594-b896-443b-aeaf-a734cf38cea0-istio-envoy\") pod \"e2e-ait-e2966c90-openshift-default-7cf67d4585-6nznw\" (UID: \"b86bb594-b896-443b-aeaf-a734cf38cea0\") " pod="openshift-ingress/e2e-ait-e2966c90-openshift-default-7cf67d4585-6nznw"
Nov 03 20:51:39.310709 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:39.310418    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gsns\" (UniqueName: \"kubernetes.io/projected/b86bb594-b896-443b-aeaf-a734cf38cea0-kube-api-access-7gsns\") pod \"e2e-ait-e2966c90-openshift-default-7cf67d4585-6nznw\" (UID: \"b86bb594-b896-443b-aeaf-a734cf38cea0\") " pod="openshift-ingress/e2e-ait-e2966c90-openshift-default-7cf67d4585-6nznw"
Nov 03 20:51:39.413172 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:39.411390    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b86bb594-b896-443b-aeaf-a734cf38cea0-workload-socket\") pod \"e2e-ait-e2966c90-openshift-default-7cf67d4585-6nznw\" (UID: \"b86bb594-b896-443b-aeaf-a734cf38cea0\") " pod="openshift-ingress/e2e-ait-e2966c90-openshift-default-7cf67d4585-6nznw"
Nov 03 20:51:39.413172 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:39.411451    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b86bb594-b896-443b-aeaf-a734cf38cea0-istiod-ca-cert\") pod \"e2e-ait-e2966c90-openshift-default-7cf67d4585-6nznw\" (UID: \"b86bb594-b896-443b-aeaf-a734cf38cea0\") " pod="openshift-ingress/e2e-ait-e2966c90-openshift-default-7cf67d4585-6nznw"
Nov 03 20:51:39.413172 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:39.411515    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b86bb594-b896-443b-aeaf-a734cf38cea0-istio-envoy\") pod \"e2e-ait-e2966c90-openshift-default-7cf67d4585-6nznw\" (UID: \"b86bb594-b896-443b-aeaf-a734cf38cea0\") " pod="openshift-ingress/e2e-ait-e2966c90-openshift-default-7cf67d4585-6nznw"
Nov 03 20:51:39.413172 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:39.411606    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gsns\" (UniqueName: \"kubernetes.io/projected/b86bb594-b896-443b-aeaf-a734cf38cea0-kube-api-access-7gsns\") pod \"e2e-ait-e2966c90-openshift-default-7cf67d4585-6nznw\" (UID: \"b86bb594-b896-443b-aeaf-a734cf38cea0\") " pod="openshift-ingress/e2e-ait-e2966c90-openshift-default-7cf67d4585-6nznw"
Nov 03 20:51:39.413172 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:39.411657    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b86bb594-b896-443b-aeaf-a734cf38cea0-istio-podinfo\") pod \"e2e-ait-e2966c90-openshift-default-7cf67d4585-6nznw\" (UID: \"b86bb594-b896-443b-aeaf-a734cf38cea0\") " pod="openshift-ingress/e2e-ait-e2966c90-openshift-default-7cf67d4585-6nznw"
Nov 03 20:51:39.413172 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:39.411681    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b86bb594-b896-443b-aeaf-a734cf38cea0-istio-token\") pod \"e2e-ait-e2966c90-openshift-default-7cf67d4585-6nznw\" (UID: \"b86bb594-b896-443b-aeaf-a734cf38cea0\") " pod="openshift-ingress/e2e-ait-e2966c90-openshift-default-7cf67d4585-6nznw"
Nov 03 20:51:39.413172 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:39.411714    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b86bb594-b896-443b-aeaf-a734cf38cea0-workload-certs\") pod \"e2e-ait-e2966c90-openshift-default-7cf67d4585-6nznw\" (UID: \"b86bb594-b896-443b-aeaf-a734cf38cea0\") " pod="openshift-ingress/e2e-ait-e2966c90-openshift-default-7cf67d4585-6nznw"
Nov 03 20:51:39.413172 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:39.411751    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b86bb594-b896-443b-aeaf-a734cf38cea0-istio-data\") pod \"e2e-ait-e2966c90-openshift-default-7cf67d4585-6nznw\" (UID: \"b86bb594-b896-443b-aeaf-a734cf38cea0\") " pod="openshift-ingress/e2e-ait-e2966c90-openshift-default-7cf67d4585-6nznw"
Nov 03 20:51:39.413172 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:39.411814    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b86bb594-b896-443b-aeaf-a734cf38cea0-credential-socket\") pod \"e2e-ait-e2966c90-openshift-default-7cf67d4585-6nznw\" (UID: \"b86bb594-b896-443b-aeaf-a734cf38cea0\") " pod="openshift-ingress/e2e-ait-e2966c90-openshift-default-7cf67d4585-6nznw"
Nov 03 20:51:39.413172 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:39.412262    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b86bb594-b896-443b-aeaf-a734cf38cea0-credential-socket\") pod \"e2e-ait-e2966c90-openshift-default-7cf67d4585-6nznw\" (UID: \"b86bb594-b896-443b-aeaf-a734cf38cea0\") " pod="openshift-ingress/e2e-ait-e2966c90-openshift-default-7cf67d4585-6nznw"
Nov 03 20:51:39.413172 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:39.412514    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b86bb594-b896-443b-aeaf-a734cf38cea0-workload-socket\") pod \"e2e-ait-e2966c90-openshift-default-7cf67d4585-6nznw\" (UID: \"b86bb594-b896-443b-aeaf-a734cf38cea0\") " pod="openshift-ingress/e2e-ait-e2966c90-openshift-default-7cf67d4585-6nznw"
Nov 03 20:51:39.427117 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:39.423050    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b86bb594-b896-443b-aeaf-a734cf38cea0-istiod-ca-cert\") pod \"e2e-ait-e2966c90-openshift-default-7cf67d4585-6nznw\" (UID: \"b86bb594-b896-443b-aeaf-a734cf38cea0\") " pod="openshift-ingress/e2e-ait-e2966c90-openshift-default-7cf67d4585-6nznw"
Nov 03 20:51:39.427117 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:39.425600    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b86bb594-b896-443b-aeaf-a734cf38cea0-workload-certs\") pod \"e2e-ait-e2966c90-openshift-default-7cf67d4585-6nznw\" (UID: \"b86bb594-b896-443b-aeaf-a734cf38cea0\") " pod="openshift-ingress/e2e-ait-e2966c90-openshift-default-7cf67d4585-6nznw"
Nov 03 20:51:39.427117 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:39.425871    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b86bb594-b896-443b-aeaf-a734cf38cea0-istio-data\") pod \"e2e-ait-e2966c90-openshift-default-7cf67d4585-6nznw\" (UID: \"b86bb594-b896-443b-aeaf-a734cf38cea0\") " pod="openshift-ingress/e2e-ait-e2966c90-openshift-default-7cf67d4585-6nznw"
Nov 03 20:51:39.427325 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:39.427288    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b86bb594-b896-443b-aeaf-a734cf38cea0-istio-podinfo\") pod \"e2e-ait-e2966c90-openshift-default-7cf67d4585-6nznw\" (UID: \"b86bb594-b896-443b-aeaf-a734cf38cea0\") " pod="openshift-ingress/e2e-ait-e2966c90-openshift-default-7cf67d4585-6nznw"
Nov 03 20:51:39.444930 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:39.439079    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b86bb594-b896-443b-aeaf-a734cf38cea0-istio-envoy\") pod \"e2e-ait-e2966c90-openshift-default-7cf67d4585-6nznw\" (UID: \"b86bb594-b896-443b-aeaf-a734cf38cea0\") " pod="openshift-ingress/e2e-ait-e2966c90-openshift-default-7cf67d4585-6nznw"
Nov 03 20:51:39.454474 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:39.452845    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b86bb594-b896-443b-aeaf-a734cf38cea0-istio-token\") pod \"e2e-ait-e2966c90-openshift-default-7cf67d4585-6nznw\" (UID: \"b86bb594-b896-443b-aeaf-a734cf38cea0\") " pod="openshift-ingress/e2e-ait-e2966c90-openshift-default-7cf67d4585-6nznw"
Nov 03 20:51:39.464827 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:39.462428    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gsns\" (UniqueName: \"kubernetes.io/projected/b86bb594-b896-443b-aeaf-a734cf38cea0-kube-api-access-7gsns\") pod \"e2e-ait-e2966c90-openshift-default-7cf67d4585-6nznw\" (UID: \"b86bb594-b896-443b-aeaf-a734cf38cea0\") " pod="openshift-ingress/e2e-ait-e2966c90-openshift-default-7cf67d4585-6nznw"
Nov 03 20:51:39.530227 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:39.530196    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e2966c90-openshift-default-7cf67d4585-6nznw"
Nov 03 20:51:39.843722 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:39.843700    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-e2966c90-openshift-default-7cf67d4585-6nznw"]
Nov 03 20:51:39.845072 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:51:39.845034    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb86bb594_b896_443b_aeaf_a734cf38cea0.slice/crio-960f315c3dbbc9a79ed1218bdf4e74235178aa2bcac725e51ff558addd5681e5 WatchSource:0}: Error finding container 960f315c3dbbc9a79ed1218bdf4e74235178aa2bcac725e51ff558addd5681e5: Status 404 returned error can't find the container with id 960f315c3dbbc9a79ed1218bdf4e74235178aa2bcac725e51ff558addd5681e5
Nov 03 20:51:39.847184 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:39.847155    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:51:39.847279 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:39.847217    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:51:39.847279 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:39.847246    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:51:39.928999 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:39.928974    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-796c56b65d-t2j9z"]
Nov 03 20:51:39.932564 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:39.932544    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-796c56b65d-t2j9z"
Nov 03 20:51:39.952873 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:39.952844    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-796c56b65d-t2j9z"]
Nov 03 20:51:40.017087 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:40.017052    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d191c3ba-8755-49a6-8ba4-d731f4cc0de2-maas-api-tls\") pod \"maas-api-796c56b65d-t2j9z\" (UID: \"d191c3ba-8755-49a6-8ba4-d731f4cc0de2\") " pod="odh-ai-gateway-infra/maas-api-796c56b65d-t2j9z"
Nov 03 20:51:40.017291 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:40.017109    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z27nt\" (UniqueName: \"kubernetes.io/projected/d191c3ba-8755-49a6-8ba4-d731f4cc0de2-kube-api-access-z27nt\") pod \"maas-api-796c56b65d-t2j9z\" (UID: \"d191c3ba-8755-49a6-8ba4-d731f4cc0de2\") " pod="odh-ai-gateway-infra/maas-api-796c56b65d-t2j9z"
Nov 03 20:51:40.017291 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:40.017249    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d191c3ba-8755-49a6-8ba4-d731f4cc0de2-metrics-certs\") pod \"maas-api-796c56b65d-t2j9z\" (UID: \"d191c3ba-8755-49a6-8ba4-d731f4cc0de2\") " pod="odh-ai-gateway-infra/maas-api-796c56b65d-t2j9z"
Nov 03 20:51:40.017410 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:40.017351    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d191c3ba-8755-49a6-8ba4-d731f4cc0de2-openshift-service-ca\") pod \"maas-api-796c56b65d-t2j9z\" (UID: \"d191c3ba-8755-49a6-8ba4-d731f4cc0de2\") " pod="odh-ai-gateway-infra/maas-api-796c56b65d-t2j9z"
Nov 03 20:51:40.118167 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:40.118095    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d191c3ba-8755-49a6-8ba4-d731f4cc0de2-metrics-certs\") pod \"maas-api-796c56b65d-t2j9z\" (UID: \"d191c3ba-8755-49a6-8ba4-d731f4cc0de2\") " pod="odh-ai-gateway-infra/maas-api-796c56b65d-t2j9z"
Nov 03 20:51:40.118322 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:40.118169    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d191c3ba-8755-49a6-8ba4-d731f4cc0de2-openshift-service-ca\") pod \"maas-api-796c56b65d-t2j9z\" (UID: \"d191c3ba-8755-49a6-8ba4-d731f4cc0de2\") " pod="odh-ai-gateway-infra/maas-api-796c56b65d-t2j9z"
Nov 03 20:51:40.118322 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:40.118199    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d191c3ba-8755-49a6-8ba4-d731f4cc0de2-maas-api-tls\") pod \"maas-api-796c56b65d-t2j9z\" (UID: \"d191c3ba-8755-49a6-8ba4-d731f4cc0de2\") " pod="odh-ai-gateway-infra/maas-api-796c56b65d-t2j9z"
Nov 03 20:51:40.118322 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:40.118236    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z27nt\" (UniqueName: \"kubernetes.io/projected/d191c3ba-8755-49a6-8ba4-d731f4cc0de2-kube-api-access-z27nt\") pod \"maas-api-796c56b65d-t2j9z\" (UID: \"d191c3ba-8755-49a6-8ba4-d731f4cc0de2\") " pod="odh-ai-gateway-infra/maas-api-796c56b65d-t2j9z"
Nov 03 20:51:40.118970 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:40.118943    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d191c3ba-8755-49a6-8ba4-d731f4cc0de2-openshift-service-ca\") pod \"maas-api-796c56b65d-t2j9z\" (UID: \"d191c3ba-8755-49a6-8ba4-d731f4cc0de2\") " pod="odh-ai-gateway-infra/maas-api-796c56b65d-t2j9z"
Nov 03 20:51:40.120690 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:40.120666    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d191c3ba-8755-49a6-8ba4-d731f4cc0de2-metrics-certs\") pod \"maas-api-796c56b65d-t2j9z\" (UID: \"d191c3ba-8755-49a6-8ba4-d731f4cc0de2\") " pod="odh-ai-gateway-infra/maas-api-796c56b65d-t2j9z"
Nov 03 20:51:40.120862 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:40.120676    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d191c3ba-8755-49a6-8ba4-d731f4cc0de2-maas-api-tls\") pod \"maas-api-796c56b65d-t2j9z\" (UID: \"d191c3ba-8755-49a6-8ba4-d731f4cc0de2\") " pod="odh-ai-gateway-infra/maas-api-796c56b65d-t2j9z"
Nov 03 20:51:40.138678 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:40.138649    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z27nt\" (UniqueName: \"kubernetes.io/projected/d191c3ba-8755-49a6-8ba4-d731f4cc0de2-kube-api-access-z27nt\") pod \"maas-api-796c56b65d-t2j9z\" (UID: \"d191c3ba-8755-49a6-8ba4-d731f4cc0de2\") " pod="odh-ai-gateway-infra/maas-api-796c56b65d-t2j9z"
Nov 03 20:51:40.165135 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:40.165094    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e2966c90-openshift-default-7cf67d4585-6nznw" event={"ID":"b86bb594-b896-443b-aeaf-a734cf38cea0","Type":"ContainerStarted","Data":"34b5e3bce94b799df83320e4f8616f51e45104a9ef12f3a9437873c128a57e61"}
Nov 03 20:51:40.165264 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:40.165153    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e2966c90-openshift-default-7cf67d4585-6nznw" event={"ID":"b86bb594-b896-443b-aeaf-a734cf38cea0","Type":"ContainerStarted","Data":"960f315c3dbbc9a79ed1218bdf4e74235178aa2bcac725e51ff558addd5681e5"}
Nov 03 20:51:40.211196 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:40.211153    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-e2966c90-openshift-default-7cf67d4585-6nznw" podStartSLOduration=1.211139154 podStartE2EDuration="1.211139154s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:51:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:51:40.209737779 +0000 UTC m=+2834.362399665" watchObservedRunningTime="2026-11-03 20:51:40.211139154 +0000 UTC m=+2834.363801039"
Nov 03 20:51:40.246314 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:40.246285    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-796c56b65d-t2j9z"
Nov 03 20:51:40.531070 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:40.531041    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-e2966c90-openshift-default-7cf67d4585-6nznw"
Nov 03 20:51:40.536228 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:40.536205    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-e2966c90-openshift-default-7cf67d4585-6nznw"
Nov 03 20:51:40.536898 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:40.536878    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-796c56b65d-t2j9z"]
Nov 03 20:51:40.537735 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:51:40.537711    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd191c3ba_8755_49a6_8ba4_d731f4cc0de2.slice/crio-f27b098d856c110bbcdba3786bce0faba9fc3e949e89eb6bd6a76ea1a2ff4399 WatchSource:0}: Error finding container f27b098d856c110bbcdba3786bce0faba9fc3e949e89eb6bd6a76ea1a2ff4399: Status 404 returned error can't find the container with id f27b098d856c110bbcdba3786bce0faba9fc3e949e89eb6bd6a76ea1a2ff4399
Nov 03 20:51:41.171391 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:41.171298    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-796c56b65d-t2j9z" event={"ID":"d191c3ba-8755-49a6-8ba4-d731f4cc0de2","Type":"ContainerStarted","Data":"0e994ac69346b39c5bb6269c2c3b17b7307d3c8439be1485e3b9b070fa0e96dd"}
Nov 03 20:51:41.171857 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:41.171401    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-796c56b65d-t2j9z"
Nov 03 20:51:41.171857 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:41.171423    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-796c56b65d-t2j9z" event={"ID":"d191c3ba-8755-49a6-8ba4-d731f4cc0de2","Type":"ContainerStarted","Data":"f27b098d856c110bbcdba3786bce0faba9fc3e949e89eb6bd6a76ea1a2ff4399"}
Nov 03 20:51:41.171857 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:41.171615    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-e2966c90-openshift-default-7cf67d4585-6nznw"
Nov 03 20:51:41.172700 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:41.172683    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-e2966c90-openshift-default-7cf67d4585-6nznw"
Nov 03 20:51:41.212041 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:41.211948    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-796c56b65d-t2j9z" podStartSLOduration=1.8708625300000001 podStartE2EDuration="2.211932182s" totalImagesPullingTime="341.069652ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:51:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:51:41.21159252 +0000 UTC m=+2835.364254406" watchObservedRunningTime="2026-11-03 20:51:41.211932182 +0000 UTC m=+2835.364594070"
Nov 03 20:51:47.181300 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:47.181273    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-796c56b65d-t2j9z"
Nov 03 20:51:47.256765 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:47.256723    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-57c467f9cd-5llh8"]
Nov 03 20:51:47.257056 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:47.257025    5570 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-57c467f9cd-5llh8" podUID="849d6b39-d800-4658-be04-10929ab6e50b" containerName="maas-api" containerID="cri-o://b3c87049f78615fd86f0fb9b27c43e38034fb6d23cbf0883df8bd34f1701b3cf" gracePeriod=30
Nov 03 20:51:47.282040 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:47.281997    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g69f8"]
Nov 03 20:51:47.622456 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:47.622432    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-57c467f9cd-5llh8"
Nov 03 20:51:47.725851 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:47.725814    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/849d6b39-d800-4658-be04-10929ab6e50b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/849d6b39-d800-4658-be04-10929ab6e50b-openshift-service-ca\") pod \"849d6b39-d800-4658-be04-10929ab6e50b\" (UID: \"849d6b39-d800-4658-be04-10929ab6e50b\") "
Nov 03 20:51:47.726047 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:47.725888    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/849d6b39-d800-4658-be04-10929ab6e50b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/849d6b39-d800-4658-be04-10929ab6e50b-metrics-certs\") pod \"849d6b39-d800-4658-be04-10929ab6e50b\" (UID: \"849d6b39-d800-4658-be04-10929ab6e50b\") "
Nov 03 20:51:47.726047 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:47.725974    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/849d6b39-d800-4658-be04-10929ab6e50b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/849d6b39-d800-4658-be04-10929ab6e50b-maas-api-tls\") pod \"849d6b39-d800-4658-be04-10929ab6e50b\" (UID: \"849d6b39-d800-4658-be04-10929ab6e50b\") "
Nov 03 20:51:47.726047 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:47.726000    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/849d6b39-d800-4658-be04-10929ab6e50b-kube-api-access-zfqnj\" (UniqueName: \"kubernetes.io/projected/849d6b39-d800-4658-be04-10929ab6e50b-kube-api-access-zfqnj\") pod \"849d6b39-d800-4658-be04-10929ab6e50b\" (UID: \"849d6b39-d800-4658-be04-10929ab6e50b\") "
Nov 03 20:51:47.726285 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:47.726260    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/849d6b39-d800-4658-be04-10929ab6e50b-openshift-service-ca" pod "849d6b39-d800-4658-be04-10929ab6e50b" (UID: "849d6b39-d800-4658-be04-10929ab6e50b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:51:47.726427 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:47.726410    5570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/849d6b39-d800-4658-be04-10929ab6e50b-openshift-service-ca\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:51:47.728168 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:47.728138    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/849d6b39-d800-4658-be04-10929ab6e50b-metrics-certs" pod "849d6b39-d800-4658-be04-10929ab6e50b" (UID: "849d6b39-d800-4658-be04-10929ab6e50b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:51:47.728277 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:47.728214    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/849d6b39-d800-4658-be04-10929ab6e50b-maas-api-tls" pod "849d6b39-d800-4658-be04-10929ab6e50b" (UID: "849d6b39-d800-4658-be04-10929ab6e50b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:51:47.728473 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:47.728449    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/849d6b39-d800-4658-be04-10929ab6e50b-kube-api-access-zfqnj" pod "849d6b39-d800-4658-be04-10929ab6e50b" (UID: "849d6b39-d800-4658-be04-10929ab6e50b"). InnerVolumeSpecName "kube-api-access-zfqnj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:51:47.827592 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:47.827553    5570 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/849d6b39-d800-4658-be04-10929ab6e50b-metrics-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:51:47.827592 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:47.827591    5570 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/849d6b39-d800-4658-be04-10929ab6e50b-maas-api-tls\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:51:47.827844 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:47.827607    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfqnj\" (UniqueName: \"kubernetes.io/projected/849d6b39-d800-4658-be04-10929ab6e50b-kube-api-access-zfqnj\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:51:48.216326 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:48.216295    5570 generic.go:356] "Generic (PLEG): container finished" podID="849d6b39-d800-4658-be04-10929ab6e50b" containerID="b3c87049f78615fd86f0fb9b27c43e38034fb6d23cbf0883df8bd34f1701b3cf" exitCode=0
Nov 03 20:51:48.216718 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:48.216347    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-57c467f9cd-5llh8"
Nov 03 20:51:48.216718 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:48.216376    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-57c467f9cd-5llh8" event={"ID":"849d6b39-d800-4658-be04-10929ab6e50b","Type":"ContainerDied","Data":"b3c87049f78615fd86f0fb9b27c43e38034fb6d23cbf0883df8bd34f1701b3cf"}
Nov 03 20:51:48.216718 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:48.216428    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-57c467f9cd-5llh8" event={"ID":"849d6b39-d800-4658-be04-10929ab6e50b","Type":"ContainerDied","Data":"d0aaf5ef5fa52ca2c35ab428ded11a1f0be59f1c4767b4fdd7eab6b9b2159e6a"}
Nov 03 20:51:48.216718 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:48.216451    5570 scope.go:122] "RemoveContainer" containerID="b3c87049f78615fd86f0fb9b27c43e38034fb6d23cbf0883df8bd34f1701b3cf"
Nov 03 20:51:48.225329 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:48.225309    5570 scope.go:122] "RemoveContainer" containerID="b3c87049f78615fd86f0fb9b27c43e38034fb6d23cbf0883df8bd34f1701b3cf"
Nov 03 20:51:48.225584 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:51:48.225564    5570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3c87049f78615fd86f0fb9b27c43e38034fb6d23cbf0883df8bd34f1701b3cf\": container with ID starting with b3c87049f78615fd86f0fb9b27c43e38034fb6d23cbf0883df8bd34f1701b3cf not found: ID does not exist" containerID="b3c87049f78615fd86f0fb9b27c43e38034fb6d23cbf0883df8bd34f1701b3cf"
Nov 03 20:51:48.225637 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:48.225591    5570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3c87049f78615fd86f0fb9b27c43e38034fb6d23cbf0883df8bd34f1701b3cf"} err="failed to get container status \"b3c87049f78615fd86f0fb9b27c43e38034fb6d23cbf0883df8bd34f1701b3cf\": rpc error: code = NotFound desc = could not find container \"b3c87049f78615fd86f0fb9b27c43e38034fb6d23cbf0883df8bd34f1701b3cf\": container with ID starting with b3c87049f78615fd86f0fb9b27c43e38034fb6d23cbf0883df8bd34f1701b3cf not found: ID does not exist"
Nov 03 20:51:48.270173 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:48.270149    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-57c467f9cd-5llh8"]
Nov 03 20:51:48.283209 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:48.283189    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-57c467f9cd-5llh8"]
Nov 03 20:51:49.966810 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:49.966756    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="849d6b39-d800-4658-be04-10929ab6e50b" path="/var/lib/kubelet/pods/849d6b39-d800-4658-be04-10929ab6e50b/volumes"
Nov 03 20:51:57.121694 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:57.121661    5570 patch_prober.go:29] interesting pod/e2e-ait-e2966c90-openshift-default-7cf67d4585-6nznw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.223:15021/healthz/ready\": read tcp 10.134.0.2:56756->10.134.0.223:15021: read: connection reset by peer" start-of-body=
Nov 03 20:51:57.122373 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:57.121732    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-e2966c90-openshift-default-7cf67d4585-6nznw" podUID="b86bb594-b896-443b-aeaf-a734cf38cea0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.223:15021/healthz/ready\": read tcp 10.134.0.2:56756->10.134.0.223:15021: read: connection reset by peer"
Nov 03 20:51:57.226146 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:51:57.226118    5570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8bwhz" containerName="extractor"
Nov 03 20:51:57.288806 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:57.288729    5570 generic.go:356] "Generic (PLEG): container finished" podID="b86bb594-b896-443b-aeaf-a734cf38cea0" containerID="34b5e3bce94b799df83320e4f8616f51e45104a9ef12f3a9437873c128a57e61" exitCode=0
Nov 03 20:51:57.288927 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:57.288799    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e2966c90-openshift-default-7cf67d4585-6nznw" event={"ID":"b86bb594-b896-443b-aeaf-a734cf38cea0","Type":"ContainerDied","Data":"34b5e3bce94b799df83320e4f8616f51e45104a9ef12f3a9437873c128a57e61"}
Nov 03 20:51:57.289184 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:57.289168    5570 scope.go:122] "RemoveContainer" containerID="34b5e3bce94b799df83320e4f8616f51e45104a9ef12f3a9437873c128a57e61"
Nov 03 20:51:57.290323 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:57.290299    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:51:57.290406 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:57.290393    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:51:57.290447 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:57.290434    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:51:58.295661 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:58.295620    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e2966c90-openshift-default-7cf67d4585-6nznw" event={"ID":"b86bb594-b896-443b-aeaf-a734cf38cea0","Type":"ContainerStarted","Data":"a8457985950e3ee31ecfe5d1408e8884e01acd46bc412a6127602641ef2d1933"}
Nov 03 20:51:58.531017 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:58.530980    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-e2966c90-openshift-default-7cf67d4585-6nznw"
Nov 03 20:51:58.531212 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:58.531170    5570 patch_prober.go:29] interesting pod/e2e-ait-e2966c90-openshift-default-7cf67d4585-6nznw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" start-of-body=
Nov 03 20:51:58.531282 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:58.531225    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e2966c90-openshift-default-7cf67d4585-6nznw" podUID="b86bb594-b896-443b-aeaf-a734cf38cea0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused"
Nov 03 20:51:59.531559 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:59.531528    5570 patch_prober.go:29] interesting pod/e2e-ait-e2966c90-openshift-default-7cf67d4585-6nznw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" start-of-body=
Nov 03 20:51:59.531990 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:51:59.531590    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e2966c90-openshift-default-7cf67d4585-6nznw" podUID="b86bb594-b896-443b-aeaf-a734cf38cea0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused"
Nov 03 20:52:00.530886 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:00.530847    5570 patch_prober.go:29] interesting pod/e2e-ait-e2966c90-openshift-default-7cf67d4585-6nznw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" start-of-body=
Nov 03 20:52:00.531109 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:00.530908    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e2966c90-openshift-default-7cf67d4585-6nznw" podUID="b86bb594-b896-443b-aeaf-a734cf38cea0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused"
Nov 03 20:52:01.531423 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:01.531385    5570 patch_prober.go:29] interesting pod/e2e-ait-e2966c90-openshift-default-7cf67d4585-6nznw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" start-of-body=
Nov 03 20:52:01.531849 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:01.531442    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e2966c90-openshift-default-7cf67d4585-6nznw" podUID="b86bb594-b896-443b-aeaf-a734cf38cea0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused"
Nov 03 20:52:02.531070 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:02.531032    5570 patch_prober.go:29] interesting pod/e2e-ait-e2966c90-openshift-default-7cf67d4585-6nznw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" start-of-body=
Nov 03 20:52:02.531249 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:02.531104    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e2966c90-openshift-default-7cf67d4585-6nznw" podUID="b86bb594-b896-443b-aeaf-a734cf38cea0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused"
Nov 03 20:52:03.534193 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:03.534165    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-e2966c90-openshift-default-7cf67d4585-6nznw"
Nov 03 20:52:03.534536 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:03.534422    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-e2966c90-openshift-default-7cf67d4585-6nznw"
Nov 03 20:52:03.535101 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:03.535085    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-e2966c90-openshift-default-7cf67d4585-6nznw"
Nov 03 20:52:13.262894 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:13.262858    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g69f8"]
Nov 03 20:52:13.988012 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:13.987977    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-e2966c90-openshift-default-7cf67d4585-6nznw"]
Nov 03 20:52:13.988242 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:13.988216    5570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-e2966c90-openshift-default-7cf67d4585-6nznw" podUID="b86bb594-b896-443b-aeaf-a734cf38cea0" containerName="istio-proxy" containerID="cri-o://a8457985950e3ee31ecfe5d1408e8884e01acd46bc412a6127602641ef2d1933" gracePeriod=30
Nov 03 20:52:15.681675 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:15.681639    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-authcfg-http-d04674-openshift-default-c886444fc-x6nwp"]
Nov 03 20:52:15.682294 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:15.682273    5570 cpu_manager.go:414] "containerMap: removing container" podUID="849d6b39-d800-4658-be04-10929ab6e50b" containerName="maas-api"
Nov 03 20:52:15.682394 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:15.682296    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="849d6b39-d800-4658-be04-10929ab6e50b" containerName="maas-api"
Nov 03 20:52:15.682459 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:15.682449    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="849d6b39-d800-4658-be04-10929ab6e50b" containerName="maas-api"
Nov 03 20:52:15.686948 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:15.686927    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-d04674-openshift-default-c886444fc-x6nwp"
Nov 03 20:52:15.707842 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:15.707818    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-authcfg-http-d04674-openshift-default-c886444fc-x6nwp"]
Nov 03 20:52:15.718962 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:15.718938    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-authcfg-http-d04674-openshift-default-dockercfg-669n9\""
Nov 03 20:52:15.730351 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:15.730324    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6dc8fcfc-a6e3-46b0-bf73-5cfbdb646188-istio-podinfo\") pod \"e2e-authcfg-http-d04674-openshift-default-c886444fc-x6nwp\" (UID: \"6dc8fcfc-a6e3-46b0-bf73-5cfbdb646188\") " pod="openshift-ingress/e2e-authcfg-http-d04674-openshift-default-c886444fc-x6nwp"
Nov 03 20:52:15.730478 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:15.730354    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6dc8fcfc-a6e3-46b0-bf73-5cfbdb646188-istio-envoy\") pod \"e2e-authcfg-http-d04674-openshift-default-c886444fc-x6nwp\" (UID: \"6dc8fcfc-a6e3-46b0-bf73-5cfbdb646188\") " pod="openshift-ingress/e2e-authcfg-http-d04674-openshift-default-c886444fc-x6nwp"
Nov 03 20:52:15.730478 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:15.730372    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzkgl\" (UniqueName: \"kubernetes.io/projected/6dc8fcfc-a6e3-46b0-bf73-5cfbdb646188-kube-api-access-rzkgl\") pod \"e2e-authcfg-http-d04674-openshift-default-c886444fc-x6nwp\" (UID: \"6dc8fcfc-a6e3-46b0-bf73-5cfbdb646188\") " pod="openshift-ingress/e2e-authcfg-http-d04674-openshift-default-c886444fc-x6nwp"
Nov 03 20:52:15.730478 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:15.730412    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dc8fcfc-a6e3-46b0-bf73-5cfbdb646188-workload-socket\") pod \"e2e-authcfg-http-d04674-openshift-default-c886444fc-x6nwp\" (UID: \"6dc8fcfc-a6e3-46b0-bf73-5cfbdb646188\") " pod="openshift-ingress/e2e-authcfg-http-d04674-openshift-default-c886444fc-x6nwp"
Nov 03 20:52:15.730678 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:15.730539    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6dc8fcfc-a6e3-46b0-bf73-5cfbdb646188-istio-token\") pod \"e2e-authcfg-http-d04674-openshift-default-c886444fc-x6nwp\" (UID: \"6dc8fcfc-a6e3-46b0-bf73-5cfbdb646188\") " pod="openshift-ingress/e2e-authcfg-http-d04674-openshift-default-c886444fc-x6nwp"
Nov 03 20:52:15.730678 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:15.730602    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6dc8fcfc-a6e3-46b0-bf73-5cfbdb646188-istiod-ca-cert\") pod \"e2e-authcfg-http-d04674-openshift-default-c886444fc-x6nwp\" (UID: \"6dc8fcfc-a6e3-46b0-bf73-5cfbdb646188\") " pod="openshift-ingress/e2e-authcfg-http-d04674-openshift-default-c886444fc-x6nwp"
Nov 03 20:52:15.730678 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:15.730630    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dc8fcfc-a6e3-46b0-bf73-5cfbdb646188-credential-socket\") pod \"e2e-authcfg-http-d04674-openshift-default-c886444fc-x6nwp\" (UID: \"6dc8fcfc-a6e3-46b0-bf73-5cfbdb646188\") " pod="openshift-ingress/e2e-authcfg-http-d04674-openshift-default-c886444fc-x6nwp"
Nov 03 20:52:15.730678 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:15.730648    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6dc8fcfc-a6e3-46b0-bf73-5cfbdb646188-workload-certs\") pod \"e2e-authcfg-http-d04674-openshift-default-c886444fc-x6nwp\" (UID: \"6dc8fcfc-a6e3-46b0-bf73-5cfbdb646188\") " pod="openshift-ingress/e2e-authcfg-http-d04674-openshift-default-c886444fc-x6nwp"
Nov 03 20:52:15.730930 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:15.730713    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6dc8fcfc-a6e3-46b0-bf73-5cfbdb646188-istio-data\") pod \"e2e-authcfg-http-d04674-openshift-default-c886444fc-x6nwp\" (UID: \"6dc8fcfc-a6e3-46b0-bf73-5cfbdb646188\") " pod="openshift-ingress/e2e-authcfg-http-d04674-openshift-default-c886444fc-x6nwp"
Nov 03 20:52:15.833735 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:15.833688    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dc8fcfc-a6e3-46b0-bf73-5cfbdb646188-workload-socket\") pod \"e2e-authcfg-http-d04674-openshift-default-c886444fc-x6nwp\" (UID: \"6dc8fcfc-a6e3-46b0-bf73-5cfbdb646188\") " pod="openshift-ingress/e2e-authcfg-http-d04674-openshift-default-c886444fc-x6nwp"
Nov 03 20:52:15.834603 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:15.834566    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6dc8fcfc-a6e3-46b0-bf73-5cfbdb646188-istio-token\") pod \"e2e-authcfg-http-d04674-openshift-default-c886444fc-x6nwp\" (UID: \"6dc8fcfc-a6e3-46b0-bf73-5cfbdb646188\") " pod="openshift-ingress/e2e-authcfg-http-d04674-openshift-default-c886444fc-x6nwp"
Nov 03 20:52:15.834761 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:15.834494    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dc8fcfc-a6e3-46b0-bf73-5cfbdb646188-workload-socket\") pod \"e2e-authcfg-http-d04674-openshift-default-c886444fc-x6nwp\" (UID: \"6dc8fcfc-a6e3-46b0-bf73-5cfbdb646188\") " pod="openshift-ingress/e2e-authcfg-http-d04674-openshift-default-c886444fc-x6nwp"
Nov 03 20:52:15.834761 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:15.834629    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6dc8fcfc-a6e3-46b0-bf73-5cfbdb646188-istiod-ca-cert\") pod \"e2e-authcfg-http-d04674-openshift-default-c886444fc-x6nwp\" (UID: \"6dc8fcfc-a6e3-46b0-bf73-5cfbdb646188\") " pod="openshift-ingress/e2e-authcfg-http-d04674-openshift-default-c886444fc-x6nwp"
Nov 03 20:52:15.834761 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:15.834724    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dc8fcfc-a6e3-46b0-bf73-5cfbdb646188-credential-socket\") pod \"e2e-authcfg-http-d04674-openshift-default-c886444fc-x6nwp\" (UID: \"6dc8fcfc-a6e3-46b0-bf73-5cfbdb646188\") " pod="openshift-ingress/e2e-authcfg-http-d04674-openshift-default-c886444fc-x6nwp"
Nov 03 20:52:15.834761 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:15.834759    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6dc8fcfc-a6e3-46b0-bf73-5cfbdb646188-workload-certs\") pod \"e2e-authcfg-http-d04674-openshift-default-c886444fc-x6nwp\" (UID: \"6dc8fcfc-a6e3-46b0-bf73-5cfbdb646188\") " pod="openshift-ingress/e2e-authcfg-http-d04674-openshift-default-c886444fc-x6nwp"
Nov 03 20:52:15.835041 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:15.834841    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6dc8fcfc-a6e3-46b0-bf73-5cfbdb646188-istio-data\") pod \"e2e-authcfg-http-d04674-openshift-default-c886444fc-x6nwp\" (UID: \"6dc8fcfc-a6e3-46b0-bf73-5cfbdb646188\") " pod="openshift-ingress/e2e-authcfg-http-d04674-openshift-default-c886444fc-x6nwp"
Nov 03 20:52:15.835041 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:15.834914    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6dc8fcfc-a6e3-46b0-bf73-5cfbdb646188-istio-podinfo\") pod \"e2e-authcfg-http-d04674-openshift-default-c886444fc-x6nwp\" (UID: \"6dc8fcfc-a6e3-46b0-bf73-5cfbdb646188\") " pod="openshift-ingress/e2e-authcfg-http-d04674-openshift-default-c886444fc-x6nwp"
Nov 03 20:52:15.835041 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:15.834938    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6dc8fcfc-a6e3-46b0-bf73-5cfbdb646188-istio-envoy\") pod \"e2e-authcfg-http-d04674-openshift-default-c886444fc-x6nwp\" (UID: \"6dc8fcfc-a6e3-46b0-bf73-5cfbdb646188\") " pod="openshift-ingress/e2e-authcfg-http-d04674-openshift-default-c886444fc-x6nwp"
Nov 03 20:52:15.835041 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:15.834963    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzkgl\" (UniqueName: \"kubernetes.io/projected/6dc8fcfc-a6e3-46b0-bf73-5cfbdb646188-kube-api-access-rzkgl\") pod \"e2e-authcfg-http-d04674-openshift-default-c886444fc-x6nwp\" (UID: \"6dc8fcfc-a6e3-46b0-bf73-5cfbdb646188\") " pod="openshift-ingress/e2e-authcfg-http-d04674-openshift-default-c886444fc-x6nwp"
Nov 03 20:52:15.835310 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:15.835286    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6dc8fcfc-a6e3-46b0-bf73-5cfbdb646188-istiod-ca-cert\") pod \"e2e-authcfg-http-d04674-openshift-default-c886444fc-x6nwp\" (UID: \"6dc8fcfc-a6e3-46b0-bf73-5cfbdb646188\") " pod="openshift-ingress/e2e-authcfg-http-d04674-openshift-default-c886444fc-x6nwp"
Nov 03 20:52:15.835625 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:15.835599    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dc8fcfc-a6e3-46b0-bf73-5cfbdb646188-credential-socket\") pod \"e2e-authcfg-http-d04674-openshift-default-c886444fc-x6nwp\" (UID: \"6dc8fcfc-a6e3-46b0-bf73-5cfbdb646188\") " pod="openshift-ingress/e2e-authcfg-http-d04674-openshift-default-c886444fc-x6nwp"
Nov 03 20:52:15.843928 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:15.835904    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6dc8fcfc-a6e3-46b0-bf73-5cfbdb646188-istio-data\") pod \"e2e-authcfg-http-d04674-openshift-default-c886444fc-x6nwp\" (UID: \"6dc8fcfc-a6e3-46b0-bf73-5cfbdb646188\") " pod="openshift-ingress/e2e-authcfg-http-d04674-openshift-default-c886444fc-x6nwp"
Nov 03 20:52:15.843928 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:15.843183    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6dc8fcfc-a6e3-46b0-bf73-5cfbdb646188-istio-envoy\") pod \"e2e-authcfg-http-d04674-openshift-default-c886444fc-x6nwp\" (UID: \"6dc8fcfc-a6e3-46b0-bf73-5cfbdb646188\") " pod="openshift-ingress/e2e-authcfg-http-d04674-openshift-default-c886444fc-x6nwp"
Nov 03 20:52:15.843928 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:15.843241    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6dc8fcfc-a6e3-46b0-bf73-5cfbdb646188-workload-certs\") pod \"e2e-authcfg-http-d04674-openshift-default-c886444fc-x6nwp\" (UID: \"6dc8fcfc-a6e3-46b0-bf73-5cfbdb646188\") " pod="openshift-ingress/e2e-authcfg-http-d04674-openshift-default-c886444fc-x6nwp"
Nov 03 20:52:15.843928 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:15.843471    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6dc8fcfc-a6e3-46b0-bf73-5cfbdb646188-istio-podinfo\") pod \"e2e-authcfg-http-d04674-openshift-default-c886444fc-x6nwp\" (UID: \"6dc8fcfc-a6e3-46b0-bf73-5cfbdb646188\") " pod="openshift-ingress/e2e-authcfg-http-d04674-openshift-default-c886444fc-x6nwp"
Nov 03 20:52:15.855686 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:15.855657    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzkgl\" (UniqueName: \"kubernetes.io/projected/6dc8fcfc-a6e3-46b0-bf73-5cfbdb646188-kube-api-access-rzkgl\") pod \"e2e-authcfg-http-d04674-openshift-default-c886444fc-x6nwp\" (UID: \"6dc8fcfc-a6e3-46b0-bf73-5cfbdb646188\") " pod="openshift-ingress/e2e-authcfg-http-d04674-openshift-default-c886444fc-x6nwp"
Nov 03 20:52:15.855894 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:15.855867    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6dc8fcfc-a6e3-46b0-bf73-5cfbdb646188-istio-token\") pod \"e2e-authcfg-http-d04674-openshift-default-c886444fc-x6nwp\" (UID: \"6dc8fcfc-a6e3-46b0-bf73-5cfbdb646188\") " pod="openshift-ingress/e2e-authcfg-http-d04674-openshift-default-c886444fc-x6nwp"
Nov 03 20:52:16.009368 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:16.009341    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-d04674-openshift-default-c886444fc-x6nwp"
Nov 03 20:52:16.316665 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:16.316636    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-authcfg-http-d04674-openshift-default-c886444fc-x6nwp"]
Nov 03 20:52:16.317537 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:52:16.317509    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6dc8fcfc_a6e3_46b0_bf73_5cfbdb646188.slice/crio-fe2f571557452f2e58c8e70e8c33e72cd202bb5e1a7e615fed9b70255d62e217 WatchSource:0}: Error finding container fe2f571557452f2e58c8e70e8c33e72cd202bb5e1a7e615fed9b70255d62e217: Status 404 returned error can't find the container with id fe2f571557452f2e58c8e70e8c33e72cd202bb5e1a7e615fed9b70255d62e217
Nov 03 20:52:16.319856 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:16.319827    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:52:16.319965 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:16.319894    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:52:16.319965 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:16.319945    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:52:16.389127 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:16.389086    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-d04674-openshift-default-c886444fc-x6nwp" event={"ID":"6dc8fcfc-a6e3-46b0-bf73-5cfbdb646188","Type":"ContainerStarted","Data":"6847c18c9cac1f04a07151d3f5ce77e9d43e6f722794e507eb69a08245f9964c"}
Nov 03 20:52:16.389255 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:16.389159    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-d04674-openshift-default-c886444fc-x6nwp" event={"ID":"6dc8fcfc-a6e3-46b0-bf73-5cfbdb646188","Type":"ContainerStarted","Data":"fe2f571557452f2e58c8e70e8c33e72cd202bb5e1a7e615fed9b70255d62e217"}
Nov 03 20:52:16.433154 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:16.433101    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-authcfg-http-d04674-openshift-default-c886444fc-x6nwp" podStartSLOduration=1.433087801 podStartE2EDuration="1.433087801s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:52:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:52:16.431629874 +0000 UTC m=+2870.584291774" watchObservedRunningTime="2026-11-03 20:52:16.433087801 +0000 UTC m=+2870.585749687"
Nov 03 20:52:17.010170 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:17.010137    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-authcfg-http-d04674-openshift-default-c886444fc-x6nwp"
Nov 03 20:52:17.015214 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:17.015184    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-authcfg-http-d04674-openshift-default-c886444fc-x6nwp"
Nov 03 20:52:17.394867 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:17.394805    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-authcfg-http-d04674-openshift-default-c886444fc-x6nwp"
Nov 03 20:52:17.395632 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:17.395611    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-authcfg-http-d04674-openshift-default-c886444fc-x6nwp"
Nov 03 20:52:18.536559 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:18.536524    5570 patch_prober.go:29] interesting pod/e2e-ait-e2966c90-openshift-default-7cf67d4585-6nznw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 20:52:18.536984 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:18.536579    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-e2966c90-openshift-default-7cf67d4585-6nznw" podUID="b86bb594-b896-443b-aeaf-a734cf38cea0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 20:52:19.250388 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:19.250327    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e2966c90-openshift-default-7cf67d4585-6nznw"
Nov 03 20:52:19.267573 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:19.267552    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b86bb594-b896-443b-aeaf-a734cf38cea0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b86bb594-b896-443b-aeaf-a734cf38cea0-workload-socket\") pod \"b86bb594-b896-443b-aeaf-a734cf38cea0\" (UID: \"b86bb594-b896-443b-aeaf-a734cf38cea0\") "
Nov 03 20:52:19.267704 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:19.267591    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b86bb594-b896-443b-aeaf-a734cf38cea0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b86bb594-b896-443b-aeaf-a734cf38cea0-workload-certs\") pod \"b86bb594-b896-443b-aeaf-a734cf38cea0\" (UID: \"b86bb594-b896-443b-aeaf-a734cf38cea0\") "
Nov 03 20:52:19.267704 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:19.267610    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b86bb594-b896-443b-aeaf-a734cf38cea0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b86bb594-b896-443b-aeaf-a734cf38cea0-credential-socket\") pod \"b86bb594-b896-443b-aeaf-a734cf38cea0\" (UID: \"b86bb594-b896-443b-aeaf-a734cf38cea0\") "
Nov 03 20:52:19.267704 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:19.267633    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b86bb594-b896-443b-aeaf-a734cf38cea0-istio-token\" (UniqueName: \"kubernetes.io/projected/b86bb594-b896-443b-aeaf-a734cf38cea0-istio-token\") pod \"b86bb594-b896-443b-aeaf-a734cf38cea0\" (UID: \"b86bb594-b896-443b-aeaf-a734cf38cea0\") "
Nov 03 20:52:19.267704 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:19.267661    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b86bb594-b896-443b-aeaf-a734cf38cea0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b86bb594-b896-443b-aeaf-a734cf38cea0-istio-envoy\") pod \"b86bb594-b896-443b-aeaf-a734cf38cea0\" (UID: \"b86bb594-b896-443b-aeaf-a734cf38cea0\") "
Nov 03 20:52:19.267704 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:19.267684    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b86bb594-b896-443b-aeaf-a734cf38cea0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b86bb594-b896-443b-aeaf-a734cf38cea0-istio-data\") pod \"b86bb594-b896-443b-aeaf-a734cf38cea0\" (UID: \"b86bb594-b896-443b-aeaf-a734cf38cea0\") "
Nov 03 20:52:19.268043 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:19.267740    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b86bb594-b896-443b-aeaf-a734cf38cea0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b86bb594-b896-443b-aeaf-a734cf38cea0-istio-podinfo\") pod \"b86bb594-b896-443b-aeaf-a734cf38cea0\" (UID: \"b86bb594-b896-443b-aeaf-a734cf38cea0\") "
Nov 03 20:52:19.268043 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:19.267795    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b86bb594-b896-443b-aeaf-a734cf38cea0-kube-api-access-7gsns\" (UniqueName: \"kubernetes.io/projected/b86bb594-b896-443b-aeaf-a734cf38cea0-kube-api-access-7gsns\") pod \"b86bb594-b896-443b-aeaf-a734cf38cea0\" (UID: \"b86bb594-b896-443b-aeaf-a734cf38cea0\") "
Nov 03 20:52:19.268043 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:19.267884    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b86bb594-b896-443b-aeaf-a734cf38cea0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b86bb594-b896-443b-aeaf-a734cf38cea0-istiod-ca-cert\") pod \"b86bb594-b896-443b-aeaf-a734cf38cea0\" (UID: \"b86bb594-b896-443b-aeaf-a734cf38cea0\") "
Nov 03 20:52:19.268043 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:19.267879    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b86bb594-b896-443b-aeaf-a734cf38cea0-workload-socket" pod "b86bb594-b896-443b-aeaf-a734cf38cea0" (UID: "b86bb594-b896-443b-aeaf-a734cf38cea0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:52:19.268043 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:19.267895    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b86bb594-b896-443b-aeaf-a734cf38cea0-credential-socket" pod "b86bb594-b896-443b-aeaf-a734cf38cea0" (UID: "b86bb594-b896-443b-aeaf-a734cf38cea0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:52:19.268043 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:19.267924    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b86bb594-b896-443b-aeaf-a734cf38cea0-workload-certs" pod "b86bb594-b896-443b-aeaf-a734cf38cea0" (UID: "b86bb594-b896-443b-aeaf-a734cf38cea0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:52:19.268337 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:19.268269    5570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b86bb594-b896-443b-aeaf-a734cf38cea0-workload-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:52:19.268337 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:19.268297    5570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b86bb594-b896-443b-aeaf-a734cf38cea0-workload-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:52:19.268337 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:19.268305    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b86bb594-b896-443b-aeaf-a734cf38cea0-istio-data" pod "b86bb594-b896-443b-aeaf-a734cf38cea0" (UID: "b86bb594-b896-443b-aeaf-a734cf38cea0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:52:19.268337 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:19.268314    5570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b86bb594-b896-443b-aeaf-a734cf38cea0-credential-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:52:19.268486 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:19.268242    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b86bb594-b896-443b-aeaf-a734cf38cea0-istiod-ca-cert" pod "b86bb594-b896-443b-aeaf-a734cf38cea0" (UID: "b86bb594-b896-443b-aeaf-a734cf38cea0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:52:19.269985 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:19.269961    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b86bb594-b896-443b-aeaf-a734cf38cea0-istio-podinfo" pod "b86bb594-b896-443b-aeaf-a734cf38cea0" (UID: "b86bb594-b896-443b-aeaf-a734cf38cea0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:52:19.270107 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:19.269999    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b86bb594-b896-443b-aeaf-a734cf38cea0-istio-envoy" pod "b86bb594-b896-443b-aeaf-a734cf38cea0" (UID: "b86bb594-b896-443b-aeaf-a734cf38cea0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:52:19.270335 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:19.270314    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b86bb594-b896-443b-aeaf-a734cf38cea0-istio-token" pod "b86bb594-b896-443b-aeaf-a734cf38cea0" (UID: "b86bb594-b896-443b-aeaf-a734cf38cea0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:52:19.270430 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:19.270342    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b86bb594-b896-443b-aeaf-a734cf38cea0-kube-api-access-7gsns" pod "b86bb594-b896-443b-aeaf-a734cf38cea0" (UID: "b86bb594-b896-443b-aeaf-a734cf38cea0"). InnerVolumeSpecName "kube-api-access-7gsns". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:52:19.369252 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:19.369222    5570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b86bb594-b896-443b-aeaf-a734cf38cea0-istio-token\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:52:19.369252 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:19.369244    5570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b86bb594-b896-443b-aeaf-a734cf38cea0-istio-envoy\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:52:19.369252 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:19.369255    5570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b86bb594-b896-443b-aeaf-a734cf38cea0-istio-data\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:52:19.369449 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:19.369262    5570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b86bb594-b896-443b-aeaf-a734cf38cea0-istio-podinfo\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:52:19.369449 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:19.369272    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gsns\" (UniqueName: \"kubernetes.io/projected/b86bb594-b896-443b-aeaf-a734cf38cea0-kube-api-access-7gsns\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:52:19.369449 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:19.369280    5570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b86bb594-b896-443b-aeaf-a734cf38cea0-istiod-ca-cert\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:52:19.407023 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:19.406991    5570 generic.go:356] "Generic (PLEG): container finished" podID="b86bb594-b896-443b-aeaf-a734cf38cea0" containerID="a8457985950e3ee31ecfe5d1408e8884e01acd46bc412a6127602641ef2d1933" exitCode=0
Nov 03 20:52:19.407164 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:19.407037    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e2966c90-openshift-default-7cf67d4585-6nznw"
Nov 03 20:52:19.407164 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:19.407071    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e2966c90-openshift-default-7cf67d4585-6nznw" event={"ID":"b86bb594-b896-443b-aeaf-a734cf38cea0","Type":"ContainerDied","Data":"a8457985950e3ee31ecfe5d1408e8884e01acd46bc412a6127602641ef2d1933"}
Nov 03 20:52:19.407164 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:19.407126    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e2966c90-openshift-default-7cf67d4585-6nznw" event={"ID":"b86bb594-b896-443b-aeaf-a734cf38cea0","Type":"ContainerDied","Data":"960f315c3dbbc9a79ed1218bdf4e74235178aa2bcac725e51ff558addd5681e5"}
Nov 03 20:52:19.407164 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:19.407156    5570 scope.go:122] "RemoveContainer" containerID="a8457985950e3ee31ecfe5d1408e8884e01acd46bc412a6127602641ef2d1933"
Nov 03 20:52:19.417157 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:19.417137    5570 scope.go:122] "RemoveContainer" containerID="34b5e3bce94b799df83320e4f8616f51e45104a9ef12f3a9437873c128a57e61"
Nov 03 20:52:19.425583 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:19.425566    5570 scope.go:122] "RemoveContainer" containerID="a8457985950e3ee31ecfe5d1408e8884e01acd46bc412a6127602641ef2d1933"
Nov 03 20:52:19.425867 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:52:19.425849    5570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8457985950e3ee31ecfe5d1408e8884e01acd46bc412a6127602641ef2d1933\": container with ID starting with a8457985950e3ee31ecfe5d1408e8884e01acd46bc412a6127602641ef2d1933 not found: ID does not exist" containerID="a8457985950e3ee31ecfe5d1408e8884e01acd46bc412a6127602641ef2d1933"
Nov 03 20:52:19.425930 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:19.425877    5570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8457985950e3ee31ecfe5d1408e8884e01acd46bc412a6127602641ef2d1933"} err="failed to get container status \"a8457985950e3ee31ecfe5d1408e8884e01acd46bc412a6127602641ef2d1933\": rpc error: code = NotFound desc = could not find container \"a8457985950e3ee31ecfe5d1408e8884e01acd46bc412a6127602641ef2d1933\": container with ID starting with a8457985950e3ee31ecfe5d1408e8884e01acd46bc412a6127602641ef2d1933 not found: ID does not exist"
Nov 03 20:52:19.425930 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:19.425895    5570 scope.go:122] "RemoveContainer" containerID="34b5e3bce94b799df83320e4f8616f51e45104a9ef12f3a9437873c128a57e61"
Nov 03 20:52:19.426157 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:52:19.426138    5570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34b5e3bce94b799df83320e4f8616f51e45104a9ef12f3a9437873c128a57e61\": container with ID starting with 34b5e3bce94b799df83320e4f8616f51e45104a9ef12f3a9437873c128a57e61 not found: ID does not exist" containerID="34b5e3bce94b799df83320e4f8616f51e45104a9ef12f3a9437873c128a57e61"
Nov 03 20:52:19.426207 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:19.426164    5570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34b5e3bce94b799df83320e4f8616f51e45104a9ef12f3a9437873c128a57e61"} err="failed to get container status \"34b5e3bce94b799df83320e4f8616f51e45104a9ef12f3a9437873c128a57e61\": rpc error: code = NotFound desc = could not find container \"34b5e3bce94b799df83320e4f8616f51e45104a9ef12f3a9437873c128a57e61\": container with ID starting with 34b5e3bce94b799df83320e4f8616f51e45104a9ef12f3a9437873c128a57e61 not found: ID does not exist"
Nov 03 20:52:19.461001 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:19.460980    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-e2966c90-openshift-default-7cf67d4585-6nznw"]
Nov 03 20:52:19.464708 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:19.464685    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-e2966c90-openshift-default-7cf67d4585-6nznw"]
Nov 03 20:52:19.962958 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:19.962929    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b86bb594-b896-443b-aeaf-a734cf38cea0" path="/var/lib/kubelet/pods/b86bb594-b896-443b-aeaf-a734cf38cea0/volumes"
Nov 03 20:52:20.552584 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:20.552552    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-d04674-77d7654458-x5hxj"]
Nov 03 20:52:20.553022 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:20.553007    5570 cpu_manager.go:414] "containerMap: removing container" podUID="b86bb594-b896-443b-aeaf-a734cf38cea0" containerName="istio-proxy"
Nov 03 20:52:20.553103 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:20.553022    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b86bb594-b896-443b-aeaf-a734cf38cea0" containerName="istio-proxy"
Nov 03 20:52:20.553103 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:20.553037    5570 cpu_manager.go:414] "containerMap: removing container" podUID="b86bb594-b896-443b-aeaf-a734cf38cea0" containerName="istio-proxy"
Nov 03 20:52:20.553103 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:20.553043    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b86bb594-b896-443b-aeaf-a734cf38cea0" containerName="istio-proxy"
Nov 03 20:52:20.553223 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:20.553104    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="b86bb594-b896-443b-aeaf-a734cf38cea0" containerName="istio-proxy"
Nov 03 20:52:20.557726 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:20.557705    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-d04674-77d7654458-x5hxj"
Nov 03 20:52:20.572417 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:20.572392    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-d04674-77d7654458-x5hxj"]
Nov 03 20:52:20.574224 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:20.574200    5570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-authcfg-http-d04674\""
Nov 03 20:52:20.587857 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:20.587835    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzm5k\" (UniqueName: \"kubernetes.io/projected/6c21c686-111d-4b12-9589-c4f64fdd44a9-kube-api-access-fzm5k\") pod \"payload-processing-e2e-authcfg-http-d04674-77d7654458-x5hxj\" (UID: \"6c21c686-111d-4b12-9589-c4f64fdd44a9\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-d04674-77d7654458-x5hxj"
Nov 03 20:52:20.588012 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:20.587992    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c21c686-111d-4b12-9589-c4f64fdd44a9-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-d04674-77d7654458-x5hxj\" (UID: \"6c21c686-111d-4b12-9589-c4f64fdd44a9\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-d04674-77d7654458-x5hxj"
Nov 03 20:52:20.604706 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:20.604686    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-authcfg-http-d04674-dockercfg-lxf5c\""
Nov 03 20:52:20.615450 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:20.615428    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-d04674-57b94ccd4c-ntf66"]
Nov 03 20:52:20.615900 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:20.615886    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="b86bb594-b896-443b-aeaf-a734cf38cea0" containerName="istio-proxy"
Nov 03 20:52:20.618964 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:20.618950    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-d04674-57b94ccd4c-ntf66"
Nov 03 20:52:20.638301 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:20.638278    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-d04674-57b94ccd4c-ntf66"]
Nov 03 20:52:20.698800 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:20.688668    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aec79c41-24ca-408e-8f1a-6783c84203fd-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-d04674-57b94ccd4c-ntf66\" (UID: \"aec79c41-24ca-408e-8f1a-6783c84203fd\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-d04674-57b94ccd4c-ntf66"
Nov 03 20:52:20.698800 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:20.688736    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzm5k\" (UniqueName: \"kubernetes.io/projected/6c21c686-111d-4b12-9589-c4f64fdd44a9-kube-api-access-fzm5k\") pod \"payload-processing-e2e-authcfg-http-d04674-77d7654458-x5hxj\" (UID: \"6c21c686-111d-4b12-9589-c4f64fdd44a9\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-d04674-77d7654458-x5hxj"
Nov 03 20:52:20.698800 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:20.693814    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49252\" (UniqueName: \"kubernetes.io/projected/aec79c41-24ca-408e-8f1a-6783c84203fd-kube-api-access-49252\") pod \"payload-pre-processing-e2e-authcfg-http-d04674-57b94ccd4c-ntf66\" (UID: \"aec79c41-24ca-408e-8f1a-6783c84203fd\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-d04674-57b94ccd4c-ntf66"
Nov 03 20:52:20.698800 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:20.693859    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c21c686-111d-4b12-9589-c4f64fdd44a9-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-d04674-77d7654458-x5hxj\" (UID: \"6c21c686-111d-4b12-9589-c4f64fdd44a9\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-d04674-77d7654458-x5hxj"
Nov 03 20:52:20.698800 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:20.694939    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c21c686-111d-4b12-9589-c4f64fdd44a9-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-d04674-77d7654458-x5hxj\" (UID: \"6c21c686-111d-4b12-9589-c4f64fdd44a9\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-d04674-77d7654458-x5hxj"
Nov 03 20:52:20.717629 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:20.717592    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzm5k\" (UniqueName: \"kubernetes.io/projected/6c21c686-111d-4b12-9589-c4f64fdd44a9-kube-api-access-fzm5k\") pod \"payload-processing-e2e-authcfg-http-d04674-77d7654458-x5hxj\" (UID: \"6c21c686-111d-4b12-9589-c4f64fdd44a9\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-d04674-77d7654458-x5hxj"
Nov 03 20:52:20.794756 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:20.794718    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aec79c41-24ca-408e-8f1a-6783c84203fd-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-d04674-57b94ccd4c-ntf66\" (UID: \"aec79c41-24ca-408e-8f1a-6783c84203fd\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-d04674-57b94ccd4c-ntf66"
Nov 03 20:52:20.794965 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:20.794838    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49252\" (UniqueName: \"kubernetes.io/projected/aec79c41-24ca-408e-8f1a-6783c84203fd-kube-api-access-49252\") pod \"payload-pre-processing-e2e-authcfg-http-d04674-57b94ccd4c-ntf66\" (UID: \"aec79c41-24ca-408e-8f1a-6783c84203fd\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-d04674-57b94ccd4c-ntf66"
Nov 03 20:52:20.795471 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:20.795443    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aec79c41-24ca-408e-8f1a-6783c84203fd-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-d04674-57b94ccd4c-ntf66\" (UID: \"aec79c41-24ca-408e-8f1a-6783c84203fd\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-d04674-57b94ccd4c-ntf66"
Nov 03 20:52:20.815107 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:20.815051    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49252\" (UniqueName: \"kubernetes.io/projected/aec79c41-24ca-408e-8f1a-6783c84203fd-kube-api-access-49252\") pod \"payload-pre-processing-e2e-authcfg-http-d04674-57b94ccd4c-ntf66\" (UID: \"aec79c41-24ca-408e-8f1a-6783c84203fd\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-d04674-57b94ccd4c-ntf66"
Nov 03 20:52:20.871078 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:20.871053    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-d04674-77d7654458-x5hxj"
Nov 03 20:52:20.930906 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:20.930881    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-d04674-57b94ccd4c-ntf66"
Nov 03 20:52:21.170269 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:21.170232    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-d04674-77d7654458-x5hxj"]
Nov 03 20:52:21.170958 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:52:21.170932    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c21c686_111d_4b12_9589_c4f64fdd44a9.slice/crio-d3cb52b297e32681c55ddd39ace9ba7f4c2f4742dfc356a2659bb7eb6b7c824e WatchSource:0}: Error finding container d3cb52b297e32681c55ddd39ace9ba7f4c2f4742dfc356a2659bb7eb6b7c824e: Status 404 returned error can't find the container with id d3cb52b297e32681c55ddd39ace9ba7f4c2f4742dfc356a2659bb7eb6b7c824e
Nov 03 20:52:21.233231 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:21.233206    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-d04674-57b94ccd4c-ntf66"]
Nov 03 20:52:21.233893 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:52:21.233871    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaec79c41_24ca_408e_8f1a_6783c84203fd.slice/crio-7593342e0a29bf220a55117a66b8624c39ea95be13b325c3ecd41cca5f00c8fc WatchSource:0}: Error finding container 7593342e0a29bf220a55117a66b8624c39ea95be13b325c3ecd41cca5f00c8fc: Status 404 returned error can't find the container with id 7593342e0a29bf220a55117a66b8624c39ea95be13b325c3ecd41cca5f00c8fc
Nov 03 20:52:21.423755 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:21.423669    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-d04674-77d7654458-x5hxj" event={"ID":"6c21c686-111d-4b12-9589-c4f64fdd44a9","Type":"ContainerStarted","Data":"60923e57dd90435bf15330011ae26680202870a922d0673274be632d6efc31b7"}
Nov 03 20:52:21.423755 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:21.423729    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-d04674-77d7654458-x5hxj" event={"ID":"6c21c686-111d-4b12-9589-c4f64fdd44a9","Type":"ContainerStarted","Data":"d3cb52b297e32681c55ddd39ace9ba7f4c2f4742dfc356a2659bb7eb6b7c824e"}
Nov 03 20:52:21.424049 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:21.423887    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-d04674-77d7654458-x5hxj"
Nov 03 20:52:21.425327 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:21.425303    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-d04674-57b94ccd4c-ntf66" event={"ID":"aec79c41-24ca-408e-8f1a-6783c84203fd","Type":"ContainerStarted","Data":"b1b7f49d48e7581b232e0f7b779355845bfbb60c98940f6e2d83ddd7ce5a1904"}
Nov 03 20:52:21.425470 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:21.425346    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-d04674-57b94ccd4c-ntf66" event={"ID":"aec79c41-24ca-408e-8f1a-6783c84203fd","Type":"ContainerStarted","Data":"7593342e0a29bf220a55117a66b8624c39ea95be13b325c3ecd41cca5f00c8fc"}
Nov 03 20:52:21.425470 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:21.425427    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-d04674-57b94ccd4c-ntf66"
Nov 03 20:52:21.508273 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:21.508234    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-d04674-57b94ccd4c-ntf66" podStartSLOduration=1.5082214409999999 podStartE2EDuration="1.508221441s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:52:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:52:21.506489483 +0000 UTC m=+2875.659151368" watchObservedRunningTime="2026-11-03 20:52:21.508221441 +0000 UTC m=+2875.660883356"
Nov 03 20:52:21.508678 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:21.508657    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-authcfg-http-d04674-77d7654458-x5hxj" podStartSLOduration=1.508650145 podStartE2EDuration="1.508650145s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:52:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:52:21.465664709 +0000 UTC m=+2875.618326595" watchObservedRunningTime="2026-11-03 20:52:21.508650145 +0000 UTC m=+2875.661312030"
Nov 03 20:52:24.038203 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:24.038171    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g69f8"]
Nov 03 20:52:27.226756 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:52:27.226720    5570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8bwhz" containerName="extractor"
Nov 03 20:52:32.434868 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:32.434840    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-d04674-57b94ccd4c-ntf66"
Nov 03 20:52:32.435331 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:32.434993    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-d04674-77d7654458-x5hxj"
Nov 03 20:52:32.488844 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:32.488817    5570 scope.go:122] "RemoveContainer" containerID="8611c520606be4969d35b1a72da4d6ca112b4e8d4f1777a323cd401810655b36"
Nov 03 20:52:32.501076 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:32.500658    5570 scope.go:122] "RemoveContainer" containerID="5f593ca61f03988073772d109dab1e466545fe3f72c4d904fd37f9d26ae3283b"
Nov 03 20:52:32.509983 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:32.509958    5570 scope.go:122] "RemoveContainer" containerID="8d6f14cc58f2af1a5a47d0e79d092137f2e24702e6c1317bab260ada50eb49d4"
Nov 03 20:52:33.085715 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:33.085676    5570 patch_prober.go:29] interesting pod/e2e-authcfg-http-d04674-openshift-default-c886444fc-x6nwp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.225:15021/healthz/ready\": read tcp 10.134.0.2:35498->10.134.0.225:15021: read: connection reset by peer" start-of-body=
Nov 03 20:52:33.085946 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:33.085746    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-authcfg-http-d04674-openshift-default-c886444fc-x6nwp" podUID="6dc8fcfc-a6e3-46b0-bf73-5cfbdb646188" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.225:15021/healthz/ready\": read tcp 10.134.0.2:35498->10.134.0.225:15021: read: connection reset by peer"
Nov 03 20:52:33.543172 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:33.543147    5570 generic.go:356] "Generic (PLEG): container finished" podID="6dc8fcfc-a6e3-46b0-bf73-5cfbdb646188" containerID="6847c18c9cac1f04a07151d3f5ce77e9d43e6f722794e507eb69a08245f9964c" exitCode=0
Nov 03 20:52:33.543496 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:33.543180    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-d04674-openshift-default-c886444fc-x6nwp" event={"ID":"6dc8fcfc-a6e3-46b0-bf73-5cfbdb646188","Type":"ContainerDied","Data":"6847c18c9cac1f04a07151d3f5ce77e9d43e6f722794e507eb69a08245f9964c"}
Nov 03 20:52:33.543539 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:33.543493    5570 scope.go:122] "RemoveContainer" containerID="6847c18c9cac1f04a07151d3f5ce77e9d43e6f722794e507eb69a08245f9964c"
Nov 03 20:52:33.545021 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:33.544988    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:52:33.545140 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:33.545068    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:52:33.545140 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:33.545105    5570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 20:52:34.566759 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:34.566720    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-d04674-openshift-default-c886444fc-x6nwp" event={"ID":"6dc8fcfc-a6e3-46b0-bf73-5cfbdb646188","Type":"ContainerStarted","Data":"a567a44d5fe0a9a1aed0c687a18675f707ab6c246a432c353f6a9361ce3b04b6"}
Nov 03 20:52:35.009715 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:35.009678    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-authcfg-http-d04674-openshift-default-c886444fc-x6nwp"
Nov 03 20:52:35.009930 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:35.009831    5570 patch_prober.go:29] interesting pod/e2e-authcfg-http-d04674-openshift-default-c886444fc-x6nwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" start-of-body=
Nov 03 20:52:35.009930 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:35.009879    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-d04674-openshift-default-c886444fc-x6nwp" podUID="6dc8fcfc-a6e3-46b0-bf73-5cfbdb646188" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused"
Nov 03 20:52:36.010232 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:36.010193    5570 patch_prober.go:29] interesting pod/e2e-authcfg-http-d04674-openshift-default-c886444fc-x6nwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" start-of-body=
Nov 03 20:52:36.010713 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:36.010251    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-d04674-openshift-default-c886444fc-x6nwp" podUID="6dc8fcfc-a6e3-46b0-bf73-5cfbdb646188" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused"
Nov 03 20:52:37.009971 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:37.009881    5570 patch_prober.go:29] interesting pod/e2e-authcfg-http-d04674-openshift-default-c886444fc-x6nwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" start-of-body=
Nov 03 20:52:37.009971 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:37.009958    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-d04674-openshift-default-c886444fc-x6nwp" podUID="6dc8fcfc-a6e3-46b0-bf73-5cfbdb646188" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused"
Nov 03 20:52:38.010645 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:38.010610    5570 patch_prober.go:29] interesting pod/e2e-authcfg-http-d04674-openshift-default-c886444fc-x6nwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" start-of-body=
Nov 03 20:52:38.011072 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:38.010670    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-d04674-openshift-default-c886444fc-x6nwp" podUID="6dc8fcfc-a6e3-46b0-bf73-5cfbdb646188" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused"
Nov 03 20:52:39.010601 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:39.010567    5570 patch_prober.go:29] interesting pod/e2e-authcfg-http-d04674-openshift-default-c886444fc-x6nwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" start-of-body=
Nov 03 20:52:39.010805 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:39.010618    5570 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-d04674-openshift-default-c886444fc-x6nwp" podUID="6dc8fcfc-a6e3-46b0-bf73-5cfbdb646188" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused"
Nov 03 20:52:40.014001 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:40.013974    5570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-authcfg-http-d04674-openshift-default-c886444fc-x6nwp"
Nov 03 20:52:40.014479 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:40.014248    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-authcfg-http-d04674-openshift-default-c886444fc-x6nwp"
Nov 03 20:52:40.014845 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:52:40.014828    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-authcfg-http-d04674-openshift-default-c886444fc-x6nwp"
Nov 03 20:52:57.226534 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:52:57.226501    5570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8bwhz" containerName="extractor"
Nov 03 20:53:05.026746 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:53:05.026703    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-8wdh8"]
Nov 03 20:53:05.039612 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:53:05.039582    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-8wdh8"]
Nov 03 20:53:05.963526 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:53:05.963490    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44794ebc-c5e7-47fc-a0d3-0cd1d9e63e53" path="/var/lib/kubelet/pods/44794ebc-c5e7-47fc-a0d3-0cd1d9e63e53/volumes"
Nov 03 20:53:27.226268 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:53:27.226223    5570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8bwhz" containerName="extractor"
Nov 03 20:53:57.231439 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:53:57.231402    5570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8bwhz" containerName="extractor"
Nov 03 20:54:03.977929 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:54:03.977898    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-d04674-77d7654458-x5hxj"]
Nov 03 20:54:04.009451 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:54:04.009423    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-d04674-57b94ccd4c-ntf66"]
Nov 03 20:54:08.978887 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:54:08.978838    5570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-authcfg-http-d04674-77d7654458-x5hxj" podUID="6c21c686-111d-4b12-9589-c4f64fdd44a9" containerName="payload-processing" containerID="cri-o://60923e57dd90435bf15330011ae26680202870a922d0673274be632d6efc31b7" gracePeriod=25
Nov 03 20:54:09.009738 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:54:09.009696    5570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-d04674-57b94ccd4c-ntf66" podUID="aec79c41-24ca-408e-8f1a-6783c84203fd" containerName="payload-pre-processing" containerID="cri-o://b1b7f49d48e7581b232e0f7b779355845bfbb60c98940f6e2d83ddd7ce5a1904" gracePeriod=25
Nov 03 20:54:09.272507 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:54:09.272478    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-d04674-77d7654458-x5hxj"
Nov 03 20:54:09.299390 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:54:09.299364    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-d04674-57b94ccd4c-ntf66"
Nov 03 20:54:09.381541 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:54:09.381508    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aec79c41-24ca-408e-8f1a-6783c84203fd-kube-api-access-49252\" (UniqueName: \"kubernetes.io/projected/aec79c41-24ca-408e-8f1a-6783c84203fd-kube-api-access-49252\") pod \"aec79c41-24ca-408e-8f1a-6783c84203fd\" (UID: \"aec79c41-24ca-408e-8f1a-6783c84203fd\") "
Nov 03 20:54:09.381727 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:54:09.381638    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c21c686-111d-4b12-9589-c4f64fdd44a9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c21c686-111d-4b12-9589-c4f64fdd44a9-plugins-config-volume\") pod \"6c21c686-111d-4b12-9589-c4f64fdd44a9\" (UID: \"6c21c686-111d-4b12-9589-c4f64fdd44a9\") "
Nov 03 20:54:09.381727 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:54:09.381700    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c21c686-111d-4b12-9589-c4f64fdd44a9-kube-api-access-fzm5k\" (UniqueName: \"kubernetes.io/projected/6c21c686-111d-4b12-9589-c4f64fdd44a9-kube-api-access-fzm5k\") pod \"6c21c686-111d-4b12-9589-c4f64fdd44a9\" (UID: \"6c21c686-111d-4b12-9589-c4f64fdd44a9\") "
Nov 03 20:54:09.381892 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:54:09.381813    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aec79c41-24ca-408e-8f1a-6783c84203fd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aec79c41-24ca-408e-8f1a-6783c84203fd-plugins-config-volume\") pod \"aec79c41-24ca-408e-8f1a-6783c84203fd\" (UID: \"aec79c41-24ca-408e-8f1a-6783c84203fd\") "
Nov 03 20:54:09.382104 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:54:09.382078    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c21c686-111d-4b12-9589-c4f64fdd44a9-plugins-config-volume" pod "6c21c686-111d-4b12-9589-c4f64fdd44a9" (UID: "6c21c686-111d-4b12-9589-c4f64fdd44a9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:54:09.382360 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:54:09.382283    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aec79c41-24ca-408e-8f1a-6783c84203fd-plugins-config-volume" pod "aec79c41-24ca-408e-8f1a-6783c84203fd" (UID: "aec79c41-24ca-408e-8f1a-6783c84203fd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:54:09.384291 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:54:09.384270    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c21c686-111d-4b12-9589-c4f64fdd44a9-kube-api-access-fzm5k" pod "6c21c686-111d-4b12-9589-c4f64fdd44a9" (UID: "6c21c686-111d-4b12-9589-c4f64fdd44a9"). InnerVolumeSpecName "kube-api-access-fzm5k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:54:09.384357 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:54:09.384302    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aec79c41-24ca-408e-8f1a-6783c84203fd-kube-api-access-49252" pod "aec79c41-24ca-408e-8f1a-6783c84203fd" (UID: "aec79c41-24ca-408e-8f1a-6783c84203fd"). InnerVolumeSpecName "kube-api-access-49252". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:54:09.483238 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:54:09.483210    5570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c21c686-111d-4b12-9589-c4f64fdd44a9-plugins-config-volume\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:54:09.483238 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:54:09.483234    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzm5k\" (UniqueName: \"kubernetes.io/projected/6c21c686-111d-4b12-9589-c4f64fdd44a9-kube-api-access-fzm5k\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:54:09.483238 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:54:09.483244    5570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aec79c41-24ca-408e-8f1a-6783c84203fd-plugins-config-volume\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:54:09.483469 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:54:09.483252    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49252\" (UniqueName: \"kubernetes.io/projected/aec79c41-24ca-408e-8f1a-6783c84203fd-kube-api-access-49252\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:54:10.076551 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:54:10.076521    5570 generic.go:356] "Generic (PLEG): container finished" podID="6c21c686-111d-4b12-9589-c4f64fdd44a9" containerID="60923e57dd90435bf15330011ae26680202870a922d0673274be632d6efc31b7" exitCode=0
Nov 03 20:54:10.076997 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:54:10.076572    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-d04674-77d7654458-x5hxj"
Nov 03 20:54:10.076997 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:54:10.076604    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-d04674-77d7654458-x5hxj" event={"ID":"6c21c686-111d-4b12-9589-c4f64fdd44a9","Type":"ContainerDied","Data":"60923e57dd90435bf15330011ae26680202870a922d0673274be632d6efc31b7"}
Nov 03 20:54:10.076997 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:54:10.076661    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-d04674-77d7654458-x5hxj" event={"ID":"6c21c686-111d-4b12-9589-c4f64fdd44a9","Type":"ContainerDied","Data":"d3cb52b297e32681c55ddd39ace9ba7f4c2f4742dfc356a2659bb7eb6b7c824e"}
Nov 03 20:54:10.076997 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:54:10.076692    5570 scope.go:122] "RemoveContainer" containerID="60923e57dd90435bf15330011ae26680202870a922d0673274be632d6efc31b7"
Nov 03 20:54:10.078028 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:54:10.077856    5570 generic.go:356] "Generic (PLEG): container finished" podID="aec79c41-24ca-408e-8f1a-6783c84203fd" containerID="b1b7f49d48e7581b232e0f7b779355845bfbb60c98940f6e2d83ddd7ce5a1904" exitCode=0
Nov 03 20:54:10.078028 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:54:10.077896    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-d04674-57b94ccd4c-ntf66"
Nov 03 20:54:10.078028 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:54:10.077903    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-d04674-57b94ccd4c-ntf66" event={"ID":"aec79c41-24ca-408e-8f1a-6783c84203fd","Type":"ContainerDied","Data":"b1b7f49d48e7581b232e0f7b779355845bfbb60c98940f6e2d83ddd7ce5a1904"}
Nov 03 20:54:10.078028 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:54:10.077955    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-d04674-57b94ccd4c-ntf66" event={"ID":"aec79c41-24ca-408e-8f1a-6783c84203fd","Type":"ContainerDied","Data":"7593342e0a29bf220a55117a66b8624c39ea95be13b325c3ecd41cca5f00c8fc"}
Nov 03 20:54:10.085680 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:54:10.085666    5570 scope.go:122] "RemoveContainer" containerID="60923e57dd90435bf15330011ae26680202870a922d0673274be632d6efc31b7"
Nov 03 20:54:10.085970 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:54:10.085948    5570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60923e57dd90435bf15330011ae26680202870a922d0673274be632d6efc31b7\": container with ID starting with 60923e57dd90435bf15330011ae26680202870a922d0673274be632d6efc31b7 not found: ID does not exist" containerID="60923e57dd90435bf15330011ae26680202870a922d0673274be632d6efc31b7"
Nov 03 20:54:10.086040 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:54:10.085979    5570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60923e57dd90435bf15330011ae26680202870a922d0673274be632d6efc31b7"} err="failed to get container status \"60923e57dd90435bf15330011ae26680202870a922d0673274be632d6efc31b7\": rpc error: code = NotFound desc = could not find container \"60923e57dd90435bf15330011ae26680202870a922d0673274be632d6efc31b7\": container with ID starting with 60923e57dd90435bf15330011ae26680202870a922d0673274be632d6efc31b7 not found: ID does not exist"
Nov 03 20:54:10.086040 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:54:10.085997    5570 scope.go:122] "RemoveContainer" containerID="b1b7f49d48e7581b232e0f7b779355845bfbb60c98940f6e2d83ddd7ce5a1904"
Nov 03 20:54:10.093784 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:54:10.093756    5570 scope.go:122] "RemoveContainer" containerID="b1b7f49d48e7581b232e0f7b779355845bfbb60c98940f6e2d83ddd7ce5a1904"
Nov 03 20:54:10.094147 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:54:10.094129    5570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1b7f49d48e7581b232e0f7b779355845bfbb60c98940f6e2d83ddd7ce5a1904\": container with ID starting with b1b7f49d48e7581b232e0f7b779355845bfbb60c98940f6e2d83ddd7ce5a1904 not found: ID does not exist" containerID="b1b7f49d48e7581b232e0f7b779355845bfbb60c98940f6e2d83ddd7ce5a1904"
Nov 03 20:54:10.094209 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:54:10.094153    5570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1b7f49d48e7581b232e0f7b779355845bfbb60c98940f6e2d83ddd7ce5a1904"} err="failed to get container status \"b1b7f49d48e7581b232e0f7b779355845bfbb60c98940f6e2d83ddd7ce5a1904\": rpc error: code = NotFound desc = could not find container \"b1b7f49d48e7581b232e0f7b779355845bfbb60c98940f6e2d83ddd7ce5a1904\": container with ID starting with b1b7f49d48e7581b232e0f7b779355845bfbb60c98940f6e2d83ddd7ce5a1904 not found: ID does not exist"
Nov 03 20:54:10.129860 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:54:10.129836    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-d04674-77d7654458-x5hxj"]
Nov 03 20:54:10.143173 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:54:10.143154    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-d04674-77d7654458-x5hxj"]
Nov 03 20:54:10.178177 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:54:10.178159    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-d04674-57b94ccd4c-ntf66"]
Nov 03 20:54:10.180448 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:54:10.180427    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-d04674-57b94ccd4c-ntf66"]
Nov 03 20:54:11.962734 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:54:11.962705    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c21c686-111d-4b12-9589-c4f64fdd44a9" path="/var/lib/kubelet/pods/6c21c686-111d-4b12-9589-c4f64fdd44a9/volumes"
Nov 03 20:54:11.963115 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:54:11.963077    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aec79c41-24ca-408e-8f1a-6783c84203fd" path="/var/lib/kubelet/pods/aec79c41-24ca-408e-8f1a-6783c84203fd/volumes"
Nov 03 20:54:14.166747 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:54:14.166711    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g69f8"]
Nov 03 20:54:19.399861 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:54:19.399816    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-authcfg-http-d04674-openshift-default-c886444fc-x6nwp"]
Nov 03 20:54:19.400376 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:54:19.400107    5570 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-authcfg-http-d04674-openshift-default-c886444fc-x6nwp" podUID="6dc8fcfc-a6e3-46b0-bf73-5cfbdb646188" containerName="istio-proxy" containerID="cri-o://a567a44d5fe0a9a1aed0c687a18675f707ab6c246a432c353f6a9361ce3b04b6" gracePeriod=30
Nov 03 20:54:21.970584 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:54:21.970541    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g69f8"]
Nov 03 20:54:25.123132 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:54:25.123099    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-d04674-openshift-default-c886444fc-x6nwp"
Nov 03 20:54:25.168987 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:54:25.168950    5570 generic.go:356] "Generic (PLEG): container finished" podID="6dc8fcfc-a6e3-46b0-bf73-5cfbdb646188" containerID="a567a44d5fe0a9a1aed0c687a18675f707ab6c246a432c353f6a9361ce3b04b6" exitCode=0
Nov 03 20:54:25.169148 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:54:25.169013    5570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-d04674-openshift-default-c886444fc-x6nwp"
Nov 03 20:54:25.169148 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:54:25.169042    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-d04674-openshift-default-c886444fc-x6nwp" event={"ID":"6dc8fcfc-a6e3-46b0-bf73-5cfbdb646188","Type":"ContainerDied","Data":"a567a44d5fe0a9a1aed0c687a18675f707ab6c246a432c353f6a9361ce3b04b6"}
Nov 03 20:54:25.169148 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:54:25.169107    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-d04674-openshift-default-c886444fc-x6nwp" event={"ID":"6dc8fcfc-a6e3-46b0-bf73-5cfbdb646188","Type":"ContainerDied","Data":"fe2f571557452f2e58c8e70e8c33e72cd202bb5e1a7e615fed9b70255d62e217"}
Nov 03 20:54:25.169148 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:54:25.169130    5570 scope.go:122] "RemoveContainer" containerID="a567a44d5fe0a9a1aed0c687a18675f707ab6c246a432c353f6a9361ce3b04b6"
Nov 03 20:54:25.178177 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:54:25.178153    5570 scope.go:122] "RemoveContainer" containerID="6847c18c9cac1f04a07151d3f5ce77e9d43e6f722794e507eb69a08245f9964c"
Nov 03 20:54:25.188423 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:54:25.188403    5570 scope.go:122] "RemoveContainer" containerID="a567a44d5fe0a9a1aed0c687a18675f707ab6c246a432c353f6a9361ce3b04b6"
Nov 03 20:54:25.188736 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:54:25.188707    5570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a567a44d5fe0a9a1aed0c687a18675f707ab6c246a432c353f6a9361ce3b04b6\": container with ID starting with a567a44d5fe0a9a1aed0c687a18675f707ab6c246a432c353f6a9361ce3b04b6 not found: ID does not exist" containerID="a567a44d5fe0a9a1aed0c687a18675f707ab6c246a432c353f6a9361ce3b04b6"
Nov 03 20:54:25.188842 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:54:25.188738    5570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a567a44d5fe0a9a1aed0c687a18675f707ab6c246a432c353f6a9361ce3b04b6"} err="failed to get container status \"a567a44d5fe0a9a1aed0c687a18675f707ab6c246a432c353f6a9361ce3b04b6\": rpc error: code = NotFound desc = could not find container \"a567a44d5fe0a9a1aed0c687a18675f707ab6c246a432c353f6a9361ce3b04b6\": container with ID starting with a567a44d5fe0a9a1aed0c687a18675f707ab6c246a432c353f6a9361ce3b04b6 not found: ID does not exist"
Nov 03 20:54:25.188842 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:54:25.188756    5570 scope.go:122] "RemoveContainer" containerID="6847c18c9cac1f04a07151d3f5ce77e9d43e6f722794e507eb69a08245f9964c"
Nov 03 20:54:25.189030 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:54:25.189012    5570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6847c18c9cac1f04a07151d3f5ce77e9d43e6f722794e507eb69a08245f9964c\": container with ID starting with 6847c18c9cac1f04a07151d3f5ce77e9d43e6f722794e507eb69a08245f9964c not found: ID does not exist" containerID="6847c18c9cac1f04a07151d3f5ce77e9d43e6f722794e507eb69a08245f9964c"
Nov 03 20:54:25.189108 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:54:25.189035    5570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6847c18c9cac1f04a07151d3f5ce77e9d43e6f722794e507eb69a08245f9964c"} err="failed to get container status \"6847c18c9cac1f04a07151d3f5ce77e9d43e6f722794e507eb69a08245f9964c\": rpc error: code = NotFound desc = could not find container \"6847c18c9cac1f04a07151d3f5ce77e9d43e6f722794e507eb69a08245f9964c\": container with ID starting with 6847c18c9cac1f04a07151d3f5ce77e9d43e6f722794e507eb69a08245f9964c not found: ID does not exist"
Nov 03 20:54:25.306617 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:54:25.306595    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6dc8fcfc-a6e3-46b0-bf73-5cfbdb646188-kube-api-access-rzkgl\" (UniqueName: \"kubernetes.io/projected/6dc8fcfc-a6e3-46b0-bf73-5cfbdb646188-kube-api-access-rzkgl\") pod \"6dc8fcfc-a6e3-46b0-bf73-5cfbdb646188\" (UID: \"6dc8fcfc-a6e3-46b0-bf73-5cfbdb646188\") "
Nov 03 20:54:25.306714 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:54:25.306651    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6dc8fcfc-a6e3-46b0-bf73-5cfbdb646188-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6dc8fcfc-a6e3-46b0-bf73-5cfbdb646188-workload-certs\") pod \"6dc8fcfc-a6e3-46b0-bf73-5cfbdb646188\" (UID: \"6dc8fcfc-a6e3-46b0-bf73-5cfbdb646188\") "
Nov 03 20:54:25.306714 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:54:25.306679    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6dc8fcfc-a6e3-46b0-bf73-5cfbdb646188-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6dc8fcfc-a6e3-46b0-bf73-5cfbdb646188-istiod-ca-cert\") pod \"6dc8fcfc-a6e3-46b0-bf73-5cfbdb646188\" (UID: \"6dc8fcfc-a6e3-46b0-bf73-5cfbdb646188\") "
Nov 03 20:54:25.306714 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:54:25.306700    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6dc8fcfc-a6e3-46b0-bf73-5cfbdb646188-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dc8fcfc-a6e3-46b0-bf73-5cfbdb646188-credential-socket\") pod \"6dc8fcfc-a6e3-46b0-bf73-5cfbdb646188\" (UID: \"6dc8fcfc-a6e3-46b0-bf73-5cfbdb646188\") "
Nov 03 20:54:25.306930 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:54:25.306740    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6dc8fcfc-a6e3-46b0-bf73-5cfbdb646188-istio-token\" (UniqueName: \"kubernetes.io/projected/6dc8fcfc-a6e3-46b0-bf73-5cfbdb646188-istio-token\") pod \"6dc8fcfc-a6e3-46b0-bf73-5cfbdb646188\" (UID: \"6dc8fcfc-a6e3-46b0-bf73-5cfbdb646188\") "
Nov 03 20:54:25.306930 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:54:25.306799    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6dc8fcfc-a6e3-46b0-bf73-5cfbdb646188-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6dc8fcfc-a6e3-46b0-bf73-5cfbdb646188-istio-data\") pod \"6dc8fcfc-a6e3-46b0-bf73-5cfbdb646188\" (UID: \"6dc8fcfc-a6e3-46b0-bf73-5cfbdb646188\") "
Nov 03 20:54:25.306930 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:54:25.306848    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6dc8fcfc-a6e3-46b0-bf73-5cfbdb646188-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dc8fcfc-a6e3-46b0-bf73-5cfbdb646188-workload-socket\") pod \"6dc8fcfc-a6e3-46b0-bf73-5cfbdb646188\" (UID: \"6dc8fcfc-a6e3-46b0-bf73-5cfbdb646188\") "
Nov 03 20:54:25.306930 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:54:25.306892    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6dc8fcfc-a6e3-46b0-bf73-5cfbdb646188-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6dc8fcfc-a6e3-46b0-bf73-5cfbdb646188-istio-podinfo\") pod \"6dc8fcfc-a6e3-46b0-bf73-5cfbdb646188\" (UID: \"6dc8fcfc-a6e3-46b0-bf73-5cfbdb646188\") "
Nov 03 20:54:25.307142 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:54:25.306978    5570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6dc8fcfc-a6e3-46b0-bf73-5cfbdb646188-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6dc8fcfc-a6e3-46b0-bf73-5cfbdb646188-istio-envoy\") pod \"6dc8fcfc-a6e3-46b0-bf73-5cfbdb646188\" (UID: \"6dc8fcfc-a6e3-46b0-bf73-5cfbdb646188\") "
Nov 03 20:54:25.307142 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:54:25.307084    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dc8fcfc-a6e3-46b0-bf73-5cfbdb646188-workload-certs" pod "6dc8fcfc-a6e3-46b0-bf73-5cfbdb646188" (UID: "6dc8fcfc-a6e3-46b0-bf73-5cfbdb646188"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:54:25.307142 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:54:25.307097    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dc8fcfc-a6e3-46b0-bf73-5cfbdb646188-credential-socket" pod "6dc8fcfc-a6e3-46b0-bf73-5cfbdb646188" (UID: "6dc8fcfc-a6e3-46b0-bf73-5cfbdb646188"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:54:25.307296 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:54:25.307172    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6dc8fcfc-a6e3-46b0-bf73-5cfbdb646188-istiod-ca-cert" pod "6dc8fcfc-a6e3-46b0-bf73-5cfbdb646188" (UID: "6dc8fcfc-a6e3-46b0-bf73-5cfbdb646188"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:54:25.307352 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:54:25.307304    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dc8fcfc-a6e3-46b0-bf73-5cfbdb646188-workload-socket" pod "6dc8fcfc-a6e3-46b0-bf73-5cfbdb646188" (UID: "6dc8fcfc-a6e3-46b0-bf73-5cfbdb646188"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:54:25.307495 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:54:25.307475    5570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6dc8fcfc-a6e3-46b0-bf73-5cfbdb646188-workload-certs\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:54:25.307561 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:54:25.307490    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dc8fcfc-a6e3-46b0-bf73-5cfbdb646188-istio-data" pod "6dc8fcfc-a6e3-46b0-bf73-5cfbdb646188" (UID: "6dc8fcfc-a6e3-46b0-bf73-5cfbdb646188"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:54:25.307561 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:54:25.307500    5570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6dc8fcfc-a6e3-46b0-bf73-5cfbdb646188-istiod-ca-cert\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:54:25.307561 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:54:25.307514    5570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dc8fcfc-a6e3-46b0-bf73-5cfbdb646188-credential-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:54:25.307561 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:54:25.307527    5570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dc8fcfc-a6e3-46b0-bf73-5cfbdb646188-workload-socket\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:54:25.308945 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:54:25.308921    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6dc8fcfc-a6e3-46b0-bf73-5cfbdb646188-kube-api-access-rzkgl" pod "6dc8fcfc-a6e3-46b0-bf73-5cfbdb646188" (UID: "6dc8fcfc-a6e3-46b0-bf73-5cfbdb646188"). InnerVolumeSpecName "kube-api-access-rzkgl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:54:25.309071 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:54:25.309053    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dc8fcfc-a6e3-46b0-bf73-5cfbdb646188-istio-envoy" pod "6dc8fcfc-a6e3-46b0-bf73-5cfbdb646188" (UID: "6dc8fcfc-a6e3-46b0-bf73-5cfbdb646188"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:54:25.309389 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:54:25.309365    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6dc8fcfc-a6e3-46b0-bf73-5cfbdb646188-istio-token" pod "6dc8fcfc-a6e3-46b0-bf73-5cfbdb646188" (UID: "6dc8fcfc-a6e3-46b0-bf73-5cfbdb646188"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:54:25.309462 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:54:25.309447    5570 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6dc8fcfc-a6e3-46b0-bf73-5cfbdb646188-istio-podinfo" pod "6dc8fcfc-a6e3-46b0-bf73-5cfbdb646188" (UID: "6dc8fcfc-a6e3-46b0-bf73-5cfbdb646188"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:54:25.408692 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:54:25.408638    5570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6dc8fcfc-a6e3-46b0-bf73-5cfbdb646188-istio-token\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:54:25.408692 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:54:25.408664    5570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6dc8fcfc-a6e3-46b0-bf73-5cfbdb646188-istio-data\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:54:25.408692 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:54:25.408677    5570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6dc8fcfc-a6e3-46b0-bf73-5cfbdb646188-istio-podinfo\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:54:25.408692 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:54:25.408688    5570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6dc8fcfc-a6e3-46b0-bf73-5cfbdb646188-istio-envoy\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:54:25.408911 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:54:25.408696    5570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzkgl\" (UniqueName: \"kubernetes.io/projected/6dc8fcfc-a6e3-46b0-bf73-5cfbdb646188-kube-api-access-rzkgl\") on node \"ip-10-0-132-4.ec2.internal\" DevicePath \"\""
Nov 03 20:54:25.514762 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:54:25.514737    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-authcfg-http-d04674-openshift-default-c886444fc-x6nwp"]
Nov 03 20:54:25.528101 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:54:25.528078    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-authcfg-http-d04674-openshift-default-c886444fc-x6nwp"]
Nov 03 20:54:25.963856 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:54:25.963823    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6dc8fcfc-a6e3-46b0-bf73-5cfbdb646188" path="/var/lib/kubelet/pods/6dc8fcfc-a6e3-46b0-bf73-5cfbdb646188/volumes"
Nov 03 20:54:26.022400 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:54:26.022377    5570 patch_prober.go:29] interesting pod/e2e-authcfg-http-d04674-openshift-default-c886444fc-x6nwp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.225:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:54:26.022507 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:54:26.022420    5570 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-authcfg-http-d04674-openshift-default-c886444fc-x6nwp" podUID="6dc8fcfc-a6e3-46b0-bf73-5cfbdb646188" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.225:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:54:27.228541 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:54:27.228498    5570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8bwhz" containerName="extractor"
Nov 03 20:54:30.155562 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:54:30.155533    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g69f8"]
Nov 03 20:54:31.026543 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:54:31.026506    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f2d30bd-v7wbx"]
Nov 03 20:54:31.039452 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:54:31.039424    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f2d30bd-v7wbx"]
Nov 03 20:54:31.981787 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:54:31.981732    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87f3acb3-331f-40a9-a7bd-f2d5b4ad3fc4" path="/var/lib/kubelet/pods/87f3acb3-331f-40a9-a7bd-f2d5b4ad3fc4/volumes"
Nov 03 20:54:32.565797 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:54:32.564323    5570 scope.go:122] "RemoveContainer" containerID="3b4111ba8ef93d9aea95435b6361cd7b2ac57f775782dd8d0d42ad3d5ea1777e"
Nov 03 20:54:48.246422 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:54:48.246385    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g69f8"]
Nov 03 20:54:57.228519 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:54:57.228479    5570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8bwhz" containerName="extractor"
Nov 03 20:55:16.247425 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:55:16.247380    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-g69f8"]
Nov 03 20:55:16.423867 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:55:16.423829    5570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-855865dcb4-dx4qk_240accf1-8b59-4221-98b8-ea9589c6e4ba/authorino/0.log"
Nov 03 20:55:26.591435 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:55:26.591402    5570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-bf6bcc597-rwjc5_77fe3c84-a597-4a9d-a49d-392c266db493/manager/0.log"
Nov 03 20:55:27.229245 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:55:27.229209    5570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8bwhz" containerName="extractor"
Nov 03 20:55:27.568228 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:55:27.568159    5570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-66656cd6fd-qq5n6_b2accca8-9328-4135-9957-d2e947b104a7/manager/0.log"
Nov 03 20:55:29.153044 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:55:29.153002    5570 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-796c56b65d-t2j9z_d191c3ba-8755-49a6-8ba4-d731f4cc0de2/maas-api/0.log"
Nov 03 20:55:29.533651 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:55:29.533625    5570 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3e31a506-svzb6_acf7732e-1f8f-4bb5-a04d-5168f3a21bb9/revoke-keys/0.log"
Nov 03 20:55:29.720471 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:55:29.720439    5570 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-452b6d7b-d2fmp_56240608-f9e3-4e52-8a4d-5fcfefbebda8/revoke-keys/0.log"
Nov 03 20:55:30.272420 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:55:30.272390    5570 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f163065b-wfvlt_9944db13-8572-49f1-9e75-5c3a31d3f040/revoke-keys/0.log"
Nov 03 20:55:31.886729 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:55:31.886695    5570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-855865dcb4-dx4qk_240accf1-8b59-4221-98b8-ea9589c6e4ba/authorino/0.log"
Nov 03 20:55:32.538934 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:55:32.538908    5570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-2kn85_f288d261-5a87-46be-b67f-89370c5a4a48/kuadrant-console-plugin/0.log"
Nov 03 20:55:32.665939 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:55:32.665914    5570 scope.go:122] "RemoveContainer" containerID="b5d714f5f9dc7b9de25cb9e37efb202473433323679acdf66cbb2ac8594aa980"
Nov 03 20:55:33.008109 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:55:33.008086    5570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-g69f8_818236b5-238c-4342-abce-f375dbebf689/limitador/0.log"
Nov 03 20:55:33.199833 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:55:33.199806    5570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-fbjbc_6308d31e-2cc7-4e66-9f9f-cf2c878b0043/manager/0.log"
Nov 03 20:55:34.144070 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:55:34.144037    5570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-5mwhf_be4487de-c7d5-4a34-9064-a38ecfee6810/discovery/0.log"
Nov 03 20:55:34.547461 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:55:34.547439    5570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6fd6dbbcd5-s95zx_c8d9abc3-2f2b-4f6f-a8c9-88b67354781b/kube-auth-proxy/0.log"
Nov 03 20:55:34.921568 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:55:34.921491    5570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-694db55c87-657vf_f089dd20-d6a6-4c63-88a5-3fbe9a2ff315/istio-proxy/0.log"
Nov 03 20:55:35.106075 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:55:35.106054    5570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-694db55c87-78txb_b65b5a79-c181-45ec-9cae-ebb5e5bbd865/istio-proxy/0.log"
Nov 03 20:55:35.904930 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:55:35.904898    5570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-694db55c87-msmxq_9ebdd5ef-b321-4dfa-8c3e-1ad7c249a0ff/istio-proxy/0.log"
Nov 03 20:55:36.501768 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:55:36.501741    5570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-694db55c87-wpxcz_36477cba-9827-41ba-997d-d3d646ef4904/istio-proxy/0.log"
Nov 03 20:55:38.353223 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:55:38.353189    5570 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-cpfzl_30b21f09-fd9e-4836-9fc3-c19f818c9022/main/0.log"
Nov 03 20:55:38.919108 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:55:38.919078    5570 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87s5mf_b7306cb9-eeae-447b-b834-870c8d49a5aa/main/0.log"
Nov 03 20:55:39.302329 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:55:39.302300    5570 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-6nh8m_b31639d5-0583-4706-b580-f5752fa502a4/main/0.log"
Nov 03 20:55:39.488011 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:55:39.487985    5570 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz_3a715ef1-38e9-4386-9eec-e10e8df11a10/main/0.log"
Nov 03 20:55:40.300590 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:55:40.300562    5570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-855865dcb4-dx4qk_240accf1-8b59-4221-98b8-ea9589c6e4ba/authorino/0.log"
Nov 03 20:55:45.025908 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:55:45.025873    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-d2fmp"]
Nov 03 20:55:45.039011 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:55:45.038984    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-d2fmp"]
Nov 03 20:55:45.964078 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:55:45.964029    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56240608-f9e3-4e52-8a4d-5fcfefbebda8" path="/var/lib/kubelet/pods/56240608-f9e3-4e52-8a4d-5fcfefbebda8/volumes"
Nov 03 20:55:48.742072 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:55:48.742042    5570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-bf6bcc597-rwjc5_77fe3c84-a597-4a9d-a49d-392c266db493/manager/0.log"
Nov 03 20:55:49.612734 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:55:49.612695    5570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-66656cd6fd-qq5n6_b2accca8-9328-4135-9957-d2e947b104a7/manager/0.log"
Nov 03 20:55:50.931713 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:55:50.931676    5570 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-796c56b65d-t2j9z_d191c3ba-8755-49a6-8ba4-d731f4cc0de2/maas-api/0.log"
Nov 03 20:55:51.301704 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:55:51.301672    5570 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3e31a506-svzb6_acf7732e-1f8f-4bb5-a04d-5168f3a21bb9/revoke-keys/0.log"
Nov 03 20:55:51.836608 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:55:51.836579    5570 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f163065b-wfvlt_9944db13-8572-49f1-9e75-5c3a31d3f040/revoke-keys/0.log"
Nov 03 20:55:53.112261 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:55:53.112230    5570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-855865dcb4-dx4qk_240accf1-8b59-4221-98b8-ea9589c6e4ba/authorino/0.log"
Nov 03 20:55:53.731718 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:55:53.731688    5570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-2kn85_f288d261-5a87-46be-b67f-89370c5a4a48/kuadrant-console-plugin/0.log"
Nov 03 20:55:54.150999 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:55:54.150919    5570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-g69f8_818236b5-238c-4342-abce-f375dbebf689/limitador/0.log"
Nov 03 20:55:54.339659 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:55:54.339631    5570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-fbjbc_6308d31e-2cc7-4e66-9f9f-cf2c878b0043/manager/0.log"
Nov 03 20:55:55.081151 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:55:55.081120    5570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-5mwhf_be4487de-c7d5-4a34-9064-a38ecfee6810/discovery/0.log"
Nov 03 20:55:55.454168 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:55:55.454149    5570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6fd6dbbcd5-s95zx_c8d9abc3-2f2b-4f6f-a8c9-88b67354781b/kube-auth-proxy/0.log"
Nov 03 20:55:55.845893 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:55:55.845830    5570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-694db55c87-657vf_f089dd20-d6a6-4c63-88a5-3fbe9a2ff315/istio-proxy/0.log"
Nov 03 20:55:56.043532 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:55:56.043507    5570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-694db55c87-78txb_b65b5a79-c181-45ec-9cae-ebb5e5bbd865/istio-proxy/0.log"
Nov 03 20:55:56.808825 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:55:56.808798    5570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-694db55c87-msmxq_9ebdd5ef-b321-4dfa-8c3e-1ad7c249a0ff/istio-proxy/0.log"
Nov 03 20:55:57.226106 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:55:57.226077    5570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8bwhz" containerName="extractor"
Nov 03 20:55:57.387381 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:55:57.387354    5570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-694db55c87-wpxcz_36477cba-9827-41ba-997d-d3d646ef4904/istio-proxy/0.log"
Nov 03 20:55:58.821620 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:55:58.821589    5570 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-cpfzl_30b21f09-fd9e-4836-9fc3-c19f818c9022/main/0.log"
Nov 03 20:55:59.397146 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:55:59.397113    5570 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87s5mf_b7306cb9-eeae-447b-b834-870c8d49a5aa/main/0.log"
Nov 03 20:55:59.774645 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:55:59.774616    5570 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-6nh8m_b31639d5-0583-4706-b580-f5752fa502a4/main/0.log"
Nov 03 20:55:59.978561 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:55:59.978534    5570 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-w8wtz_3a715ef1-38e9-4386-9eec-e10e8df11a10/main/0.log"
Nov 03 20:56:01.027614 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:56:01.027581    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-svzb6"]
Nov 03 20:56:01.041001 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:56:01.040970    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-svzb6"]
Nov 03 20:56:01.963643 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:56:01.963586    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="acf7732e-1f8f-4bb5-a04d-5168f3a21bb9" path="/var/lib/kubelet/pods/acf7732e-1f8f-4bb5-a04d-5168f3a21bb9/volumes"
Nov 03 20:56:24.026708 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:56:24.026678    5570 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f163065b-wfvlt"]
Nov 03 20:56:24.040045 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:56:24.040009    5570 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f163065b-wfvlt"]
Nov 03 20:56:25.963854 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:56:25.963756    5570 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9944db13-8572-49f1-9e75-5c3a31d3f040" path="/var/lib/kubelet/pods/9944db13-8572-49f1-9e75-5c3a31d3f040/volumes"
Nov 03 20:56:27.227449 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:56:27.227411    5570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8bwhz" containerName="extractor"
Nov 03 20:56:29.865242 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:56:29.865211    5570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-66656cd6fd-qq5n6_b2accca8-9328-4135-9957-d2e947b104a7/manager/0.log"
Nov 03 20:56:30.124355 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:56:30.124288    5570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-66656cd6fd-qq5n6_b2accca8-9328-4135-9957-d2e947b104a7/manager/0.log"
Nov 03 20:56:32.685348 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:56:32.685314    5570 scope.go:122] "RemoveContainer" containerID="d2542e1e84125cfe5c2172dd3c025299ed6d67955ab79c3866c39db032016fc8"
Nov 03 20:56:57.230183 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:56:57.230142    5570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8bwhz" containerName="extractor"
Nov 03 20:57:08.581621 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:57:08.581587    5570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-66656cd6fd-qq5n6_b2accca8-9328-4135-9957-d2e947b104a7/manager/0.log"
Nov 03 20:57:08.857455 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:57:08.857385    5570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-66656cd6fd-qq5n6_b2accca8-9328-4135-9957-d2e947b104a7/manager/0.log"
Nov 03 20:57:14.369580 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:57:14.369551    5570 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-j4jzv_51dffd76-5c30-440c-85ed-d81221663833/global-pull-secret-syncer/0.log"
Nov 03 20:57:15.265952 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:57:15.265924    5570 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-5p48g_15b5070f-d0d3-4ad8-aa1d-93cc7b96003a/konnectivity-agent/0.log"
Nov 03 20:57:15.558765 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:57:15.558662    5570 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-132-4.ec2.internal_a66c9af704a53c4c9b04653e0d8d8d8f/haproxy/0.log"
Nov 03 20:57:21.307956 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:57:21.307927    5570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-855865dcb4-dx4qk_240accf1-8b59-4221-98b8-ea9589c6e4ba/authorino/0.log"
Nov 03 20:57:21.660017 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:57:21.659954    5570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-2kn85_f288d261-5a87-46be-b67f-89370c5a4a48/kuadrant-console-plugin/0.log"
Nov 03 20:57:21.936316 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:57:21.936240    5570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-g69f8_818236b5-238c-4342-abce-f375dbebf689/limitador/0.log"
Nov 03 20:57:22.099597 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:57:22.099570    5570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-fbjbc_6308d31e-2cc7-4e66-9f9f-cf2c878b0043/manager/0.log"
Nov 03 20:57:23.454659 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:57:23.454629    5570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_4727cec2-c7e7-4965-ae91-a78dfc4ec7f5/alertmanager/0.log"
Nov 03 20:57:23.577359 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:57:23.577332    5570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_4727cec2-c7e7-4965-ae91-a78dfc4ec7f5/config-reloader/0.log"
Nov 03 20:57:23.612682 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:57:23.612659    5570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-fng2b/perf-node-gather-daemonset-j7762"]
Nov 03 20:57:23.613093 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:57:23.613081    5570 cpu_manager.go:414] "containerMap: removing container" podUID="6dc8fcfc-a6e3-46b0-bf73-5cfbdb646188" containerName="istio-proxy"
Nov 03 20:57:23.613141 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:57:23.613095    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6dc8fcfc-a6e3-46b0-bf73-5cfbdb646188" containerName="istio-proxy"
Nov 03 20:57:23.613141 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:57:23.613114    5570 cpu_manager.go:414] "containerMap: removing container" podUID="6dc8fcfc-a6e3-46b0-bf73-5cfbdb646188" containerName="istio-proxy"
Nov 03 20:57:23.613141 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:57:23.613119    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6dc8fcfc-a6e3-46b0-bf73-5cfbdb646188" containerName="istio-proxy"
Nov 03 20:57:23.613141 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:57:23.613126    5570 cpu_manager.go:414] "containerMap: removing container" podUID="6c21c686-111d-4b12-9589-c4f64fdd44a9" containerName="payload-processing"
Nov 03 20:57:23.613141 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:57:23.613131    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c21c686-111d-4b12-9589-c4f64fdd44a9" containerName="payload-processing"
Nov 03 20:57:23.613308 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:57:23.613149    5570 cpu_manager.go:414] "containerMap: removing container" podUID="aec79c41-24ca-408e-8f1a-6783c84203fd" containerName="payload-pre-processing"
Nov 03 20:57:23.613308 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:57:23.613154    5570 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aec79c41-24ca-408e-8f1a-6783c84203fd" containerName="payload-pre-processing"
Nov 03 20:57:23.613308 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:57:23.613222    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="6dc8fcfc-a6e3-46b0-bf73-5cfbdb646188" containerName="istio-proxy"
Nov 03 20:57:23.613308 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:57:23.613231    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c21c686-111d-4b12-9589-c4f64fdd44a9" containerName="payload-processing"
Nov 03 20:57:23.613308 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:57:23.613242    5570 memory_manager.go:365] "RemoveStaleState removing state" podUID="aec79c41-24ca-408e-8f1a-6783c84203fd" containerName="payload-pre-processing"
Nov 03 20:57:23.616439 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:57:23.616420    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fng2b/perf-node-gather-daemonset-j7762"
Nov 03 20:57:23.637715 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:57:23.637693    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fng2b/perf-node-gather-daemonset-j7762"]
Nov 03 20:57:23.648845 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:57:23.648824    5570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-fng2b\"/\"default-dockercfg-6f4g7\""
Nov 03 20:57:23.649080 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:57:23.648842    5570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-fng2b\"/\"kube-root-ca.crt\""
Nov 03 20:57:23.649229 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:57:23.648886    5570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-fng2b\"/\"openshift-service-ca.crt\""
Nov 03 20:57:23.693935 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:57:23.693916    5570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_4727cec2-c7e7-4965-ae91-a78dfc4ec7f5/kube-rbac-proxy-web/0.log"
Nov 03 20:57:23.766535 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:57:23.766488    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/97002830-a962-4fff-bf0d-582f48ea30f2-sys\") pod \"perf-node-gather-daemonset-j7762\" (UID: \"97002830-a962-4fff-bf0d-582f48ea30f2\") " pod="openshift-must-gather-fng2b/perf-node-gather-daemonset-j7762"
Nov 03 20:57:23.766535 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:57:23.766523    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/97002830-a962-4fff-bf0d-582f48ea30f2-proc\") pod \"perf-node-gather-daemonset-j7762\" (UID: \"97002830-a962-4fff-bf0d-582f48ea30f2\") " pod="openshift-must-gather-fng2b/perf-node-gather-daemonset-j7762"
Nov 03 20:57:23.766674 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:57:23.766611    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/97002830-a962-4fff-bf0d-582f48ea30f2-podres\") pod \"perf-node-gather-daemonset-j7762\" (UID: \"97002830-a962-4fff-bf0d-582f48ea30f2\") " pod="openshift-must-gather-fng2b/perf-node-gather-daemonset-j7762"
Nov 03 20:57:23.766674 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:57:23.766663    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/97002830-a962-4fff-bf0d-582f48ea30f2-lib-modules\") pod \"perf-node-gather-daemonset-j7762\" (UID: \"97002830-a962-4fff-bf0d-582f48ea30f2\") " pod="openshift-must-gather-fng2b/perf-node-gather-daemonset-j7762"
Nov 03 20:57:23.766804 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:57:23.766687    5570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmpnc\" (UniqueName: \"kubernetes.io/projected/97002830-a962-4fff-bf0d-582f48ea30f2-kube-api-access-vmpnc\") pod \"perf-node-gather-daemonset-j7762\" (UID: \"97002830-a962-4fff-bf0d-582f48ea30f2\") " pod="openshift-must-gather-fng2b/perf-node-gather-daemonset-j7762"
Nov 03 20:57:23.810333 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:57:23.810313    5570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_4727cec2-c7e7-4965-ae91-a78dfc4ec7f5/kube-rbac-proxy/0.log"
Nov 03 20:57:23.867097 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:57:23.867075    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/97002830-a962-4fff-bf0d-582f48ea30f2-podres\") pod \"perf-node-gather-daemonset-j7762\" (UID: \"97002830-a962-4fff-bf0d-582f48ea30f2\") " pod="openshift-must-gather-fng2b/perf-node-gather-daemonset-j7762"
Nov 03 20:57:23.867212 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:57:23.867119    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/97002830-a962-4fff-bf0d-582f48ea30f2-lib-modules\") pod \"perf-node-gather-daemonset-j7762\" (UID: \"97002830-a962-4fff-bf0d-582f48ea30f2\") " pod="openshift-must-gather-fng2b/perf-node-gather-daemonset-j7762"
Nov 03 20:57:23.867212 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:57:23.867140    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmpnc\" (UniqueName: \"kubernetes.io/projected/97002830-a962-4fff-bf0d-582f48ea30f2-kube-api-access-vmpnc\") pod \"perf-node-gather-daemonset-j7762\" (UID: \"97002830-a962-4fff-bf0d-582f48ea30f2\") " pod="openshift-must-gather-fng2b/perf-node-gather-daemonset-j7762"
Nov 03 20:57:23.867212 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:57:23.867161    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/97002830-a962-4fff-bf0d-582f48ea30f2-sys\") pod \"perf-node-gather-daemonset-j7762\" (UID: \"97002830-a962-4fff-bf0d-582f48ea30f2\") " pod="openshift-must-gather-fng2b/perf-node-gather-daemonset-j7762"
Nov 03 20:57:23.867212 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:57:23.867188    5570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/97002830-a962-4fff-bf0d-582f48ea30f2-proc\") pod \"perf-node-gather-daemonset-j7762\" (UID: \"97002830-a962-4fff-bf0d-582f48ea30f2\") " pod="openshift-must-gather-fng2b/perf-node-gather-daemonset-j7762"
Nov 03 20:57:23.867390 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:57:23.867229    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/97002830-a962-4fff-bf0d-582f48ea30f2-lib-modules\") pod \"perf-node-gather-daemonset-j7762\" (UID: \"97002830-a962-4fff-bf0d-582f48ea30f2\") " pod="openshift-must-gather-fng2b/perf-node-gather-daemonset-j7762"
Nov 03 20:57:23.867390 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:57:23.867279    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/97002830-a962-4fff-bf0d-582f48ea30f2-proc\") pod \"perf-node-gather-daemonset-j7762\" (UID: \"97002830-a962-4fff-bf0d-582f48ea30f2\") " pod="openshift-must-gather-fng2b/perf-node-gather-daemonset-j7762"
Nov 03 20:57:23.867390 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:57:23.867284    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/97002830-a962-4fff-bf0d-582f48ea30f2-podres\") pod \"perf-node-gather-daemonset-j7762\" (UID: \"97002830-a962-4fff-bf0d-582f48ea30f2\") " pod="openshift-must-gather-fng2b/perf-node-gather-daemonset-j7762"
Nov 03 20:57:23.867390 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:57:23.867284    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/97002830-a962-4fff-bf0d-582f48ea30f2-sys\") pod \"perf-node-gather-daemonset-j7762\" (UID: \"97002830-a962-4fff-bf0d-582f48ea30f2\") " pod="openshift-must-gather-fng2b/perf-node-gather-daemonset-j7762"
Nov 03 20:57:23.887476 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:57:23.887455    5570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmpnc\" (UniqueName: \"kubernetes.io/projected/97002830-a962-4fff-bf0d-582f48ea30f2-kube-api-access-vmpnc\") pod \"perf-node-gather-daemonset-j7762\" (UID: \"97002830-a962-4fff-bf0d-582f48ea30f2\") " pod="openshift-must-gather-fng2b/perf-node-gather-daemonset-j7762"
Nov 03 20:57:23.925184 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:57:23.925165    5570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_4727cec2-c7e7-4965-ae91-a78dfc4ec7f5/kube-rbac-proxy-metric/0.log"
Nov 03 20:57:23.929162 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:57:23.929147    5570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fng2b/perf-node-gather-daemonset-j7762"
Nov 03 20:57:24.046807 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:57:24.046693    5570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_4727cec2-c7e7-4965-ae91-a78dfc4ec7f5/prom-label-proxy/0.log"
Nov 03 20:57:24.155604 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:57:24.155581    5570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_4727cec2-c7e7-4965-ae91-a78dfc4ec7f5/init-config-reloader/0.log"
Nov 03 20:57:24.221699 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:57:24.221673    5570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fng2b/perf-node-gather-daemonset-j7762"]
Nov 03 20:57:24.223126 ip-10-0-132-4 kubenswrapper[5570]: W1103 20:57:24.223101    5570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod97002830_a962_4fff_bf0d_582f48ea30f2.slice/crio-296d7ad4e74cc68c4f9693df2012263898c4ec200f239eca1d8c9f8383f798da WatchSource:0}: Error finding container 296d7ad4e74cc68c4f9693df2012263898c4ec200f239eca1d8c9f8383f798da: Status 404 returned error can't find the container with id 296d7ad4e74cc68c4f9693df2012263898c4ec200f239eca1d8c9f8383f798da
Nov 03 20:57:24.401990 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:57:24.401968    5570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-q8b6s_ffe074b1-e2a0-403b-9f81-581fdb4d2775/kube-state-metrics/0.log"
Nov 03 20:57:24.510858 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:57:24.510837    5570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-q8b6s_ffe074b1-e2a0-403b-9f81-581fdb4d2775/kube-rbac-proxy-main/0.log"
Nov 03 20:57:24.623491 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:57:24.623442    5570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-q8b6s_ffe074b1-e2a0-403b-9f81-581fdb4d2775/kube-rbac-proxy-self/0.log"
Nov 03 20:57:24.739592 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:57:24.739570    5570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-654b7fcdc8-l6l9f_d5d14caa-6661-4146-b888-59c3518216d4/metrics-server/0.log"
Nov 03 20:57:25.132016 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:57:25.131986    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fng2b/perf-node-gather-daemonset-j7762" event={"ID":"97002830-a962-4fff-bf0d-582f48ea30f2","Type":"ContainerStarted","Data":"c7fdfa6e7ca2b501e7b535dea07b1f66bed59abcc27b2c826b1d8139e119ade7"}
Nov 03 20:57:25.132146 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:57:25.132037    5570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fng2b/perf-node-gather-daemonset-j7762" event={"ID":"97002830-a962-4fff-bf0d-582f48ea30f2","Type":"ContainerStarted","Data":"296d7ad4e74cc68c4f9693df2012263898c4ec200f239eca1d8c9f8383f798da"}
Nov 03 20:57:25.132146 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:57:25.132116    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-fng2b/perf-node-gather-daemonset-j7762"
Nov 03 20:57:25.171558 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:57:25.171514    5570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-fng2b/perf-node-gather-daemonset-j7762" podStartSLOduration=2.171501684 podStartE2EDuration="2.171501684s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:57:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:57:25.170546342 +0000 UTC m=+3179.323208227" watchObservedRunningTime="2026-11-03 20:57:25.171501684 +0000 UTC m=+3179.324163571"
Nov 03 20:57:25.682383 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:57:25.682345    5570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-j46kr_74d4808e-24ce-4c47-b4ce-e1188eeea6f1/node-exporter/0.log"
Nov 03 20:57:25.792200 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:57:25.792174    5570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-j46kr_74d4808e-24ce-4c47-b4ce-e1188eeea6f1/kube-rbac-proxy/0.log"
Nov 03 20:57:25.903022 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:57:25.903001    5570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-j46kr_74d4808e-24ce-4c47-b4ce-e1188eeea6f1/init-textfile/0.log"
Nov 03 20:57:26.018921 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:57:26.018896    5570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-rwhnf_cb9bef9b-c7cd-410e-8c1e-1b27474daf81/kube-rbac-proxy-main/0.log"
Nov 03 20:57:26.135849 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:57:26.129872    5570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-rwhnf_cb9bef9b-c7cd-410e-8c1e-1b27474daf81/kube-rbac-proxy-self/0.log"
Nov 03 20:57:26.241242 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:57:26.241219    5570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-rwhnf_cb9bef9b-c7cd-410e-8c1e-1b27474daf81/openshift-state-metrics/0.log"
Nov 03 20:57:27.227977 ip-10-0-132-4 kubenswrapper[5570]: E1103 20:57:27.227944    5570 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8bwhz" containerName="extractor"
Nov 03 20:57:27.556365 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:57:27.556295    5570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-567ccc7984-qnsvp_4f0d1d22-fa21-4f7c-8330-363e32444e3c/telemeter-client/0.log"
Nov 03 20:57:27.669610 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:57:27.669580    5570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-567ccc7984-qnsvp_4f0d1d22-fa21-4f7c-8330-363e32444e3c/reload/0.log"
Nov 03 20:57:27.782049 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:57:27.782024    5570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-567ccc7984-qnsvp_4f0d1d22-fa21-4f7c-8330-363e32444e3c/kube-rbac-proxy/0.log"
Nov 03 20:57:27.901569 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:57:27.901508    5570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5bd4bfb68d-9449x_59f6a4d6-afe0-4c96-a54a-d8e6b5c66a38/thanos-query/0.log"
Nov 03 20:57:28.012934 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:57:28.012907    5570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5bd4bfb68d-9449x_59f6a4d6-afe0-4c96-a54a-d8e6b5c66a38/kube-rbac-proxy-web/0.log"
Nov 03 20:57:28.125473 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:57:28.125449    5570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5bd4bfb68d-9449x_59f6a4d6-afe0-4c96-a54a-d8e6b5c66a38/kube-rbac-proxy/0.log"
Nov 03 20:57:28.238571 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:57:28.238547    5570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5bd4bfb68d-9449x_59f6a4d6-afe0-4c96-a54a-d8e6b5c66a38/prom-label-proxy/0.log"
Nov 03 20:57:28.350101 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:57:28.350077    5570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5bd4bfb68d-9449x_59f6a4d6-afe0-4c96-a54a-d8e6b5c66a38/kube-rbac-proxy-rules/0.log"
Nov 03 20:57:28.461891 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:57:28.461870    5570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5bd4bfb68d-9449x_59f6a4d6-afe0-4c96-a54a-d8e6b5c66a38/kube-rbac-proxy-metrics/0.log"
Nov 03 20:57:31.152256 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:57:31.152226    5570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-fng2b/perf-node-gather-daemonset-j7762"
Nov 03 20:57:32.706701 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:57:32.706672    5570 scope.go:122] "RemoveContainer" containerID="13bb865a6218f93bff6f3ea21004248a12a035f9bcfef72acabe68e18cb9ad93"
Nov 03 20:57:32.715078 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:57:32.715056    5570 scope.go:122] "RemoveContainer" containerID="177533e9e7e2a055099a8e08174c9255b62ad140e3148b66c808c57f589ae7bb"
Nov 03 20:57:32.722675 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:57:32.722659    5570 scope.go:122] "RemoveContainer" containerID="bc093fef1a201af7ac1598c0023022844d1319ea61daee70113aeecb883b0408"
Nov 03 20:57:35.877736 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:57:35.877712    5570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-7zl2f_2fd3c650-2f4c-404e-ab06-c50612af72ce/dns/0.log"
Nov 03 20:57:35.992416 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:57:35.992386    5570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-7zl2f_2fd3c650-2f4c-404e-ab06-c50612af72ce/kube-rbac-proxy/0.log"
Nov 03 20:57:36.240151 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:57:36.240120    5570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-t6dqw_716cc9cf-d26a-481b-8114-169c6104ce56/dns-node-resolver/0.log"
Nov 03 20:57:36.604135 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:57:36.604077    5570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-5mwhf_be4487de-c7d5-4a34-9064-a38ecfee6810/discovery/0.log"
Nov 03 20:57:36.699850 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:57:36.699822    5570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6fd6dbbcd5-s95zx_c8d9abc3-2f2b-4f6f-a8c9-88b67354781b/kube-auth-proxy/0.log"
Nov 03 20:57:36.828482 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:57:36.828449    5570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-694db55c87-657vf_f089dd20-d6a6-4c63-88a5-3fbe9a2ff315/istio-proxy/0.log"
Nov 03 20:57:36.891215 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:57:36.891162    5570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-694db55c87-78txb_b65b5a79-c181-45ec-9cae-ebb5e5bbd865/istio-proxy/0.log"
Nov 03 20:57:37.139486 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:57:37.139464    5570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-694db55c87-msmxq_9ebdd5ef-b321-4dfa-8c3e-1ad7c249a0ff/istio-proxy/0.log"
Nov 03 20:57:37.331985 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:57:37.331956    5570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-694db55c87-wpxcz_36477cba-9827-41ba-997d-d3d646ef4904/istio-proxy/0.log"
Nov 03 20:57:37.753942 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:57:37.753917    5570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-zzt8t_1d4ce60c-e058-4c29-99e8-5f4f6eead744/node-ca/0.log"
Nov 03 20:57:40.616814 ip-10-0-132-4 kubenswrapper[5570]: I1103 20:57:40.616783    5570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-5mwhf_be4487de-c7d5-4a34-9064-a38ecfee6810/discovery/0.log"